Click here to Skip to main content
14,485,088 members

Cloud Computing

 
QuestionMessage Closed Pin
24-Mar-20 3:15
MemberMember 1469214024-Mar-20 3:15 
QuestionASP.NET + Multi Form using Textbox problem Pin
kerek224-Feb-20 21:42
Memberkerek224-Feb-20 21:42 
AnswerRe: ASP.NET + Multi Form using Textbox problem Pin
Richard MacCutchan24-Feb-20 23:17
mveRichard MacCutchan24-Feb-20 23:17 
QuestionAzure Data Factory pipeline into compressed Parquet file: “java.lang.OutOfMemoryError:Java heap space” Pin
VictorSotnikov16-Sep-19 10:20
MemberVictorSotnikov16-Sep-19 10:20 
I have met the following problem. I have a pipeline that reads data from an MS SQL Server and stores them into a file in a BLOB container in Azure Storage. The file has Parquet (or Apache Parquet, as it is also called) format.
So, when the “sink” (output) file is stored in a compressed way (snappy, or gzip – does not matter) AND the file is large enough (more than 50 Mb), the pipeline failed. The message was the following:
"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=UserErrorJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.lang.OutOfMemoryError:Java heap space\ntotal entry:11\r\njava.util.ArrayDeque.doubleCapacity(Unknown Source)\r\njava.util.ArrayDeque.addFirst(Unknown Source)\r\njava.util.ArrayDeque.push(Unknown Source)\r\norg.apache.parquet.io.ValidatingRecordConsumer.endField(ValidatingRecordConsumer.java:108)\r\norg.apache.parquet.example.data.GroupWriter.writeGroup(GroupWriter.java:58)\r\norg.apache.parquet.example.data.GroupWriter.write(GroupWriter.java:37)\r\norg.apache.parquet.hadoop.example.GroupWriteSupport.write(GroupWriteSupport.java:87)\r\norg.apache.parquet.hadoop.example.GroupWriteSupport.write(GroupWriteSupport.java:37)\r\norg.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:123)\r\norg.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:292)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetBatchWriter.addRows(ParquetBatchWriter.java:60)\r\n,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'",
"failureType": "UserError",
"target": "Work_Work"
}

The "Work_Work" is name of a Copy Data activity in the pipeline.
If I turn the compression off (the generated BLOB file is uncompressed), the error does not happen.
Is this the error described in https://docs.microsoft.com/en-us/azure/data-factory/format-parquet: the “…If you copy data to/from Parquet format using Self-hosted Integration Runtime and hit error saying "An error occurred when invoking java, message: java.lang.OutOfMemoryError:Java heap space", you can add an environment variable _JAVA_OPTIONS in the machine that hosts the Self-hosted IR to adjust the min/max heap size for JVM to empower such copy, then rerun the pipeline….”?

If it is, have I understood correctly that I have to do the following:
To go to a server where the “Self-hosted Integration Runtime” (still have no idea what it is) and increase the max heap size for JVM. Is this correct?
If it is, my next question is: how large the max heap size should be? My pipeline can generate a file whose size will be 30 GB.
What “max heap size” can guarantee that such a file will not cause the fail?
QuestionHow to search for a word or color on a website? Pin
l@rs_30-Aug-19 8:17
Memberl@rs_30-Aug-19 8:17 
AnswerRe: How to search for a word or color on a website? Pin
Afzaal Ahmad Zeeshan30-Aug-19 12:02
mvaAfzaal Ahmad Zeeshan30-Aug-19 12:02 
QuestionWhat are some of Azure costing to get minimal app running? Pin
maze328-May-19 5:11
professionalmaze328-May-19 5:11 
QuestionHow to see the payload sent to my Azure web service Pin
Hypermommy2-May-18 2:57
MemberHypermommy2-May-18 2:57 
AnswerRe: How to see the payload sent to my Azure web service Pin
Richard Deeming2-May-18 8:45
mveRichard Deeming2-May-18 8:45 
AnswerRe: How to see the payload sent to my Azure web service Pin
Afzaal Ahmad Zeeshan6-May-18 3:26
mvaAfzaal Ahmad Zeeshan6-May-18 3:26 
GeneralRe: How to see the payload sent to my Azure web service Pin
Hypermommy11-Jul-18 4:20
MemberHypermommy11-Jul-18 4:20 
QuestionAccessing Graph.microsoft.com on behalf of a customer tennant Pin
me@dagsunde.com26-Feb-18 0:48
professionalme@dagsunde.com26-Feb-18 0:48 
AnswerRe: Accessing Graph.microsoft.com on behalf of a customer tennant Pin
Afzaal Ahmad Zeeshan6-May-18 3:15
mvaAfzaal Ahmad Zeeshan6-May-18 3:15 
QuestionWindows IIS Hosting - Upgrading Options From Shared Hosting With a UK Data Centre to Something With Better Performance Pin
Will Stanley21-Jan-17 0:45
MemberWill Stanley21-Jan-17 0:45 
AnswerRe: Windows IIS Hosting - Upgrading Options From Shared Hosting With a UK Data Centre to Something With Better Performance Pin
Afzaal Ahmad Zeeshan21-Jan-17 2:13
mvaAfzaal Ahmad Zeeshan21-Jan-17 2:13 
GeneralRe: Windows IIS Hosting - Upgrading Options From Shared Hosting With a UK Data Centre to Something With Better Performance Pin
Will Stanley23-Jan-17 9:31
MemberWill Stanley23-Jan-17 9:31 
AnswerMessage Closed Pin
25-Jul-17 21:05
professionalpolesmith25-Jul-17 21:05 
QuestionRead Azure Filesystem-Files (not VM) Pin
gpc4419-Jun-16 23:44
Membergpc4419-Jun-16 23:44 
AnswerRe: Read Azure Filesystem-Files (not VM) Pin
debasish mishra7-Jan-18 20:31
professionaldebasish mishra7-Jan-18 20:31 
QuestionAzure subdomains... Pin
R. Giskard Reventlov8-May-16 14:18
MemberR. Giskard Reventlov8-May-16 14:18 
AnswerRe: Azure subdomains... Pin
User 17164928-May-16 22:04
professionalUser 17164928-May-16 22:04 
GeneralRe: Azure subdomains... Pin
R. Giskard Reventlov9-May-16 5:00
MemberR. Giskard Reventlov9-May-16 5:00 
Questiondesign hadoop Pin
Member 123788191-Apr-16 5:43
MemberMember 123788191-Apr-16 5:43 
AnswerRe: design hadoop Pin
Richard MacCutchan1-Apr-16 6:10
mveRichard MacCutchan1-Apr-16 6:10 
AnswerRe: design hadoop Pin
ZurdoDev12-Apr-16 5:13
mveZurdoDev12-Apr-16 5:13 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.