[jira] [Commented] (BEAM-11002) XmlIO buffer overflow exception

["Beam JIRA Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-11002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235596#comment-17235596 ] 

Beam JIRA Bot commented on BEAM-11002:
--------------------------------------

This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned.

> XmlIO buffer overflow exception 
> --------------------------------
>
>                 Key: BEAM-11002
>                 URL: https://issues.apache.org/jira/browse/BEAM-11002
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-xml
>    Affects Versions: 2.23.0, 2.24.0
>            Reporter: Duncan Lew
>            Assignee: Chamikara Madhusanka Jayalath
>            Priority: P1
>              Labels: Clarified, stale-assigned
>
> We're making using of Apache Beam in Google Dataflow.
> We're using XmlIO to read in an XML file with such a setup
> {code:java}
> pipeline
>                     .apply("Read Storage Bucket",
>                             XmlIO.read<XmlProduct>()
>                                     .from(sourcePath)
>                                     .withRootElement(xmlProductRoot)
>                                     .withRecordElement(xmlProductRecord)
>                                     .withRecordClass(XmlProduct::class.java)
>                     )
> {code}
> However, from time to time, we're getting buffer overflow exception from reading random xml files:
> {code:java}
> "Error message from worker: java.io.IOException: Failed to start reading from source: gs://path-to-xml-file.xml range [1722550, 2684411)
> 	org.apache.beam.runners.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.start(WorkerCustomSources.java:610)
> 	org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:359)
> 	org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:194)
> 	org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
> 	org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
> 	org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:417)
> 	org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:386)
> 	org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:311)
> 	org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
> 	org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
> 	org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
> 	java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 	java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 	java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 	java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.nio.BufferOverflowException
> 	java.base/java.nio.Buffer.nextPutIndex(Buffer.java:662)
> 	java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:196)
> 	org.apache.beam.sdk.io.xml.XmlSource$XMLReader.getFirstOccurenceOfRecordElement(XmlSource.java:285)
> 	org.apache.beam.sdk.io.xml.XmlSource$XMLReader.startReading(XmlSource.java:192)
> 	org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:476)
> 	org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)
> 	org.apache.beam.runners.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.start(WorkerCustomSources.java:607)
> 	... 14 more
> {code}
> We can't reproduce this buffer overflow exception locally with the DirectRunner. If we rerun the dataflow job in the Google Cloud, it can run correctly without any exceptions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)