You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Zhiheng Huang <sy...@gmail.com> on 2019/08/27 18:40:47 UTC
NoSuchElementException after Adding SplittableDoFn
Hi all,
Looking for help to understand an internal error from beam dataflow runner.
I have a streaming pipeline that is on google dataflow runner(beam version
2.11, java). Recently I added a SplittableDoFn to my pipeline to
continuously generate a sequence. However, after the job runs for a few
hours, I start to see the following exception:
java.util.NoSuchElementException
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.MultitransformedIterator.next(MultitransformedIterator.java:63)
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:308)
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:294)
org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.getUnderlyingWindow(DataflowProcessFnRunner.java:98)
org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.placeIntoElementWindow(DataflowProcessFnRunner.java:72)
org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.processElement(DataflowProcessFnRunner.java:62)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:325)
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1269)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:146)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1008)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
This is 100% reproducible and I am clueless about how this stacktrace can
be debugged since there's no pointer to user code nor the failing
PTransform. This happens only if I add the SplittableDoFn, whose rough
structure is like this
<https://gist.github.com/sylvon/cbcccdcb64aeb15002721977398dc308>.
Appreciate any pointers on what might be going wrong or how this exception
can be debugged.
Thanks a lot!
Re: NoSuchElementException after Adding SplittableDoFn
Posted by Zhiheng Huang <sy...@gmail.com>.
Thanks!
On Tue, Aug 27, 2019 at 2:14 PM Lukasz Cwik <lc...@google.com> wrote:
> You could clone the 2.X.0 branch of the Beam repo and patch in the commit
> I referenced above. Then build the gradle target
> :runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar
> which should produce a jar in the build/libs/ directory. Pass the
> additional flag --dataflowWorkerJar=path/to/worker/shadow.jar when running
> your pipeline using 2.X
>
> The code in the patched file hasn't changed in a while and you might be
> fine building it using 2.11.0 but if that doesn't work try using one of the
> newer releases.
>
>
> On Tue, Aug 27, 2019 at 1:12 PM Zhiheng Huang <sy...@gmail.com>
> wrote:
>
>> Thanks! Glad that this will be fixed in future release.
>>
>> Is there anyway that I can avoid hitting this problem before 2.16 is
>> released?
>>
>> On Tue, Aug 27, 2019 at 12:57 PM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> This is a known issue and was fixed with
>>> https://github.com/apache/beam/commit/5d9bb4595c763025a369a959e18c6dd288e72314#diff-f149847d2c06f56ea591cab8d862c960
>>>
>>> It is meant to be released as part of 2.16.0
>>>
>>> On Tue, Aug 27, 2019 at 11:41 AM Zhiheng Huang <sy...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Looking for help to understand an internal error from beam dataflow
>>>> runner.
>>>>
>>>> I have a streaming pipeline that is on google dataflow runner(beam
>>>> version 2.11, java). Recently I added a SplittableDoFn to my pipeline to
>>>> continuously generate a sequence. However, after the job runs for a few
>>>> hours, I start to see the following exception:
>>>>
>>>> java.util.NoSuchElementException
>>>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.MultitransformedIterator.next(MultitransformedIterator.java:63)
>>>>
>>>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
>>>>
>>>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:308)
>>>>
>>>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:294)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.getUnderlyingWindow(DataflowProcessFnRunner.java:98)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.placeIntoElementWindow(DataflowProcessFnRunner.java:72)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.processElement(DataflowProcessFnRunner.java:62)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:325)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1269)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:146)
>>>>
>>>> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1008)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> java.lang.Thread.run(Thread.java:745)
>>>>
>>>> This is 100% reproducible and I am clueless about how this stacktrace
>>>> can be debugged since there's no pointer to user code nor the failing
>>>> PTransform. This happens only if I add the SplittableDoFn, whose rough
>>>> structure is like this
>>>> <https://gist.github.com/sylvon/cbcccdcb64aeb15002721977398dc308>.
>>>>
>>>> Appreciate any pointers on what might be going wrong or how this
>>>> exception can be debugged.
>>>>
>>>> Thanks a lot!
>>>>
>>>>
>>
>> --
>> Sylvon Huang
>>
>
--
Sylvon Huang
Re: NoSuchElementException after Adding SplittableDoFn
Posted by Lukasz Cwik <lc...@google.com>.
You could clone the 2.X.0 branch of the Beam repo and patch in the commit I
referenced above. Then build the gradle target
:runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar
which should produce a jar in the build/libs/ directory. Pass the
additional flag --dataflowWorkerJar=path/to/worker/shadow.jar when running
your pipeline using 2.X
The code in the patched file hasn't changed in a while and you might be
fine building it using 2.11.0 but if that doesn't work try using one of the
newer releases.
On Tue, Aug 27, 2019 at 1:12 PM Zhiheng Huang <sy...@gmail.com> wrote:
> Thanks! Glad that this will be fixed in future release.
>
> Is there anyway that I can avoid hitting this problem before 2.16 is
> released?
>
> On Tue, Aug 27, 2019 at 12:57 PM Lukasz Cwik <lc...@google.com> wrote:
>
>> This is a known issue and was fixed with
>> https://github.com/apache/beam/commit/5d9bb4595c763025a369a959e18c6dd288e72314#diff-f149847d2c06f56ea591cab8d862c960
>>
>> It is meant to be released as part of 2.16.0
>>
>> On Tue, Aug 27, 2019 at 11:41 AM Zhiheng Huang <sy...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> Looking for help to understand an internal error from beam dataflow
>>> runner.
>>>
>>> I have a streaming pipeline that is on google dataflow runner(beam
>>> version 2.11, java). Recently I added a SplittableDoFn to my pipeline to
>>> continuously generate a sequence. However, after the job runs for a few
>>> hours, I start to see the following exception:
>>>
>>> java.util.NoSuchElementException
>>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.MultitransformedIterator.next(MultitransformedIterator.java:63)
>>>
>>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
>>>
>>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:308)
>>>
>>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:294)
>>>
>>> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.getUnderlyingWindow(DataflowProcessFnRunner.java:98)
>>>
>>> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.placeIntoElementWindow(DataflowProcessFnRunner.java:72)
>>>
>>> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.processElement(DataflowProcessFnRunner.java:62)
>>>
>>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:325)
>>>
>>> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>>>
>>> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>>>
>>> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
>>>
>>> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
>>>
>>> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
>>>
>>> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1269)
>>>
>>> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:146)
>>>
>>> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1008)
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> java.lang.Thread.run(Thread.java:745)
>>>
>>> This is 100% reproducible and I am clueless about how this stacktrace
>>> can be debugged since there's no pointer to user code nor the failing
>>> PTransform. This happens only if I add the SplittableDoFn, whose rough
>>> structure is like this
>>> <https://gist.github.com/sylvon/cbcccdcb64aeb15002721977398dc308>.
>>>
>>> Appreciate any pointers on what might be going wrong or how this
>>> exception can be debugged.
>>>
>>> Thanks a lot!
>>>
>>>
>
> --
> Sylvon Huang
>
Re: NoSuchElementException after Adding SplittableDoFn
Posted by Zhiheng Huang <sy...@gmail.com>.
Thanks! Glad that this will be fixed in future release.
Is there anyway that I can avoid hitting this problem before 2.16 is
released?
On Tue, Aug 27, 2019 at 12:57 PM Lukasz Cwik <lc...@google.com> wrote:
> This is a known issue and was fixed with
> https://github.com/apache/beam/commit/5d9bb4595c763025a369a959e18c6dd288e72314#diff-f149847d2c06f56ea591cab8d862c960
>
> It is meant to be released as part of 2.16.0
>
> On Tue, Aug 27, 2019 at 11:41 AM Zhiheng Huang <sy...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> Looking for help to understand an internal error from beam dataflow
>> runner.
>>
>> I have a streaming pipeline that is on google dataflow runner(beam
>> version 2.11, java). Recently I added a SplittableDoFn to my pipeline to
>> continuously generate a sequence. However, after the job runs for a few
>> hours, I start to see the following exception:
>>
>> java.util.NoSuchElementException
>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.MultitransformedIterator.next(MultitransformedIterator.java:63)
>>
>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
>>
>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:308)
>>
>> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:294)
>>
>> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.getUnderlyingWindow(DataflowProcessFnRunner.java:98)
>>
>> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.placeIntoElementWindow(DataflowProcessFnRunner.java:72)
>>
>> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.processElement(DataflowProcessFnRunner.java:62)
>>
>> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:325)
>>
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>>
>> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>>
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
>>
>> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
>>
>> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
>>
>> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1269)
>>
>> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:146)
>>
>> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1008)
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> java.lang.Thread.run(Thread.java:745)
>>
>> This is 100% reproducible and I am clueless about how this stacktrace can
>> be debugged since there's no pointer to user code nor the failing
>> PTransform. This happens only if I add the SplittableDoFn, whose rough
>> structure is like this
>> <https://gist.github.com/sylvon/cbcccdcb64aeb15002721977398dc308>.
>>
>> Appreciate any pointers on what might be going wrong or how this
>> exception can be debugged.
>>
>> Thanks a lot!
>>
>>
--
Sylvon Huang
Re: NoSuchElementException after Adding SplittableDoFn
Posted by Lukasz Cwik <lc...@google.com>.
This is a known issue and was fixed with
https://github.com/apache/beam/commit/5d9bb4595c763025a369a959e18c6dd288e72314#diff-f149847d2c06f56ea591cab8d862c960
It is meant to be released as part of 2.16.0
On Tue, Aug 27, 2019 at 11:41 AM Zhiheng Huang <sy...@gmail.com>
wrote:
> Hi all,
>
> Looking for help to understand an internal error from beam dataflow runner.
>
> I have a streaming pipeline that is on google dataflow runner(beam version
> 2.11, java). Recently I added a SplittableDoFn to my pipeline to
> continuously generate a sequence. However, after the job runs for a few
> hours, I start to see the following exception:
>
> java.util.NoSuchElementException
> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.MultitransformedIterator.next(MultitransformedIterator.java:63)
>
> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
>
> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:308)
>
> org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:294)
>
> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.getUnderlyingWindow(DataflowProcessFnRunner.java:98)
>
> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.placeIntoElementWindow(DataflowProcessFnRunner.java:72)
>
> org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.processElement(DataflowProcessFnRunner.java:62)
>
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:325)
>
> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
>
> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
>
> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
>
> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
>
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
>
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1269)
>
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:146)
>
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1008)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
>
> This is 100% reproducible and I am clueless about how this stacktrace can
> be debugged since there's no pointer to user code nor the failing
> PTransform. This happens only if I add the SplittableDoFn, whose rough
> structure is like this
> <https://gist.github.com/sylvon/cbcccdcb64aeb15002721977398dc308>.
>
> Appreciate any pointers on what might be going wrong or how this exception
> can be debugged.
>
> Thanks a lot!
>
>