You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Mohil Khare <mo...@prosimo.io> on 2020/07/24 21:59:32 UTC
On DRAIN: Attempt to deliver a timer to a DoFn, but timers are not
supported in Dataflow.
Hello,
I am on java sdk 2.19 and using dataflow for beam job. I use Timers for my
stateful transformations, but recently I started seeing the following
exception on DRAINING a job. It used to work fine and not sure what changed.
java.lang.UnsupportedOperationException:
1.
1. atorg.apache.beam.runners.dataflow.worker.
StreamingSideInputDoFnRunner.onTimer (
StreamingSideInputDoFnRunner.java:86
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
)
2. atorg.apache.beam.runners.dataflow.worker.
SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
)
3. atorg.apache.beam.runners.dataflow.worker.SimpleParDoFn.access$600
(SimpleParDoFn.java:73
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
)
4. atorg.apache.beam.runners.dataflow.worker.
SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
)
5. atorg.apache.beam.runners.dataflow.worker.
SimpleParDoFn.processTimers (SimpleParDoFn.java:473
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
)
6. atorg.apache.beam.runners.dataflow.worker.
SimpleParDoFn.processTimers (SimpleParDoFn.java:353
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
)
7. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
ParDoOperation.finish (ParDoOperation.java:52
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
)
8. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
MapTaskExecutor.execute (MapTaskExecutor.java:85
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
)
9. atorg.apache.beam.runners.dataflow.worker.
StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
)
10. atorg.apache.beam.runners.dataflow.worker.
StreamingDataflowWorker.access$1100 (StreamingDataflowWorker.java:152
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
)
2. Sometime back I opened jira for an issue which is related to this
while doing DRAIN: https://issues.apache.org/jira/browse/BEAM-10053 (Looks
like no one has taken a stab on this Jira)
3. Not sure if the reason is same and due to multiple side inputs that i
use in my PTransforms.
Any help would be appreciated.
Thanks and Regards
Mohil
Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are
not supported in Dataflow.
Posted by Mohil Khare <mo...@prosimo.io>.
Hi Kenneth,
I am on beam java sdk 2.19 With enableStreamingEngine set to true and using
default machine type (n1-standard-2).
Thanks and regards
Mohil
On Wed, Jul 29, 2020 at 10:36 AM Kenneth Knowles <ke...@apache.org> wrote:
> Hi Mohil,
>
> It helps also to tell us what version of Beam you are using and some more
> details. This looks related to
> https://issues.apache.org/jira/browse/BEAM-6855 which claims to be
> resolved in 2.17.0
>
> Kenn
>
> On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare <mo...@prosimo.io> wrote:
>
>> Hello all,
>>
>> I think I found the reason for the issue. Since the exception was thrown
>> by StreamingSideInputDoFnRunner.java, I realize that I recently added side
>> input to one of my ParDo that does stateful transformations.
>> It looks like there is some issue when you add side input (My side input
>> was coming via Global window to ParDo in a Fixed Window) to stateful DoFn.
>>
>> As a work around, instead of adding side input to stateful ParDo, I
>> introduced another ParDo that enriches streaming data with side input
>> before flowing into stateful DoFn. That seems to have fixed the problem.
>>
>>
>> Thanks and regards
>> Mohil
>>
>>
>>
>> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare <mo...@prosimo.io> wrote:
>>
>>> Hello All,
>>>
>>> Any idea how to debug this and find out which stage, which DoFn or which
>>> side input is causing the problem?
>>> Do I need to override OnTimer with every DoFn to avoid this problem?
>>> I thought that some uncaught exceptions were causing this and added
>>> various checks and exception handling in all DoFn and still seeing this
>>> issue.
>>> It has been driving me nuts. And now forget DRAIN, it happens during
>>> normal functioning as well. Any help would be appreciated.
>>>
>>> java.lang.UnsupportedOperationException: Attempt to deliver a timer to a
>>> DoFn, but timers are not supported in Dataflow.
>>>
>>> 1.
>>> 1. at org.apache.beam.runners.dataflow.worker.
>>> StreamingSideInputDoFnRunner.onTimer (
>>> StreamingSideInputDoFnRunner.java:86
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>> )
>>> 2. at org.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>> )
>>> 3. at org.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>> )
>>> 4. at org.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>> )
>>> 5. at org.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>> )
>>> 6. at org.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>> )
>>> 7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>>> ParDoOperation.finish (ParDoOperation.java:52
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>> )
>>> 8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>> )
>>> 9. at org.apache.beam.runners.dataflow.worker.
>>> StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>> )
>>> 10. at org.apache.beam.runners.dataflow.worker.
>>> StreamingDataflowWorker.access$1100 (
>>> StreamingDataflowWorker.java:152
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>> )
>>> 2.
>>> 1. at org.apache.beam.runners.dataflow.worker.
>>> StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073)
>>> 2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
>>> ThreadPoolExecutor.java:1149)
>>> 3. at java.util.concurrent.ThreadPoolExecutor$Worker.run (
>>> ThreadPoolExecutor.java:624)
>>> 4. at java.lang.Thread.run (Thread.java:748)
>>>
>>> Thanks
>>> Mohil
>>>
>>>
>>> On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>
>>>> and it seems be due to TimerType User
>>>>
>>>> Thanks
>>>> Mohil
>>>>
>>>> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I was looking at source code of
>>>>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
>>>>> ,
>>>>>
>>>>> It seems the default implementation of OnTimer is to throw (Attempt
>>>>> to deliver a timer to a DoFn, but timers are not supported in Dataflow).
>>>>>
>>>>> Do you know under what circumstances, My code might be throwing this
>>>>> ? Not sure if its some issue in 2.19 which might have been fixed now with
>>>>> 2.22
>>>>>
>>>>> Thanks and Regards
>>>>> Mohil
>>>>>
>>>>>
>>>>> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>
>>>>>> Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's
>>>>>> getting flooded with this exception
>>>>>>
>>>>>> Attempt to deliver a timer to a DoFn, but timers are not supported in
>>>>>> Dataflow.
>>>>>> The changes that I did are the following:
>>>>>> 1. Read one set of logs from one of the kafka topics and create
>>>>>> KV(ID, log1)
>>>>>> 2. Read 2nd set of logs from another kafka topic and create KV(ID,
>>>>>> log2)
>>>>>> 3. Put above in session window
>>>>>> 4. CoGroupByKey both logs
>>>>>>
>>>>>> Apart from aforementioned exceptions, I am also seeing the following
>>>>>> in worker logs. What I have noticed is that it happens when the pipeline is
>>>>>> silent for a while. i.e. no new logs to be read from Kafka (Not sure if it
>>>>>> is the actual reason).
>>>>>>
>>>>>> 2020-07-24 17:06:43.532 PDT
>>>>>> Execution of work for P188 for key
>>>>>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will
>>>>>> retry locally.
>>>>>>
>>>>>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
>>>>>> 2020-07-24 17:06:53.863 PDT
>>>>>> Uncaught exception:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I am on java sdk 2.19 and using dataflow for beam job. I use Timers
>>>>>>> for my stateful transformations, but recently I started seeing the
>>>>>>> following exception on DRAINING a job. It used to work fine and not sure
>>>>>>> what changed.
>>>>>>>
>>>>>>> *java.lang.UnsupportedOperationException:*
>>>>>>>
>>>>>> *Attempt to deliver a timer to a DoFn, but timers are not supported
>>>>>> in Dataflow. *
>>>>>>
>>>>>>>
>>>>>>> 1.
>>>>>>> 1. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> StreamingSideInputDoFnRunner.onTimer (
>>>>>>> StreamingSideInputDoFnRunner.java:86
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>>>>> )
>>>>>>> 2. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>>>>> )
>>>>>>> 3. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>>>>> )
>>>>>>> 4. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>>>>> )
>>>>>>> 5. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>>>>> )
>>>>>>> 6. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>>>>> )
>>>>>>> 7. at
>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>>>>> )
>>>>>>> 8. at
>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>>>>> )
>>>>>>> 9. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> StreamingDataflowWorker.process (
>>>>>>> StreamingDataflowWorker.java:1350
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>>>>> )
>>>>>>> 10. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> StreamingDataflowWorker.access$1100 (
>>>>>>> StreamingDataflowWorker.java:152
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>>>>> )
>>>>>>> 2. Sometime back I opened jira for an issue which is related to
>>>>>>> this while doing DRAIN:
>>>>>>> https://issues.apache.org/jira/browse/BEAM-10053 (Looks like no
>>>>>>> one has taken a stab on this Jira)
>>>>>>> 3. Not sure if the reason is same and due to multiple side
>>>>>>> inputs that i use in my PTransforms.
>>>>>>>
>>>>>>>
>>>>>>> Any help would be appreciated.
>>>>>>>
>>>>>>> Thanks and Regards
>>>>>>> Mohil
>>>>>>>
>>>>>>>
Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are
not supported in Dataflow.
Posted by Mohil Khare <mo...@prosimo.io>.
Thanks a lot Luke..
Regards
Mohil
On Tue, Aug 4, 2020 at 12:01 PM Luke Cwik <lc...@google.com> wrote:
> BEAM-6855 is still open and I updated it linking to this thread that a
> user is still being impacted.
>
> On Tue, Aug 4, 2020 at 10:20 AM Mohil Khare <mo...@prosimo.io> wrote:
>
>> yeah .. looks like a bug still exists.
>>
>> So how does this work ? Shall I open a new Jira ?
>>
>> Thanks and regards
>> Mohil
>>
>> On Thu, Jul 30, 2020 at 10:39 PM Reuven Lax <re...@google.com> wrote:
>>
>>> I believe that the person trying to fix BEAM-6855 was unable to
>>> reproduce it in test, and therefore assumed that the bug was fixed. However
>>> it appears that the bug still exists.
>>>
>>> On Wed, Jul 29, 2020 at 10:36 AM Kenneth Knowles <ke...@apache.org>
>>> wrote:
>>>
>>>> Hi Mohil,
>>>>
>>>> It helps also to tell us what version of Beam you are using and some
>>>> more details. This looks related to
>>>> https://issues.apache.org/jira/browse/BEAM-6855 which claims to be
>>>> resolved in 2.17.0
>>>>
>>>> Kenn
>>>>
>>>> On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> I think I found the reason for the issue. Since the exception was
>>>>> thrown by StreamingSideInputDoFnRunner.java, I realize that I recently
>>>>> added side input to one of my ParDo that does stateful transformations.
>>>>> It looks like there is some issue when you add side input (My side
>>>>> input was coming via Global window to ParDo in a Fixed Window) to stateful
>>>>> DoFn.
>>>>>
>>>>> As a work around, instead of adding side input to stateful ParDo, I
>>>>> introduced another ParDo that enriches streaming data with side input
>>>>> before flowing into stateful DoFn. That seems to have fixed the problem.
>>>>>
>>>>>
>>>>> Thanks and regards
>>>>> Mohil
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>
>>>>>> Hello All,
>>>>>>
>>>>>> Any idea how to debug this and find out which stage, which DoFn or
>>>>>> which side input is causing the problem?
>>>>>> Do I need to override OnTimer with every DoFn to avoid this problem?
>>>>>> I thought that some uncaught exceptions were causing this and added
>>>>>> various checks and exception handling in all DoFn and still seeing this
>>>>>> issue.
>>>>>> It has been driving me nuts. And now forget DRAIN, it happens during
>>>>>> normal functioning as well. Any help would be appreciated.
>>>>>>
>>>>>> java.lang.UnsupportedOperationException: Attempt to deliver a timer
>>>>>> to a DoFn, but timers are not supported in Dataflow.
>>>>>>
>>>>>> 1.
>>>>>> 1. at org.apache.beam.runners.dataflow.worker.
>>>>>> StreamingSideInputDoFnRunner.onTimer (
>>>>>> StreamingSideInputDoFnRunner.java:86
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>>>> )
>>>>>> 2. at org.apache.beam.runners.dataflow.worker.
>>>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>>>> )
>>>>>> 3. at org.apache.beam.runners.dataflow.worker.
>>>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>>>> )
>>>>>> 4. at org.apache.beam.runners.dataflow.worker.
>>>>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>>>> )
>>>>>> 5. at org.apache.beam.runners.dataflow.worker.
>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>>>> )
>>>>>> 6. at org.apache.beam.runners.dataflow.worker.
>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>>>> )
>>>>>> 7. at
>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>>>> )
>>>>>> 8. at
>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>>>> )
>>>>>> 9. at org.apache.beam.runners.dataflow.worker.
>>>>>> StreamingDataflowWorker.process (
>>>>>> StreamingDataflowWorker.java:1350
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>>>> )
>>>>>> 10. at org.apache.beam.runners.dataflow.worker.
>>>>>> StreamingDataflowWorker.access$1100 (
>>>>>> StreamingDataflowWorker.java:152
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>>>> )
>>>>>> 2.
>>>>>> 1. at org.apache.beam.runners.dataflow.worker.
>>>>>> StreamingDataflowWorker$7.run (
>>>>>> StreamingDataflowWorker.java:1073)
>>>>>> 2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
>>>>>> ThreadPoolExecutor.java:1149)
>>>>>> 3. at java.util.concurrent.ThreadPoolExecutor$Worker.run (
>>>>>> ThreadPoolExecutor.java:624)
>>>>>> 4. at java.lang.Thread.run (Thread.java:748)
>>>>>>
>>>>>> Thanks
>>>>>> Mohil
>>>>>>
>>>>>>
>>>>>> On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>>
>>>>>>> and it seems be due to TimerType User
>>>>>>>
>>>>>>> Thanks
>>>>>>> Mohil
>>>>>>>
>>>>>>> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I was looking at source code of
>>>>>>>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
>>>>>>>> ,
>>>>>>>>
>>>>>>>> It seems the default implementation of OnTimer is to throw (Attempt
>>>>>>>> to deliver a timer to a DoFn, but timers are not supported in Dataflow).
>>>>>>>>
>>>>>>>> Do you know under what circumstances, My code might be throwing
>>>>>>>> this ? Not sure if its some issue in 2.19 which might have been fixed now
>>>>>>>> with 2.22
>>>>>>>>
>>>>>>>> Thanks and Regards
>>>>>>>> Mohil
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Actually NOT JUST DRAIN, seeing it during regular RUN as well.
>>>>>>>>> It's getting flooded with this exception
>>>>>>>>>
>>>>>>>>> Attempt to deliver a timer to a DoFn, but timers are not supported
>>>>>>>>> in Dataflow.
>>>>>>>>> The changes that I did are the following:
>>>>>>>>> 1. Read one set of logs from one of the kafka topics and create
>>>>>>>>> KV(ID, log1)
>>>>>>>>> 2. Read 2nd set of logs from another kafka topic and create KV(ID,
>>>>>>>>> log2)
>>>>>>>>> 3. Put above in session window
>>>>>>>>> 4. CoGroupByKey both logs
>>>>>>>>>
>>>>>>>>> Apart from aforementioned exceptions, I am also seeing the
>>>>>>>>> following in worker logs. What I have noticed is that it happens when the
>>>>>>>>> pipeline is silent for a while. i.e. no new logs to be read from Kafka (Not
>>>>>>>>> sure if it is the actual reason).
>>>>>>>>>
>>>>>>>>> 2020-07-24 17:06:43.532 PDT
>>>>>>>>> Execution of work for P188 for key
>>>>>>>>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed.
>>>>>>>>> Will retry locally.
>>>>>>>>>
>>>>>>>>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
>>>>>>>>> 2020-07-24 17:06:53.863 PDT
>>>>>>>>> Uncaught exception:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I am on java sdk 2.19 and using dataflow for beam job. I use
>>>>>>>>>> Timers for my stateful transformations, but recently I started seeing the
>>>>>>>>>> following exception on DRAINING a job. It used to work fine and not sure
>>>>>>>>>> what changed.
>>>>>>>>>>
>>>>>>>>>> *java.lang.UnsupportedOperationException:*
>>>>>>>>>>
>>>>>>>>> *Attempt to deliver a timer to a DoFn, but timers are not
>>>>>>>>> supported in Dataflow. *
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1.
>>>>>>>>>> 1. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>>> StreamingSideInputDoFnRunner.onTimer (
>>>>>>>>>> StreamingSideInputDoFnRunner.java:86
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>>>>>>>> )
>>>>>>>>>> 2. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>>>>>>>> )
>>>>>>>>>> 3. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>>>>>>>> )
>>>>>>>>>> 4. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>>> SimpleParDoFn$TimerType$1.processTimer (
>>>>>>>>>> SimpleParDoFn.java:444
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>>>>>>>> )
>>>>>>>>>> 5. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>>>>>>>> )
>>>>>>>>>> 6. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>>>>>>>> )
>>>>>>>>>> 7. at
>>>>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>>>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>>>>>>>> )
>>>>>>>>>> 8. at
>>>>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>>>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>>>>>>>> )
>>>>>>>>>> 9. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>>> StreamingDataflowWorker.process (
>>>>>>>>>> StreamingDataflowWorker.java:1350
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>>>>>>>> )
>>>>>>>>>> 10. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>>> StreamingDataflowWorker.access$1100 (
>>>>>>>>>> StreamingDataflowWorker.java:152
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>>>>>>>> )
>>>>>>>>>> 2. Sometime back I opened jira for an issue which is related
>>>>>>>>>> to this while doing DRAIN:
>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-10053 (Looks like
>>>>>>>>>> no one has taken a stab on this Jira)
>>>>>>>>>> 3. Not sure if the reason is same and due to multiple side
>>>>>>>>>> inputs that i use in my PTransforms.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Any help would be appreciated.
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards
>>>>>>>>>> Mohil
>>>>>>>>>>
>>>>>>>>>>
Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are
not supported in Dataflow.
Posted by Luke Cwik <lc...@google.com>.
BEAM-6855 is still open and I updated it linking to this thread that a user
is still being impacted.
On Tue, Aug 4, 2020 at 10:20 AM Mohil Khare <mo...@prosimo.io> wrote:
> yeah .. looks like a bug still exists.
>
> So how does this work ? Shall I open a new Jira ?
>
> Thanks and regards
> Mohil
>
> On Thu, Jul 30, 2020 at 10:39 PM Reuven Lax <re...@google.com> wrote:
>
>> I believe that the person trying to fix BEAM-6855 was unable to reproduce
>> it in test, and therefore assumed that the bug was fixed. However it
>> appears that the bug still exists.
>>
>> On Wed, Jul 29, 2020 at 10:36 AM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> Hi Mohil,
>>>
>>> It helps also to tell us what version of Beam you are using and some
>>> more details. This looks related to
>>> https://issues.apache.org/jira/browse/BEAM-6855 which claims to be
>>> resolved in 2.17.0
>>>
>>> Kenn
>>>
>>> On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>
>>>> Hello all,
>>>>
>>>> I think I found the reason for the issue. Since the exception was
>>>> thrown by StreamingSideInputDoFnRunner.java, I realize that I recently
>>>> added side input to one of my ParDo that does stateful transformations.
>>>> It looks like there is some issue when you add side input (My side
>>>> input was coming via Global window to ParDo in a Fixed Window) to stateful
>>>> DoFn.
>>>>
>>>> As a work around, instead of adding side input to stateful ParDo, I
>>>> introduced another ParDo that enriches streaming data with side input
>>>> before flowing into stateful DoFn. That seems to have fixed the problem.
>>>>
>>>>
>>>> Thanks and regards
>>>> Mohil
>>>>
>>>>
>>>>
>>>> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare <mo...@prosimo.io> wrote:
>>>>
>>>>> Hello All,
>>>>>
>>>>> Any idea how to debug this and find out which stage, which DoFn or
>>>>> which side input is causing the problem?
>>>>> Do I need to override OnTimer with every DoFn to avoid this problem?
>>>>> I thought that some uncaught exceptions were causing this and added
>>>>> various checks and exception handling in all DoFn and still seeing this
>>>>> issue.
>>>>> It has been driving me nuts. And now forget DRAIN, it happens during
>>>>> normal functioning as well. Any help would be appreciated.
>>>>>
>>>>> java.lang.UnsupportedOperationException: Attempt to deliver a timer to
>>>>> a DoFn, but timers are not supported in Dataflow.
>>>>>
>>>>> 1.
>>>>> 1. at org.apache.beam.runners.dataflow.worker.
>>>>> StreamingSideInputDoFnRunner.onTimer (
>>>>> StreamingSideInputDoFnRunner.java:86
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>>> )
>>>>> 2. at org.apache.beam.runners.dataflow.worker.
>>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>>> )
>>>>> 3. at org.apache.beam.runners.dataflow.worker.
>>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>>> )
>>>>> 4. at org.apache.beam.runners.dataflow.worker.
>>>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>>> )
>>>>> 5. at org.apache.beam.runners.dataflow.worker.
>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>>> )
>>>>> 6. at org.apache.beam.runners.dataflow.worker.
>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>>> )
>>>>> 7. at
>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>>> )
>>>>> 8. at
>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>>> )
>>>>> 9. at org.apache.beam.runners.dataflow.worker.
>>>>> StreamingDataflowWorker.process (
>>>>> StreamingDataflowWorker.java:1350
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>>> )
>>>>> 10. at org.apache.beam.runners.dataflow.worker.
>>>>> StreamingDataflowWorker.access$1100 (
>>>>> StreamingDataflowWorker.java:152
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>>> )
>>>>> 2.
>>>>> 1. at org.apache.beam.runners.dataflow.worker.
>>>>> StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073
>>>>> )
>>>>> 2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
>>>>> ThreadPoolExecutor.java:1149)
>>>>> 3. at java.util.concurrent.ThreadPoolExecutor$Worker.run (
>>>>> ThreadPoolExecutor.java:624)
>>>>> 4. at java.lang.Thread.run (Thread.java:748)
>>>>>
>>>>> Thanks
>>>>> Mohil
>>>>>
>>>>>
>>>>> On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>
>>>>>> and it seems be due to TimerType User
>>>>>>
>>>>>> Thanks
>>>>>> Mohil
>>>>>>
>>>>>> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I was looking at source code of
>>>>>>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
>>>>>>> ,
>>>>>>>
>>>>>>> It seems the default implementation of OnTimer is to throw (Attempt
>>>>>>> to deliver a timer to a DoFn, but timers are not supported in Dataflow).
>>>>>>>
>>>>>>> Do you know under what circumstances, My code might be throwing this
>>>>>>> ? Not sure if its some issue in 2.19 which might have been fixed now with
>>>>>>> 2.22
>>>>>>>
>>>>>>> Thanks and Regards
>>>>>>> Mohil
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Actually NOT JUST DRAIN, seeing it during regular RUN as well.
>>>>>>>> It's getting flooded with this exception
>>>>>>>>
>>>>>>>> Attempt to deliver a timer to a DoFn, but timers are not supported
>>>>>>>> in Dataflow.
>>>>>>>> The changes that I did are the following:
>>>>>>>> 1. Read one set of logs from one of the kafka topics and create
>>>>>>>> KV(ID, log1)
>>>>>>>> 2. Read 2nd set of logs from another kafka topic and create KV(ID,
>>>>>>>> log2)
>>>>>>>> 3. Put above in session window
>>>>>>>> 4. CoGroupByKey both logs
>>>>>>>>
>>>>>>>> Apart from aforementioned exceptions, I am also seeing the
>>>>>>>> following in worker logs. What I have noticed is that it happens when the
>>>>>>>> pipeline is silent for a while. i.e. no new logs to be read from Kafka (Not
>>>>>>>> sure if it is the actual reason).
>>>>>>>>
>>>>>>>> 2020-07-24 17:06:43.532 PDT
>>>>>>>> Execution of work for P188 for key
>>>>>>>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will
>>>>>>>> retry locally.
>>>>>>>>
>>>>>>>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
>>>>>>>> 2020-07-24 17:06:53.863 PDT
>>>>>>>> Uncaught exception:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I am on java sdk 2.19 and using dataflow for beam job. I use
>>>>>>>>> Timers for my stateful transformations, but recently I started seeing the
>>>>>>>>> following exception on DRAINING a job. It used to work fine and not sure
>>>>>>>>> what changed.
>>>>>>>>>
>>>>>>>>> *java.lang.UnsupportedOperationException:*
>>>>>>>>>
>>>>>>>> *Attempt to deliver a timer to a DoFn, but timers are not supported
>>>>>>>> in Dataflow. *
>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1.
>>>>>>>>> 1. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>> StreamingSideInputDoFnRunner.onTimer (
>>>>>>>>> StreamingSideInputDoFnRunner.java:86
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>>>>>>> )
>>>>>>>>> 2. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>>>>>>> )
>>>>>>>>> 3. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>>>>>>> )
>>>>>>>>> 4. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>> SimpleParDoFn$TimerType$1.processTimer (
>>>>>>>>> SimpleParDoFn.java:444
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>>>>>>> )
>>>>>>>>> 5. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>>>>>>> )
>>>>>>>>> 6. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>>>>>>> )
>>>>>>>>> 7. at
>>>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>>>>>>> )
>>>>>>>>> 8. at
>>>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>>>>>>> )
>>>>>>>>> 9. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>> StreamingDataflowWorker.process (
>>>>>>>>> StreamingDataflowWorker.java:1350
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>>>>>>> )
>>>>>>>>> 10. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>>> StreamingDataflowWorker.access$1100 (
>>>>>>>>> StreamingDataflowWorker.java:152
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>>>>>>> )
>>>>>>>>> 2. Sometime back I opened jira for an issue which is related
>>>>>>>>> to this while doing DRAIN:
>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-10053 (Looks like
>>>>>>>>> no one has taken a stab on this Jira)
>>>>>>>>> 3. Not sure if the reason is same and due to multiple side
>>>>>>>>> inputs that i use in my PTransforms.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any help would be appreciated.
>>>>>>>>>
>>>>>>>>> Thanks and Regards
>>>>>>>>> Mohil
>>>>>>>>>
>>>>>>>>>
Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are
not supported in Dataflow.
Posted by Mohil Khare <mo...@prosimo.io>.
yeah .. looks like a bug still exists.
So how does this work ? Shall I open a new Jira ?
Thanks and regards
Mohil
On Thu, Jul 30, 2020 at 10:39 PM Reuven Lax <re...@google.com> wrote:
> I believe that the person trying to fix BEAM-6855 was unable to reproduce
> it in test, and therefore assumed that the bug was fixed. However it
> appears that the bug still exists.
>
> On Wed, Jul 29, 2020 at 10:36 AM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Hi Mohil,
>>
>> It helps also to tell us what version of Beam you are using and some more
>> details. This looks related to
>> https://issues.apache.org/jira/browse/BEAM-6855 which claims to be
>> resolved in 2.17.0
>>
>> Kenn
>>
>> On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare <mo...@prosimo.io> wrote:
>>
>>> Hello all,
>>>
>>> I think I found the reason for the issue. Since the exception was
>>> thrown by StreamingSideInputDoFnRunner.java, I realize that I recently
>>> added side input to one of my ParDo that does stateful transformations.
>>> It looks like there is some issue when you add side input (My side input
>>> was coming via Global window to ParDo in a Fixed Window) to stateful DoFn.
>>>
>>> As a work around, instead of adding side input to stateful ParDo, I
>>> introduced another ParDo that enriches streaming data with side input
>>> before flowing into stateful DoFn. That seems to have fixed the problem.
>>>
>>>
>>> Thanks and regards
>>> Mohil
>>>
>>>
>>>
>>> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare <mo...@prosimo.io> wrote:
>>>
>>>> Hello All,
>>>>
>>>> Any idea how to debug this and find out which stage, which DoFn or
>>>> which side input is causing the problem?
>>>> Do I need to override OnTimer with every DoFn to avoid this problem?
>>>> I thought that some uncaught exceptions were causing this and added
>>>> various checks and exception handling in all DoFn and still seeing this
>>>> issue.
>>>> It has been driving me nuts. And now forget DRAIN, it happens during
>>>> normal functioning as well. Any help would be appreciated.
>>>>
>>>> java.lang.UnsupportedOperationException: Attempt to deliver a timer to
>>>> a DoFn, but timers are not supported in Dataflow.
>>>>
>>>> 1.
>>>> 1. at org.apache.beam.runners.dataflow.worker.
>>>> StreamingSideInputDoFnRunner.onTimer (
>>>> StreamingSideInputDoFnRunner.java:86
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>> )
>>>> 2. at org.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>> )
>>>> 3. at org.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>> )
>>>> 4. at org.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>> )
>>>> 5. at org.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>> )
>>>> 6. at org.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>> )
>>>> 7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>> )
>>>> 8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>> )
>>>> 9. at org.apache.beam.runners.dataflow.worker.
>>>> StreamingDataflowWorker.process (
>>>> StreamingDataflowWorker.java:1350
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>> )
>>>> 10. at org.apache.beam.runners.dataflow.worker.
>>>> StreamingDataflowWorker.access$1100 (
>>>> StreamingDataflowWorker.java:152
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>> )
>>>> 2.
>>>> 1. at org.apache.beam.runners.dataflow.worker.
>>>> StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073)
>>>> 2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
>>>> ThreadPoolExecutor.java:1149)
>>>> 3. at java.util.concurrent.ThreadPoolExecutor$Worker.run (
>>>> ThreadPoolExecutor.java:624)
>>>> 4. at java.lang.Thread.run (Thread.java:748)
>>>>
>>>> Thanks
>>>> Mohil
>>>>
>>>>
>>>> On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>
>>>>> and it seems be due to TimerType User
>>>>>
>>>>> Thanks
>>>>> Mohil
>>>>>
>>>>> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I was looking at source code of
>>>>>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
>>>>>> ,
>>>>>>
>>>>>> It seems the default implementation of OnTimer is to throw (Attempt
>>>>>> to deliver a timer to a DoFn, but timers are not supported in Dataflow).
>>>>>>
>>>>>> Do you know under what circumstances, My code might be throwing this
>>>>>> ? Not sure if its some issue in 2.19 which might have been fixed now with
>>>>>> 2.22
>>>>>>
>>>>>> Thanks and Regards
>>>>>> Mohil
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>>
>>>>>>> Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's
>>>>>>> getting flooded with this exception
>>>>>>>
>>>>>>> Attempt to deliver a timer to a DoFn, but timers are not supported
>>>>>>> in Dataflow.
>>>>>>> The changes that I did are the following:
>>>>>>> 1. Read one set of logs from one of the kafka topics and create
>>>>>>> KV(ID, log1)
>>>>>>> 2. Read 2nd set of logs from another kafka topic and create KV(ID,
>>>>>>> log2)
>>>>>>> 3. Put above in session window
>>>>>>> 4. CoGroupByKey both logs
>>>>>>>
>>>>>>> Apart from aforementioned exceptions, I am also seeing the following
>>>>>>> in worker logs. What I have noticed is that it happens when the pipeline is
>>>>>>> silent for a while. i.e. no new logs to be read from Kafka (Not sure if it
>>>>>>> is the actual reason).
>>>>>>>
>>>>>>> 2020-07-24 17:06:43.532 PDT
>>>>>>> Execution of work for P188 for key
>>>>>>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will
>>>>>>> retry locally.
>>>>>>>
>>>>>>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
>>>>>>> 2020-07-24 17:06:53.863 PDT
>>>>>>> Uncaught exception:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I am on java sdk 2.19 and using dataflow for beam job. I use Timers
>>>>>>>> for my stateful transformations, but recently I started seeing the
>>>>>>>> following exception on DRAINING a job. It used to work fine and not sure
>>>>>>>> what changed.
>>>>>>>>
>>>>>>>> *java.lang.UnsupportedOperationException:*
>>>>>>>>
>>>>>>> *Attempt to deliver a timer to a DoFn, but timers are not supported
>>>>>>> in Dataflow. *
>>>>>>>
>>>>>>>>
>>>>>>>> 1.
>>>>>>>> 1. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>> StreamingSideInputDoFnRunner.onTimer (
>>>>>>>> StreamingSideInputDoFnRunner.java:86
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>>>>>> )
>>>>>>>> 2. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>>>>>> )
>>>>>>>> 3. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>>>>>> )
>>>>>>>> 4. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>> SimpleParDoFn$TimerType$1.processTimer (
>>>>>>>> SimpleParDoFn.java:444
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>>>>>> )
>>>>>>>> 5. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>>>>>> )
>>>>>>>> 6. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>>>>>> )
>>>>>>>> 7. at
>>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>>>>>> )
>>>>>>>> 8. at
>>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>>>>>> )
>>>>>>>> 9. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>> StreamingDataflowWorker.process (
>>>>>>>> StreamingDataflowWorker.java:1350
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>>>>>> )
>>>>>>>> 10. atorg.apache.beam.runners.dataflow.worker.
>>>>>>>> StreamingDataflowWorker.access$1100 (
>>>>>>>> StreamingDataflowWorker.java:152
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>>>>>> )
>>>>>>>> 2. Sometime back I opened jira for an issue which is related to
>>>>>>>> this while doing DRAIN:
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-10053 (Looks like no
>>>>>>>> one has taken a stab on this Jira)
>>>>>>>> 3. Not sure if the reason is same and due to multiple side
>>>>>>>> inputs that i use in my PTransforms.
>>>>>>>>
>>>>>>>>
>>>>>>>> Any help would be appreciated.
>>>>>>>>
>>>>>>>> Thanks and Regards
>>>>>>>> Mohil
>>>>>>>>
>>>>>>>>
Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are
not supported in Dataflow.
Posted by Reuven Lax <re...@google.com>.
I believe that the person trying to fix BEAM-6855 was unable to reproduce
it in test, and therefore assumed that the bug was fixed. However it
appears that the bug still exists.
On Wed, Jul 29, 2020 at 10:36 AM Kenneth Knowles <ke...@apache.org> wrote:
> Hi Mohil,
>
> It helps also to tell us what version of Beam you are using and some more
> details. This looks related to
> https://issues.apache.org/jira/browse/BEAM-6855 which claims to be
> resolved in 2.17.0
>
> Kenn
>
> On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare <mo...@prosimo.io> wrote:
>
>> Hello all,
>>
>> I think I found the reason for the issue. Since the exception was thrown
>> by StreamingSideInputDoFnRunner.java, I realize that I recently added side
>> input to one of my ParDo that does stateful transformations.
>> It looks like there is some issue when you add side input (My side input
>> was coming via Global window to ParDo in a Fixed Window) to stateful DoFn.
>>
>> As a work around, instead of adding side input to stateful ParDo, I
>> introduced another ParDo that enriches streaming data with side input
>> before flowing into stateful DoFn. That seems to have fixed the problem.
>>
>>
>> Thanks and regards
>> Mohil
>>
>>
>>
>> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare <mo...@prosimo.io> wrote:
>>
>>> Hello All,
>>>
>>> Any idea how to debug this and find out which stage, which DoFn or which
>>> side input is causing the problem?
>>> Do I need to override OnTimer with every DoFn to avoid this problem?
>>> I thought that some uncaught exceptions were causing this and added
>>> various checks and exception handling in all DoFn and still seeing this
>>> issue.
>>> It has been driving me nuts. And now forget DRAIN, it happens during
>>> normal functioning as well. Any help would be appreciated.
>>>
>>> java.lang.UnsupportedOperationException: Attempt to deliver a timer to a
>>> DoFn, but timers are not supported in Dataflow.
>>>
>>> 1.
>>> 1. at org.apache.beam.runners.dataflow.worker.
>>> StreamingSideInputDoFnRunner.onTimer (
>>> StreamingSideInputDoFnRunner.java:86
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>> )
>>> 2. at org.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>> )
>>> 3. at org.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>> )
>>> 4. at org.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>> )
>>> 5. at org.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>> )
>>> 6. at org.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>> )
>>> 7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>>> ParDoOperation.finish (ParDoOperation.java:52
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>> )
>>> 8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>> )
>>> 9. at org.apache.beam.runners.dataflow.worker.
>>> StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>> )
>>> 10. at org.apache.beam.runners.dataflow.worker.
>>> StreamingDataflowWorker.access$1100 (
>>> StreamingDataflowWorker.java:152
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>> )
>>> 2.
>>> 1. at org.apache.beam.runners.dataflow.worker.
>>> StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073)
>>> 2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
>>> ThreadPoolExecutor.java:1149)
>>> 3. at java.util.concurrent.ThreadPoolExecutor$Worker.run (
>>> ThreadPoolExecutor.java:624)
>>> 4. at java.lang.Thread.run (Thread.java:748)
>>>
>>> Thanks
>>> Mohil
>>>
>>>
>>> On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>
>>>> and it seems be due to TimerType User
>>>>
>>>> Thanks
>>>> Mohil
>>>>
>>>> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I was looking at source code of
>>>>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
>>>>> ,
>>>>>
>>>>> It seems the default implementation of OnTimer is to throw (Attempt
>>>>> to deliver a timer to a DoFn, but timers are not supported in Dataflow).
>>>>>
>>>>> Do you know under what circumstances, My code might be throwing this
>>>>> ? Not sure if its some issue in 2.19 which might have been fixed now with
>>>>> 2.22
>>>>>
>>>>> Thanks and Regards
>>>>> Mohil
>>>>>
>>>>>
>>>>> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>
>>>>>> Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's
>>>>>> getting flooded with this exception
>>>>>>
>>>>>> Attempt to deliver a timer to a DoFn, but timers are not supported in
>>>>>> Dataflow.
>>>>>> The changes that I did are the following:
>>>>>> 1. Read one set of logs from one of the kafka topics and create
>>>>>> KV(ID, log1)
>>>>>> 2. Read 2nd set of logs from another kafka topic and create KV(ID,
>>>>>> log2)
>>>>>> 3. Put above in session window
>>>>>> 4. CoGroupByKey both logs
>>>>>>
>>>>>> Apart from aforementioned exceptions, I am also seeing the following
>>>>>> in worker logs. What I have noticed is that it happens when the pipeline is
>>>>>> silent for a while. i.e. no new logs to be read from Kafka (Not sure if it
>>>>>> is the actual reason).
>>>>>>
>>>>>> 2020-07-24 17:06:43.532 PDT
>>>>>> Execution of work for P188 for key
>>>>>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will
>>>>>> retry locally.
>>>>>>
>>>>>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
>>>>>> 2020-07-24 17:06:53.863 PDT
>>>>>> Uncaught exception:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I am on java sdk 2.19 and using dataflow for beam job. I use Timers
>>>>>>> for my stateful transformations, but recently I started seeing the
>>>>>>> following exception on DRAINING a job. It used to work fine and not sure
>>>>>>> what changed.
>>>>>>>
>>>>>>> *java.lang.UnsupportedOperationException:*
>>>>>>>
>>>>>> *Attempt to deliver a timer to a DoFn, but timers are not supported
>>>>>> in Dataflow. *
>>>>>>
>>>>>>>
>>>>>>> 1.
>>>>>>> 1. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> StreamingSideInputDoFnRunner.onTimer (
>>>>>>> StreamingSideInputDoFnRunner.java:86
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>>>>> )
>>>>>>> 2. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>>>>> )
>>>>>>> 3. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>>>>> )
>>>>>>> 4. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>>>>> )
>>>>>>> 5. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>>>>> )
>>>>>>> 6. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>>>>> )
>>>>>>> 7. at
>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>>>>> )
>>>>>>> 8. at
>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>>>>> )
>>>>>>> 9. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> StreamingDataflowWorker.process (
>>>>>>> StreamingDataflowWorker.java:1350
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>>>>> )
>>>>>>> 10. atorg.apache.beam.runners.dataflow.worker.
>>>>>>> StreamingDataflowWorker.access$1100 (
>>>>>>> StreamingDataflowWorker.java:152
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>>>>> )
>>>>>>> 2. Sometime back I opened jira for an issue which is related to
>>>>>>> this while doing DRAIN:
>>>>>>> https://issues.apache.org/jira/browse/BEAM-10053 (Looks like no
>>>>>>> one has taken a stab on this Jira)
>>>>>>> 3. Not sure if the reason is same and due to multiple side
>>>>>>> inputs that i use in my PTransforms.
>>>>>>>
>>>>>>>
>>>>>>> Any help would be appreciated.
>>>>>>>
>>>>>>> Thanks and Regards
>>>>>>> Mohil
>>>>>>>
>>>>>>>
Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are
not supported in Dataflow.
Posted by Kenneth Knowles <ke...@apache.org>.
Hi Mohil,
It helps also to tell us what version of Beam you are using and some more
details. This looks related to
https://issues.apache.org/jira/browse/BEAM-6855 which claims to be resolved
in 2.17.0
Kenn
On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare <mo...@prosimo.io> wrote:
> Hello all,
>
> I think I found the reason for the issue. Since the exception was thrown
> by StreamingSideInputDoFnRunner.java, I realize that I recently added side
> input to one of my ParDo that does stateful transformations.
> It looks like there is some issue when you add side input (My side input
> was coming via Global window to ParDo in a Fixed Window) to stateful DoFn.
>
> As a work around, instead of adding side input to stateful ParDo, I
> introduced another ParDo that enriches streaming data with side input
> before flowing into stateful DoFn. That seems to have fixed the problem.
>
>
> Thanks and regards
> Mohil
>
>
>
> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare <mo...@prosimo.io> wrote:
>
>> Hello All,
>>
>> Any idea how to debug this and find out which stage, which DoFn or which
>> side input is causing the problem?
>> Do I need to override OnTimer with every DoFn to avoid this problem?
>> I thought that some uncaught exceptions were causing this and added
>> various checks and exception handling in all DoFn and still seeing this
>> issue.
>> It has been driving me nuts. And now forget DRAIN, it happens during
>> normal functioning as well. Any help would be appreciated.
>>
>> java.lang.UnsupportedOperationException: Attempt to deliver a timer to a
>> DoFn, but timers are not supported in Dataflow.
>>
>> 1.
>> 1. at org.apache.beam.runners.dataflow.worker.
>> StreamingSideInputDoFnRunner.onTimer (
>> StreamingSideInputDoFnRunner.java:86
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>> )
>> 2. at org.apache.beam.runners.dataflow.worker.
>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>> )
>> 3. at org.apache.beam.runners.dataflow.worker.
>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>> )
>> 4. at org.apache.beam.runners.dataflow.worker.
>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>> )
>> 5. at org.apache.beam.runners.dataflow.worker.
>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>> )
>> 6. at org.apache.beam.runners.dataflow.worker.
>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>> )
>> 7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>> ParDoOperation.finish (ParDoOperation.java:52
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>> )
>> 8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>> )
>> 9. at org.apache.beam.runners.dataflow.worker.
>> StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>> )
>> 10. at org.apache.beam.runners.dataflow.worker.
>> StreamingDataflowWorker.access$1100 (
>> StreamingDataflowWorker.java:152
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>> )
>> 2.
>> 1. at org.apache.beam.runners.dataflow.worker.
>> StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073)
>> 2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
>> ThreadPoolExecutor.java:1149)
>> 3. at java.util.concurrent.ThreadPoolExecutor$Worker.run (
>> ThreadPoolExecutor.java:624)
>> 4. at java.lang.Thread.run (Thread.java:748)
>>
>> Thanks
>> Mohil
>>
>>
>> On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare <mo...@prosimo.io> wrote:
>>
>>> and it seems be due to TimerType User
>>>
>>> Thanks
>>> Mohil
>>>
>>> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>
>>>> Hello,
>>>>
>>>> I was looking at source code of
>>>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
>>>> ,
>>>>
>>>> It seems the default implementation of OnTimer is to throw (Attempt to
>>>> deliver a timer to a DoFn, but timers are not supported in Dataflow).
>>>>
>>>> Do you know under what circumstances, My code might be throwing this ?
>>>> Not sure if its some issue in 2.19 which might have been fixed now with
>>>> 2.22
>>>>
>>>> Thanks and Regards
>>>> Mohil
>>>>
>>>>
>>>> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>
>>>>> Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's
>>>>> getting flooded with this exception
>>>>>
>>>>> Attempt to deliver a timer to a DoFn, but timers are not supported in
>>>>> Dataflow.
>>>>> The changes that I did are the following:
>>>>> 1. Read one set of logs from one of the kafka topics and create KV(ID,
>>>>> log1)
>>>>> 2. Read 2nd set of logs from another kafka topic and create KV(ID,
>>>>> log2)
>>>>> 3. Put above in session window
>>>>> 4. CoGroupByKey both logs
>>>>>
>>>>> Apart from aforementioned exceptions, I am also seeing the following
>>>>> in worker logs. What I have noticed is that it happens when the pipeline is
>>>>> silent for a while. i.e. no new logs to be read from Kafka (Not sure if it
>>>>> is the actual reason).
>>>>>
>>>>> 2020-07-24 17:06:43.532 PDT
>>>>> Execution of work for P188 for key
>>>>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will
>>>>> retry locally.
>>>>>
>>>>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
>>>>> 2020-07-24 17:06:53.863 PDT
>>>>> Uncaught exception:
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am on java sdk 2.19 and using dataflow for beam job. I use Timers
>>>>>> for my stateful transformations, but recently I started seeing the
>>>>>> following exception on DRAINING a job. It used to work fine and not sure
>>>>>> what changed.
>>>>>>
>>>>>> *java.lang.UnsupportedOperationException:*
>>>>>>
>>>>> *Attempt to deliver a timer to a DoFn, but timers are not supported in
>>>>> Dataflow. *
>>>>>
>>>>>>
>>>>>> 1.
>>>>>> 1. atorg.apache.beam.runners.dataflow.worker.
>>>>>> StreamingSideInputDoFnRunner.onTimer (
>>>>>> StreamingSideInputDoFnRunner.java:86
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>>>> )
>>>>>> 2. atorg.apache.beam.runners.dataflow.worker.
>>>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>>>> )
>>>>>> 3. atorg.apache.beam.runners.dataflow.worker.
>>>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>>>> )
>>>>>> 4. atorg.apache.beam.runners.dataflow.worker.
>>>>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>>>> )
>>>>>> 5. atorg.apache.beam.runners.dataflow.worker.
>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>>>> )
>>>>>> 6. atorg.apache.beam.runners.dataflow.worker.
>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>>>> )
>>>>>> 7. at
>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>>>> )
>>>>>> 8. at
>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>>>> )
>>>>>> 9. atorg.apache.beam.runners.dataflow.worker.
>>>>>> StreamingDataflowWorker.process (
>>>>>> StreamingDataflowWorker.java:1350
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>>>> )
>>>>>> 10. atorg.apache.beam.runners.dataflow.worker.
>>>>>> StreamingDataflowWorker.access$1100 (
>>>>>> StreamingDataflowWorker.java:152
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>>>> )
>>>>>> 2. Sometime back I opened jira for an issue which is related to
>>>>>> this while doing DRAIN:
>>>>>> https://issues.apache.org/jira/browse/BEAM-10053 (Looks like no
>>>>>> one has taken a stab on this Jira)
>>>>>> 3. Not sure if the reason is same and due to multiple side inputs
>>>>>> that i use in my PTransforms.
>>>>>>
>>>>>>
>>>>>> Any help would be appreciated.
>>>>>>
>>>>>> Thanks and Regards
>>>>>> Mohil
>>>>>>
>>>>>>
Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are
not supported in Dataflow.
Posted by Mohil Khare <mo...@prosimo.io>.
Hello all,
I think I found the reason for the issue. Since the exception was thrown
by StreamingSideInputDoFnRunner.java, I realize that I recently added side
input to one of my ParDo that does stateful transformations.
It looks like there is some issue when you add side input (My side input
was coming via Global window to ParDo in a Fixed Window) to stateful DoFn.
As a work around, instead of adding side input to stateful ParDo, I
introduced another ParDo that enriches streaming data with side input
before flowing into stateful DoFn. That seems to have fixed the problem.
Thanks and regards
Mohil
On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare <mo...@prosimo.io> wrote:
> Hello All,
>
> Any idea how to debug this and find out which stage, which DoFn or which
> side input is causing the problem?
> Do I need to override OnTimer with every DoFn to avoid this problem?
> I thought that some uncaught exceptions were causing this and added
> various checks and exception handling in all DoFn and still seeing this
> issue.
> It has been driving me nuts. And now forget DRAIN, it happens during
> normal functioning as well. Any help would be appreciated.
>
> java.lang.UnsupportedOperationException: Attempt to deliver a timer to a
> DoFn, but timers are not supported in Dataflow.
>
> 1.
> 1. at org.apache.beam.runners.dataflow.worker.
> StreamingSideInputDoFnRunner.onTimer (
> StreamingSideInputDoFnRunner.java:86
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
> )
> 2. at org.apache.beam.runners.dataflow.worker.
> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
> )
> 3. at org.apache.beam.runners.dataflow.worker.
> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
> )
> 4. at org.apache.beam.runners.dataflow.worker.
> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
> )
> 5. at org.apache.beam.runners.dataflow.worker.
> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
> )
> 6. at org.apache.beam.runners.dataflow.worker.
> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
> )
> 7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
> ParDoOperation.finish (ParDoOperation.java:52
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
> )
> 8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
> MapTaskExecutor.execute (MapTaskExecutor.java:85
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
> )
> 9. at org.apache.beam.runners.dataflow.worker.
> StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
> )
> 10. at org.apache.beam.runners.dataflow.worker.
> StreamingDataflowWorker.access$1100 (
> StreamingDataflowWorker.java:152
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
> )
> 2.
> 1. at org.apache.beam.runners.dataflow.worker.
> StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073)
> 2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
> ThreadPoolExecutor.java:1149)
> 3. at java.util.concurrent.ThreadPoolExecutor$Worker.run (
> ThreadPoolExecutor.java:624)
> 4. at java.lang.Thread.run (Thread.java:748)
>
> Thanks
> Mohil
>
>
> On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare <mo...@prosimo.io> wrote:
>
>> and it seems be due to TimerType User
>>
>> Thanks
>> Mohil
>>
>> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote:
>>
>>> Hello,
>>>
>>> I was looking at source code of
>>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
>>> ,
>>>
>>> It seems the default implementation of OnTimer is to throw (Attempt to
>>> deliver a timer to a DoFn, but timers are not supported in Dataflow).
>>>
>>> Do you know under what circumstances, My code might be throwing this ?
>>> Not sure if its some issue in 2.19 which might have been fixed now with
>>> 2.22
>>>
>>> Thanks and Regards
>>> Mohil
>>>
>>>
>>> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>
>>>> Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's
>>>> getting flooded with this exception
>>>>
>>>> Attempt to deliver a timer to a DoFn, but timers are not supported in
>>>> Dataflow.
>>>> The changes that I did are the following:
>>>> 1. Read one set of logs from one of the kafka topics and create KV(ID,
>>>> log1)
>>>> 2. Read 2nd set of logs from another kafka topic and create KV(ID, log2)
>>>> 3. Put above in session window
>>>> 4. CoGroupByKey both logs
>>>>
>>>> Apart from aforementioned exceptions, I am also seeing the following in
>>>> worker logs. What I have noticed is that it happens when the pipeline is
>>>> silent for a while. i.e. no new logs to be read from Kafka (Not sure if it
>>>> is the actual reason).
>>>>
>>>> 2020-07-24 17:06:43.532 PDT
>>>> Execution of work for P188 for key
>>>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will
>>>> retry locally.
>>>>
>>>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
>>>> 2020-07-24 17:06:53.863 PDT
>>>> Uncaught exception:
>>>>
>>>>
>>>>
>>>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am on java sdk 2.19 and using dataflow for beam job. I use Timers
>>>>> for my stateful transformations, but recently I started seeing the
>>>>> following exception on DRAINING a job. It used to work fine and not sure
>>>>> what changed.
>>>>>
>>>>> *java.lang.UnsupportedOperationException:*
>>>>>
>>>> *Attempt to deliver a timer to a DoFn, but timers are not supported in
>>>> Dataflow. *
>>>>
>>>>>
>>>>> 1.
>>>>> 1. atorg.apache.beam.runners.dataflow.worker.
>>>>> StreamingSideInputDoFnRunner.onTimer (
>>>>> StreamingSideInputDoFnRunner.java:86
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>>> )
>>>>> 2. atorg.apache.beam.runners.dataflow.worker.
>>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>>> )
>>>>> 3. atorg.apache.beam.runners.dataflow.worker.
>>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>>> )
>>>>> 4. atorg.apache.beam.runners.dataflow.worker.
>>>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>>> )
>>>>> 5. atorg.apache.beam.runners.dataflow.worker.
>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>>> )
>>>>> 6. atorg.apache.beam.runners.dataflow.worker.
>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>>> )
>>>>> 7. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>>> )
>>>>> 8. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
>>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>>> )
>>>>> 9. atorg.apache.beam.runners.dataflow.worker.
>>>>> StreamingDataflowWorker.process (
>>>>> StreamingDataflowWorker.java:1350
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>>> )
>>>>> 10. atorg.apache.beam.runners.dataflow.worker.
>>>>> StreamingDataflowWorker.access$1100 (
>>>>> StreamingDataflowWorker.java:152
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>>> )
>>>>> 2. Sometime back I opened jira for an issue which is related to
>>>>> this while doing DRAIN:
>>>>> https://issues.apache.org/jira/browse/BEAM-10053 (Looks like no
>>>>> one has taken a stab on this Jira)
>>>>> 3. Not sure if the reason is same and due to multiple side inputs
>>>>> that i use in my PTransforms.
>>>>>
>>>>>
>>>>> Any help would be appreciated.
>>>>>
>>>>> Thanks and Regards
>>>>> Mohil
>>>>>
>>>>>
Exceptions: Attempt to deliver a timer to a DoFn, but timers are not
supported in Dataflow.
Posted by Mohil Khare <mo...@prosimo.io>.
Hello All,
Any idea how to debug this and find out which stage, which DoFn or which
side input is causing the problem?
Do I need to override OnTimer with every DoFn to avoid this problem?
I thought that some uncaught exceptions were causing this and added various
checks and exception handling in all DoFn and still seeing this issue.
It has been driving me nuts. And now forget DRAIN, it happens during normal
functioning as well. Any help would be appreciated.
java.lang.UnsupportedOperationException: Attempt to deliver a timer to a
DoFn, but timers are not supported in Dataflow.
1.
1. at org.apache.beam.runners.dataflow.worker.
StreamingSideInputDoFnRunner.onTimer (
StreamingSideInputDoFnRunner.java:86
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
)
2. at org.apache.beam.runners.dataflow.worker.
SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
)
3. at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.access$600
(SimpleParDoFn.java:73
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
)
4. at org.apache.beam.runners.dataflow.worker.
SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
)
5. at org.apache.beam.runners.dataflow.worker.
SimpleParDoFn.processTimers (SimpleParDoFn.java:473
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
)
6. at org.apache.beam.runners.dataflow.worker.
SimpleParDoFn.processTimers (SimpleParDoFn.java:353
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
)
7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
ParDoOperation.finish (ParDoOperation.java:52
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
)
8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
MapTaskExecutor.execute (MapTaskExecutor.java:85
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
)
9. at org.apache.beam.runners.dataflow.worker.
StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
)
10. at org.apache.beam.runners.dataflow.worker.
StreamingDataflowWorker.access$1100 (StreamingDataflowWorker.java:152
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
)
2.
1. at org.apache.beam.runners.dataflow.worker.
StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073)
2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
ThreadPoolExecutor.java:1149)
3. at java.util.concurrent.ThreadPoolExecutor$Worker.run (
ThreadPoolExecutor.java:624)
4. at java.lang.Thread.run (Thread.java:748)
Thanks
Mohil
On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare <mo...@prosimo.io> wrote:
> and it seems be due to TimerType User
>
> Thanks
> Mohil
>
> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote:
>
>> Hello,
>>
>> I was looking at source code of
>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
>> ,
>>
>> It seems the default implementation of OnTimer is to throw (Attempt to
>> deliver a timer to a DoFn, but timers are not supported in Dataflow).
>>
>> Do you know under what circumstances, My code might be throwing this ?
>> Not sure if its some issue in 2.19 which might have been fixed now with
>> 2.22
>>
>> Thanks and Regards
>> Mohil
>>
>>
>> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote:
>>
>>> Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's
>>> getting flooded with this exception
>>>
>>> Attempt to deliver a timer to a DoFn, but timers are not supported in
>>> Dataflow.
>>> The changes that I did are the following:
>>> 1. Read one set of logs from one of the kafka topics and create KV(ID,
>>> log1)
>>> 2. Read 2nd set of logs from another kafka topic and create KV(ID, log2)
>>> 3. Put above in session window
>>> 4. CoGroupByKey both logs
>>>
>>> Apart from aforementioned exceptions, I am also seeing the following in
>>> worker logs. What I have noticed is that it happens when the pipeline is
>>> silent for a while. i.e. no new logs to be read from Kafka (Not sure if it
>>> is the actual reason).
>>>
>>> 2020-07-24 17:06:43.532 PDT
>>> Execution of work for P188 for key
>>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will
>>> retry locally.
>>>
>>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
>>> 2020-07-24 17:06:53.863 PDT
>>> Uncaught exception:
>>>
>>>
>>>
>>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am on java sdk 2.19 and using dataflow for beam job. I use Timers for
>>>> my stateful transformations, but recently I started seeing the following
>>>> exception on DRAINING a job. It used to work fine and not sure what changed.
>>>>
>>>> *java.lang.UnsupportedOperationException:*
>>>>
>>> *Attempt to deliver a timer to a DoFn, but timers are not supported in
>>> Dataflow. *
>>>
>>>>
>>>> 1.
>>>> 1. atorg.apache.beam.runners.dataflow.worker.
>>>> StreamingSideInputDoFnRunner.onTimer (
>>>> StreamingSideInputDoFnRunner.java:86
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>> )
>>>> 2. atorg.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>> )
>>>> 3. atorg.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>> )
>>>> 4. atorg.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>> )
>>>> 5. atorg.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>> )
>>>> 6. atorg.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>> )
>>>> 7. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>> )
>>>> 8. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>> )
>>>> 9. atorg.apache.beam.runners.dataflow.worker.
>>>> StreamingDataflowWorker.process (
>>>> StreamingDataflowWorker.java:1350
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>> )
>>>> 10. atorg.apache.beam.runners.dataflow.worker.
>>>> StreamingDataflowWorker.access$1100 (
>>>> StreamingDataflowWorker.java:152
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>> )
>>>> 2. Sometime back I opened jira for an issue which is related to
>>>> this while doing DRAIN:
>>>> https://issues.apache.org/jira/browse/BEAM-10053 (Looks like no one
>>>> has taken a stab on this Jira)
>>>> 3. Not sure if the reason is same and due to multiple side inputs
>>>> that i use in my PTransforms.
>>>>
>>>>
>>>> Any help would be appreciated.
>>>>
>>>> Thanks and Regards
>>>> Mohil
>>>>
>>>>
Re: On DRAIN: Attempt to deliver a timer to a DoFn, but timers are
not supported in Dataflow.
Posted by Mohil Khare <mo...@prosimo.io>.
and it seems be due to TimerType User
Thanks
Mohil
On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote:
> Hello,
>
> I was looking at source code of
> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
> ,
>
> It seems the default implementation of OnTimer is to throw (Attempt to
> deliver a timer to a DoFn, but timers are not supported in Dataflow).
>
> Do you know under what circumstances, My code might be throwing this ?
> Not sure if its some issue in 2.19 which might have been fixed now with
> 2.22
>
> Thanks and Regards
> Mohil
>
>
> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote:
>
>> Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's
>> getting flooded with this exception
>>
>> Attempt to deliver a timer to a DoFn, but timers are not supported in
>> Dataflow.
>> The changes that I did are the following:
>> 1. Read one set of logs from one of the kafka topics and create KV(ID,
>> log1)
>> 2. Read 2nd set of logs from another kafka topic and create KV(ID, log2)
>> 3. Put above in session window
>> 4. CoGroupByKey both logs
>>
>> Apart from aforementioned exceptions, I am also seeing the following in
>> worker logs. What I have noticed is that it happens when the pipeline is
>> silent for a while. i.e. no new logs to be read from Kafka (Not sure if it
>> is the actual reason).
>>
>> 2020-07-24 17:06:43.532 PDT
>> Execution of work for P188 for key
>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will retry
>> locally.
>>
>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
>> 2020-07-24 17:06:53.863 PDT
>> Uncaught exception:
>>
>>
>>
>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote:
>>
>>> Hello,
>>>
>>> I am on java sdk 2.19 and using dataflow for beam job. I use Timers for
>>> my stateful transformations, but recently I started seeing the following
>>> exception on DRAINING a job. It used to work fine and not sure what changed.
>>>
>>> *java.lang.UnsupportedOperationException:*
>>>
>> *Attempt to deliver a timer to a DoFn, but timers are not supported in
>> Dataflow. *
>>
>>>
>>> 1.
>>> 1. atorg.apache.beam.runners.dataflow.worker.
>>> StreamingSideInputDoFnRunner.onTimer (
>>> StreamingSideInputDoFnRunner.java:86
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>> )
>>> 2. atorg.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>> )
>>> 3. atorg.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>> )
>>> 4. atorg.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>> )
>>> 5. atorg.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>> )
>>> 6. atorg.apache.beam.runners.dataflow.worker.
>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>> )
>>> 7. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
>>> ParDoOperation.finish (ParDoOperation.java:52
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>> )
>>> 8. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>> )
>>> 9. atorg.apache.beam.runners.dataflow.worker.
>>> StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>> )
>>> 10. atorg.apache.beam.runners.dataflow.worker.
>>> StreamingDataflowWorker.access$1100 (
>>> StreamingDataflowWorker.java:152
>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>> )
>>> 2. Sometime back I opened jira for an issue which is related to this
>>> while doing DRAIN: https://issues.apache.org/jira/browse/BEAM-10053 (Looks
>>> like no one has taken a stab on this Jira)
>>> 3. Not sure if the reason is same and due to multiple side inputs
>>> that i use in my PTransforms.
>>>
>>>
>>> Any help would be appreciated.
>>>
>>> Thanks and Regards
>>> Mohil
>>>
>>>
Re: On DRAIN: Attempt to deliver a timer to a DoFn, but timers are
not supported in Dataflow.
Posted by Mohil Khare <mo...@prosimo.io>.
Hello,
I was looking at source code of
https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
,
It seems the default implementation of OnTimer is to throw (Attempt to
deliver a timer to a DoFn, but timers are not supported in Dataflow).
Do you know under what circumstances, My code might be throwing this ? Not
sure if its some issue in 2.19 which might have been fixed now with 2.22
Thanks and Regards
Mohil
On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote:
> Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's
> getting flooded with this exception
>
> Attempt to deliver a timer to a DoFn, but timers are not supported in
> Dataflow.
> The changes that I did are the following:
> 1. Read one set of logs from one of the kafka topics and create KV(ID,
> log1)
> 2. Read 2nd set of logs from another kafka topic and create KV(ID, log2)
> 3. Put above in session window
> 4. CoGroupByKey both logs
>
> Apart from aforementioned exceptions, I am also seeing the following in
> worker logs. What I have noticed is that it happens when the pipeline is
> silent for a while. i.e. no new logs to be read from Kafka (Not sure if it
> is the actual reason).
>
> 2020-07-24 17:06:43.532 PDT
> Execution of work for P188 for key
> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will retry
> locally.
>
> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
> 2020-07-24 17:06:53.863 PDT
> Uncaught exception:
>
>
>
> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote:
>
>> Hello,
>>
>> I am on java sdk 2.19 and using dataflow for beam job. I use Timers for
>> my stateful transformations, but recently I started seeing the following
>> exception on DRAINING a job. It used to work fine and not sure what changed.
>>
>> *java.lang.UnsupportedOperationException:*
>>
> *Attempt to deliver a timer to a DoFn, but timers are not supported in
> Dataflow. *
>
>>
>> 1.
>> 1. atorg.apache.beam.runners.dataflow.worker.
>> StreamingSideInputDoFnRunner.onTimer (
>> StreamingSideInputDoFnRunner.java:86
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>> )
>> 2. atorg.apache.beam.runners.dataflow.worker.
>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>> )
>> 3. atorg.apache.beam.runners.dataflow.worker.
>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>> )
>> 4. atorg.apache.beam.runners.dataflow.worker.
>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>> )
>> 5. atorg.apache.beam.runners.dataflow.worker.
>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>> )
>> 6. atorg.apache.beam.runners.dataflow.worker.
>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>> )
>> 7. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
>> ParDoOperation.finish (ParDoOperation.java:52
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>> )
>> 8. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>> )
>> 9. atorg.apache.beam.runners.dataflow.worker.
>> StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>> )
>> 10. atorg.apache.beam.runners.dataflow.worker.
>> StreamingDataflowWorker.access$1100 (
>> StreamingDataflowWorker.java:152
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>> )
>> 2. Sometime back I opened jira for an issue which is related to this
>> while doing DRAIN: https://issues.apache.org/jira/browse/BEAM-10053 (Looks
>> like no one has taken a stab on this Jira)
>> 3. Not sure if the reason is same and due to multiple side inputs
>> that i use in my PTransforms.
>>
>>
>> Any help would be appreciated.
>>
>> Thanks and Regards
>> Mohil
>>
>>
Re: On DRAIN: Attempt to deliver a timer to a DoFn, but timers are
not supported in Dataflow.
Posted by Mohil Khare <mo...@prosimo.io>.
Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's
getting flooded with this exception
Attempt to deliver a timer to a DoFn, but timers are not supported in
Dataflow.
The changes that I did are the following:
1. Read one set of logs from one of the kafka topics and create KV(ID, log1)
2. Read 2nd set of logs from another kafka topic and create KV(ID, log2)
3. Put above in session window
4. CoGroupByKey both logs
Apart from aforementioned exceptions, I am also seeing the following in
worker logs. What I have noticed is that it happens when the pipeline is
silent for a while. i.e. no new logs to be read from Kafka (Not sure if it
is the actual reason).
2020-07-24 17:06:43.532 PDT
Execution of work for P188 for key
cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will retry
locally.
<https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
2020-07-24 17:06:53.863 PDT
Uncaught exception:
On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote:
> Hello,
>
> I am on java sdk 2.19 and using dataflow for beam job. I use Timers for my
> stateful transformations, but recently I started seeing the following
> exception on DRAINING a job. It used to work fine and not sure what changed.
>
> *java.lang.UnsupportedOperationException:*
>
*Attempt to deliver a timer to a DoFn, but timers are not supported in
Dataflow. *
>
> 1.
> 1. atorg.apache.beam.runners.dataflow.worker.
> StreamingSideInputDoFnRunner.onTimer (
> StreamingSideInputDoFnRunner.java:86
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
> )
> 2. atorg.apache.beam.runners.dataflow.worker.
> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
> )
> 3. atorg.apache.beam.runners.dataflow.worker.
> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
> )
> 4. atorg.apache.beam.runners.dataflow.worker.
> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
> )
> 5. atorg.apache.beam.runners.dataflow.worker.
> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
> )
> 6. atorg.apache.beam.runners.dataflow.worker.
> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
> )
> 7. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
> ParDoOperation.finish (ParDoOperation.java:52
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
> )
> 8. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
> MapTaskExecutor.execute (MapTaskExecutor.java:85
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
> )
> 9. atorg.apache.beam.runners.dataflow.worker.
> StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
> )
> 10. atorg.apache.beam.runners.dataflow.worker.
> StreamingDataflowWorker.access$1100 (
> StreamingDataflowWorker.java:152
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
> )
> 2. Sometime back I opened jira for an issue which is related to this
> while doing DRAIN: https://issues.apache.org/jira/browse/BEAM-10053 (Looks
> like no one has taken a stab on this Jira)
> 3. Not sure if the reason is same and due to multiple side inputs that
> i use in my PTransforms.
>
>
> Any help would be appreciated.
>
> Thanks and Regards
> Mohil
>
>