You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Mohil Khare <mo...@prosimo.io> on 2020/07/09 04:53:29 UTC
Unable to read value from state/Unable to fetch data due to token
mismatch for key
Hello,
I am using beam java sdk 2.19.0 (with enableStreamingEngine set as true)
and very heavily use stateful beam processing model.
However, sometimes I am seeing the following exception while reading value
from state for a key (Please note : here my key is a POJO where fields
create a kind of composite key. Also I am using AvroCoder for this key):
Caused by: java.util.concurrent.ExecutionException:
org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException: Unable to
fetch data due to token mismatch for key
0ggadot_static@prosimo.ioHaa552bec-25f2-11ea-8705-267acc424a25H9219bdd5-335f-11ea-bd4f-de07a30b09ca
@ OC-AU sydney
1.
1. at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
AbstractFuture.getDoneValue (AbstractFuture.java:531
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FAbstractFuture.java&line=531&project=prosimo>
)
2. at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
AbstractFuture.get (AbstractFuture.java:492
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FAbstractFuture.java&line=492&project=prosimo>
)
3. at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
AbstractFuture$TrustedFuture.get (AbstractFuture.java:83
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FAbstractFuture.java&line=83&project=prosimo>
)
4. at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
ForwardingFuture.get (ForwardingFuture.java:62
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FForwardingFuture.java&line=62&project=prosimo>
)
5. atorg.apache.beam.runners.dataflow.worker.
WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:316
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=316&project=prosimo>
)
6. atorg.apache.beam.runners.dataflow.worker.
WindmillStateInternals$WindmillValue.read (
WindmillStateInternals.java:385
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateInternals.java&line=385&project=prosimo>
)
Caused by: org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException:
Unable to fetch data due to token mismatch for key <key>
1.
1. atorg.apache.beam.runners.dataflow.worker.
WindmillStateReader.consumeResponse (WindmillStateReader.java:482
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=482&project=prosimo>
)
2. atorg.apache.beam.runners.dataflow.worker.
WindmillStateReader.startBatchAndBlock (WindmillStateReader.java:420
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=420&project=prosimo>
)
3. atorg.apache.beam.runners.dataflow.worker.
WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:313
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=313&project=prosimo>
)
Any help to fix this issue would be greatly appreciated.
Thanks and Regards
Mohil
Re: Unable to read value from state/Unable to fetch data due to token
mismatch for key
Posted by Mohil Khare <mo...@prosimo.io>.
Thanks Reuven for your reply.
Good to know that it is benign.
Regards
Mohil
On Wed, Jul 8, 2020 at 10:19 PM Reuven Lax <re...@google.com> wrote:
> This error should be benign. It often means that ownership of the work
> item was moved to a different worker (possibly caused by autoscaling or
> other source of work rebalancing), so the in-progress work item on that
> worker failed. However the work item will be processed successfully on the
> new worker that owns it. This should not cause a persistent failure.
>
> On Wed, Jul 8, 2020 at 9:53 PM Mohil Khare <mo...@prosimo.io> wrote:
>
>> Hello,
>>
>> I am using beam java sdk 2.19.0 (with enableStreamingEngine set as true)
>> and very heavily use stateful beam processing model.
>> However, sometimes I am seeing the following exception while reading
>> value from state for a key (Please note : here my key is a POJO where
>> fields create a kind of composite key. Also I am using AvroCoder for this
>> key):
>>
>> Caused by: java.util.concurrent.ExecutionException:
>> org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException: Unable to
>> fetch data due to token mismatch for key
>> 0ggadot_static@prosimo.ioHaa552bec-25f2-11ea-8705-267acc424a25H9219bdd5-335f-11ea-bd4f-de07a30b09ca
>> @ OC-AU sydney
>>
>> 1.
>> 1. at
>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>> AbstractFuture.getDoneValue (AbstractFuture.java:531
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FAbstractFuture.java&line=531&project=prosimo>
>> )
>> 2. at
>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>> AbstractFuture.get (AbstractFuture.java:492
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FAbstractFuture.java&line=492&project=prosimo>
>> )
>> 3. at
>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>> AbstractFuture$TrustedFuture.get (AbstractFuture.java:83
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FAbstractFuture.java&line=83&project=prosimo>
>> )
>> 4. at
>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>> ForwardingFuture.get (ForwardingFuture.java:62
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FForwardingFuture.java&line=62&project=prosimo>
>> )
>> 5. atorg.apache.beam.runners.dataflow.worker.
>> WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:316
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=316&project=prosimo>
>> )
>> 6. atorg.apache.beam.runners.dataflow.worker.
>> WindmillStateInternals$WindmillValue.read (
>> WindmillStateInternals.java:385
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateInternals.java&line=385&project=prosimo>
>> )
>>
>> Caused by: org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException:
>> Unable to fetch data due to token mismatch for key <key>
>>
>> 1.
>> 1. atorg.apache.beam.runners.dataflow.worker.
>> WindmillStateReader.consumeResponse (WindmillStateReader.java:482
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=482&project=prosimo>
>> )
>> 2. atorg.apache.beam.runners.dataflow.worker.
>> WindmillStateReader.startBatchAndBlock (
>> WindmillStateReader.java:420
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=420&project=prosimo>
>> )
>> 3. atorg.apache.beam.runners.dataflow.worker.
>> WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:313
>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=313&project=prosimo>
>> )
>>
>>
>> Any help to fix this issue would be greatly appreciated.
>>
>> Thanks and Regards
>> Mohil
>>
>
Re: Unable to read value from state/Unable to fetch data due to token
mismatch for key
Posted by Reuven Lax <re...@google.com>.
This error should be benign. It often means that ownership of the work item
was moved to a different worker (possibly caused by autoscaling or other
source of work rebalancing), so the in-progress work item on that worker
failed. However the work item will be processed successfully on the new
worker that owns it. This should not cause a persistent failure.
On Wed, Jul 8, 2020 at 9:53 PM Mohil Khare <mo...@prosimo.io> wrote:
> Hello,
>
> I am using beam java sdk 2.19.0 (with enableStreamingEngine set as true)
> and very heavily use stateful beam processing model.
> However, sometimes I am seeing the following exception while reading value
> from state for a key (Please note : here my key is a POJO where fields
> create a kind of composite key. Also I am using AvroCoder for this key):
>
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException: Unable to
> fetch data due to token mismatch for key
> 0ggadot_static@prosimo.ioHaa552bec-25f2-11ea-8705-267acc424a25H9219bdd5-335f-11ea-bd4f-de07a30b09ca
> @ OC-AU sydney
>
> 1.
> 1. at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
> AbstractFuture.getDoneValue (AbstractFuture.java:531
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FAbstractFuture.java&line=531&project=prosimo>
> )
> 2. at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
> AbstractFuture.get (AbstractFuture.java:492
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FAbstractFuture.java&line=492&project=prosimo>
> )
> 3. at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
> AbstractFuture$TrustedFuture.get (AbstractFuture.java:83
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FAbstractFuture.java&line=83&project=prosimo>
> )
> 4. at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
> ForwardingFuture.get (ForwardingFuture.java:62
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent%2FForwardingFuture.java&line=62&project=prosimo>
> )
> 5. atorg.apache.beam.runners.dataflow.worker.
> WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:316
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=316&project=prosimo>
> )
> 6. atorg.apache.beam.runners.dataflow.worker.
> WindmillStateInternals$WindmillValue.read (
> WindmillStateInternals.java:385
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateInternals.java&line=385&project=prosimo>
> )
>
> Caused by: org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException:
> Unable to fetch data due to token mismatch for key <key>
>
> 1.
> 1. atorg.apache.beam.runners.dataflow.worker.
> WindmillStateReader.consumeResponse (WindmillStateReader.java:482
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=482&project=prosimo>
> )
> 2. atorg.apache.beam.runners.dataflow.worker.
> WindmillStateReader.startBatchAndBlock (WindmillStateReader.java:420
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=420&project=prosimo>
> )
> 3. atorg.apache.beam.runners.dataflow.worker.
> WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:313
> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-02_10_32_35-16490648164981155775&file=org%2Fapache.beam.runners.dataflow.worker%2FWindmillStateReader.java&line=313&project=prosimo>
> )
>
>
> Any help to fix this issue would be greatly appreciated.
>
> Thanks and Regards
> Mohil
>