You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Jason Bowman (Jira)" <ji...@apache.org> on 2019/09/23 19:33:00 UTC
[jira] [Commented] (BEAM-8029) Using BigQueryIO.read with
DIRECT_READ causes Illegal Mutation
[ https://issues.apache.org/jira/browse/BEAM-8029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936139#comment-16936139 ]
Jason Bowman commented on BEAM-8029:
------------------------------------
I'm seeing field mutation/corruption in the generic record results using Method.DIRECT_READ with the DataflowRunner and beam 2.15.0, and this runtime exception from the directrunner so it seems to be legit.
> Using BigQueryIO.read with DIRECT_READ causes Illegal Mutation
> ---------------------------------------------------------------
>
> Key: BEAM-8029
> URL: https://issues.apache.org/jira/browse/BEAM-8029
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.14.0
> Reporter: Chris Larsen
> Priority: Major
>
>
> Code to read from BigQuery that is causing the issue:
> {code:java}
> pipeline
> .apply(BigQueryIO
> .read(SchemaAndRecord::getRecord)
> .from(options.getTableRef())
> .withMethod(Method.DIRECT_READ)
> .withCoder(AvroCoder.of(schema)))
> {code}
> If we remove .withMethod(Method.DIRECT_READ) then there is no issue.
>
> The error is:
> {code:java}
> org.apache.beam.sdk.util.IllegalMutationException: PTransform BigQueryIO.TypedRead/Read(BigQueryStorageTableSource) mutated value {"device_id": "rpi-rpi0-thermostat", "temperature_c": 20.0, "temperature_f": 52.0, "sample_time": 1564412307969368, "humidity": 74.3} after it was output (new value was {"device_id": "rpi-rpi0-thermostat", "temperature_c": 20.0, "temperature_f": 52.0, "sample_time": 1564412360458615, "humidity": 74.7}). Values must not be mutated in any way after being output.
> at org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.commit (ImmutabilityCheckingBundleFactory.java:134)
> at org.apache.beam.runners.direct.EvaluationContext.commitBundles (EvaluationContext.java:210)
> at org.apache.beam.runners.direct.EvaluationContext.handleResult (EvaluationContext.java:151)
> at org.apache.beam.runners.direct.QuiescenceDriver$TimerIterableCompletionCallback.handleResult (QuiescenceDriver.java:262)
> at org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle (DirectTransformExecutor.java:189)
> at org.apache.beam.runners.direct.DirectTransformExecutor.run (DirectTransformExecutor.java:126)
> at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
> at java.lang.Thread.run (Thread.java:748)
> Caused by: org.apache.beam.sdk.util.IllegalMutationException: Value {"device_id": "rpi-rpi0-thermostat", "temperature_c": 20.0, "temperature_f": 52.0, "sample_time": 1564412307969368, "humidity": 74.3} mutated illegally, new value was {"device_id": "rpi-rpi0-thermostat", "temperature_c": 20.0, "temperature_f": 52.0, "sample_time": 1564412360458615, "humidity": 74.7}. Encoding was AiZycGktcnBpMC10aGVybW9zdGF0AgAAAAAAADRAAgAAAAAAAEpAArDVsP7jtMcFAjMzMzMzk1JA, now AiZycGktcnBpMC10aGVybW9zdGF0AgAAAAAAADRAAgAAAAAAAEpAAu6FuLDktMcFAs3MzMzMrFJA.
> at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.illegalMutation (MutationDetectors.java:153)
> at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodifiedThrowingCheckedExceptions (MutationDetectors.java:148)
> at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodified (MutationDetectors.java:123)
> at org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.commit (ImmutabilityCheckingBundleFactory.java:124)
> at org.apache.beam.runners.direct.EvaluationContext.commitBundles (EvaluationContext.java:210)
> at org.apache.beam.runners.direct.EvaluationContext.handleResult (EvaluationContext.java:151)
> at org.apache.beam.runners.direct.QuiescenceDriver$TimerIterableCompletionCallback.handleResult (QuiescenceDriver.java:262)
> at org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle (DirectTransformExecutor.java:189)
> at org.apache.beam.runners.direct.DirectTransformExecutor.run (DirectTransformExecutor.java:126)
> at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
> at java.lang.Thread.run (Thread.java:748){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)