You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Oscar Korz (JIRA)" <ji...@apache.org> on 2018/01/30 17:21:00 UTC
[jira] [Comment Edited] (BEAM-3569) SpannerIO.write throws on
delete mutations
[ https://issues.apache.org/jira/browse/BEAM-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345402#comment-16345402 ]
Oscar Korz edited comment on BEAM-3569 at 1/30/18 5:20 PM:
-----------------------------------------------------------
I've attached a minimal example to demonstrate the issue. It seems reproducing the error does not even require a valid Spanner deployment to write to:
{code}
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.IllegalStateException: values() cannot be called for a DELETE mutation
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:344)
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:314)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:208)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
at Main.main(Main.java:42)
Caused by: java.lang.IllegalStateException: values() cannot be called for a DELETE mutation
at com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
at org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
at org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
at org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.processElement(SpannerWriteGroupFn.java:77)
{code}
was (Author: okorz001):
I've attached a minimal example to demonstrate the issue. It seems reproducing the error does not even require a valid Spanner deployment to write to:
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.IllegalStateException: values() cannot be called for a DELETE mutation
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:344)
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:314)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:208)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
at Main.main(Main.java:42)
Caused by: java.lang.IllegalStateException: values() cannot be called for a DELETE mutation
at com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
at org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
at org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
at org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.processElement(SpannerWriteGroupFn.java:77)
> SpannerIO.write throws on delete mutations
> ------------------------------------------
>
> Key: BEAM-3569
> URL: https://issues.apache.org/jira/browse/BEAM-3569
> Project: Beam
> Issue Type: Bug
> Components: runner-core
> Affects Versions: 2.2.0
> Reporter: Oscar Korz
> Assignee: Kenneth Knowles
> Priority: Major
> Attachments: beam-spanner-io-delete.tar.gz
>
>
> It is currently impossible to delete a Spanner row in Beam with SpannerIO. The exception is generated by trying to guess the size of a delete mutation which cannot contain any values (deletes are simply by key).
> The root exception stack trace:
> {code:java}
> Caused by: java.lang.IllegalStateException: values() cannot be called for a DELETE mutation
> at com.google.common.base.Preconditions.checkState(Preconditions.java:456)
> at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
> at org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
> at org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
> {code}
> I believe this can be fixed by special casing MutationSizeEstimator.sizeOf to either 0 or 1 for Mutations with getOperation() = Op.DELETE.
> The workaround is to avoid using SpannerIO and use Spanner client API directly in a custom DoFn, but this forces users to either reimplement all the intelligent batching that SpannerIO does or suffer poor performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)