You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Dan Kruchinin (JIRA)" <ji...@apache.org> on 2019/01/15 12:54:00 UTC
[jira] [Updated] (BEAM-6434) NPE in BigQuery client library in
StreamingBuffer.toPb
[ https://issues.apache.org/jira/browse/BEAM-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dan Kruchinin updated BEAM-6434:
--------------------------------
Description:
Sometimes I see the following NPE in my streaming pipeline:
{code:java}
Caused by: java.lang.NullPointerException com.google.cloud.bigquery.StandardTableDefinition$StreamingBuffer.toPb(StandardTableDefinition.java:106) com.google.cloud.bigquery.StandardTableDefinition.toPb(StandardTableDefinition.java:230) com.google.cloud.bigquery.TableInfo.toPb(TableInfo.java:453) com.google.cloud.bigquery.BigQueryImpl.update(BigQueryImpl.java:383) com.google.cloud.bigquery.Table.update(Table.java:212) com.heysphere.nosegment.dofn.TableRowSchemaMutatorFn.processElement(TableRowSchemaMutatorFn.java:132){code}
It happens extremely rarely and I'm not sure what can cause it. To add a bit more context, the pipeline performs `Table.update` when it handles failed inserts from BigQueryIO and detects that the problem can be fixed by adding missing columns (sort of like https://github.com/ryanmcdowell/dataflow-dynamic-schema )
Somewhat reminds BEAM-6076
was:
Sometimes I see the following NPE in my streaming pipeline:
{code:java}
Caused by: java.lang.NullPointerException com.google.cloud.bigquery.StandardTableDefinition$StreamingBuffer.toPb(StandardTableDefinition.java:106) com.google.cloud.bigquery.StandardTableDefinition.toPb(StandardTableDefinition.java:230) com.google.cloud.bigquery.TableInfo.toPb(TableInfo.java:453) com.google.cloud.bigquery.BigQueryImpl.update(BigQueryImpl.java:383) com.google.cloud.bigquery.Table.update(Table.java:212) com.heysphere.nosegment.dofn.TableRowSchemaMutatorFn.processElement(TableRowSchemaMutatorFn.java:132){code}
It happens extremely rarely and I'm not sure what can cause it. To add a bit more context, the pipeline performs `Table.update` when it handles failed inserts from BigQueryIO and detects that the problem can be fixed by adding missing columns (sort of like [https://github.com/ryanmcdowell/dataflow-dynamic-schema).]
Somewhat reminds BEAM-6076
> NPE in BigQuery client library in StreamingBuffer.toPb
> ------------------------------------------------------
>
> Key: BEAM-6434
> URL: https://issues.apache.org/jira/browse/BEAM-6434
> Project: Beam
> Issue Type: Bug
> Components: testing
> Affects Versions: 2.9.0
> Reporter: Dan Kruchinin
> Assignee: Jason Kuster
> Priority: Major
>
> Sometimes I see the following NPE in my streaming pipeline:
>
>
> {code:java}
> Caused by: java.lang.NullPointerException com.google.cloud.bigquery.StandardTableDefinition$StreamingBuffer.toPb(StandardTableDefinition.java:106) com.google.cloud.bigquery.StandardTableDefinition.toPb(StandardTableDefinition.java:230) com.google.cloud.bigquery.TableInfo.toPb(TableInfo.java:453) com.google.cloud.bigquery.BigQueryImpl.update(BigQueryImpl.java:383) com.google.cloud.bigquery.Table.update(Table.java:212) com.heysphere.nosegment.dofn.TableRowSchemaMutatorFn.processElement(TableRowSchemaMutatorFn.java:132){code}
>
>
> It happens extremely rarely and I'm not sure what can cause it. To add a bit more context, the pipeline performs `Table.update` when it handles failed inserts from BigQueryIO and detects that the problem can be fixed by adding missing columns (sort of like https://github.com/ryanmcdowell/dataflow-dynamic-schema )
>
> Somewhat reminds BEAM-6076
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)