You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Virinchi2595 (via GitHub)" <gi...@apache.org> on 2023/07/22 08:32:19 UTC

[GitHub] [beam] Virinchi2595 opened a new issue, #27616: [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java

Virinchi2595 opened a new issue, #27616:
URL: https://github.com/apache/beam/issues/27616

   ### What happened?
   
   Create two tables in bigquery with primary key on table1 and table2 then create simple pipeline to read from table1 and upsert into table 2 with same schema
   
   ```
   PCollection<TableRow> rows1 = p.apply(BigQueryIO.readTableRows().from("table1"));
   
   
   PCollection<RowMutation> latestIngestionTime = rows1.apply(MapElements
                   .into(new TypeDescriptor<RowMutation>() {
                   })
                   .via(tableRow -> RowMutation.of(tableRow,
                           RowMutationInformation.of(RowMutationInformation.MutationType.UPSERT, new Random().nextLong(1,1000000)))));
   
   
           latestIngestionTime.setCoder(RowMutation.RowMutationCoder.of())
   
                   .apply(BigQueryIO.applyRowMutations()
                           .to("table2")
                           .withSchema(createSchema())
                           .withMethod(BigQueryIO.Write.Method.STORAGE_API_AT_LEAST_ONCE)
                   .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER))
   ```
   I am facing these two exceptions 
   
   `com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The primary keys and clustering keys of the table are required to create an upsert stream. Entity: projects/myproject/datasets/mydataset/tables/mytable/streams/_default`
   
   
   `Caused by: java.lang.RuntimeException: Append to stream projects/myproject/datasets/dataset/tables/table/streams/_default failed with stream doesn't exist
   `
   
   
   
   
   
   
   
   ### Issue Priority
   
   Priority: 1 (data loss / total loss of function)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [X] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [X] Component: IO connector
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Virinchi2595 commented on issue #27616: [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java

Posted by "Virinchi2595 (via GitHub)" <gi...@apache.org>.
Virinchi2595 commented on issue #27616:
URL: https://github.com/apache/beam/issues/27616#issuecomment-1649986745

   @reuvenlax need your assistance here can you please check what is issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] liferoad commented on issue #27616: [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #27616:
URL: https://github.com/apache/beam/issues/27616#issuecomment-1653547995

   post his stack overflow question here: https://stackoverflow.com/questions/76739967/apache-beam-bigqueryiojavaio-grpc-statusruntimeexceptioninvalid-argumentthe?noredirect=1#comment135312528_76739967


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Virinchi2595 commented on issue #27616: [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java

Posted by "Virinchi2595 (via GitHub)" <gi...@apache.org>.
Virinchi2595 commented on issue #27616:
URL: https://github.com/apache/beam/issues/27616#issuecomment-1652995286

   > have you set a primary key on this table when creating it?
   
   yes @reuvenlax 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] reuvenlax commented on issue #27616: [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java

Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on issue #27616:
URL: https://github.com/apache/beam/issues/27616#issuecomment-1654423633

   The tablespec in to should be in the form <project>.<dataset>.<table>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Virinchi2595 commented on issue #27616: [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java

Posted by "Virinchi2595 (via GitHub)" <gi...@apache.org>.
Virinchi2595 commented on issue #27616:
URL: https://github.com/apache/beam/issues/27616#issuecomment-1654787295

   > The tablespec in to should be in the form project.dataset.table
   
   Yes I used table spec in same format(project.dataset.table) in the issue i mentioned table1 and table2 just for reference.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] reuvenlax commented on issue #27616: [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java

Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on issue #27616:
URL: https://github.com/apache/beam/issues/27616#issuecomment-1652810389

   have you set a primary key on this table when creating it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] liferoad commented on issue #27616: [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #27616:
URL: https://github.com/apache/beam/issues/27616#issuecomment-1646718797

   @Abacn @ahmedabu98 Can you check this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java [beam]

Posted by "kota-yata (via GitHub)" <gi...@apache.org>.
kota-yata commented on issue #27616:
URL: https://github.com/apache/beam/issues/27616#issuecomment-1785199534

   The project id and the dataset name should be joined together with colon, not with period. `myproject:testing.test_new` instead of `myproject.testing.test_new`.
   
   The code above should be as follows:
   ```
   rows.apply(BigQueryIO.applyRowMutations()
                           .to("myproject:testing.test_new")
                           .withJsonSchema(tableSchemaJson)
                           .withMethod(BigQueryIO.Write.Method.STORAGE_API_AT_LEAST_ONCE)
                   .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER))
   ```
   
   I've actually posted my answer to the stack overflow question.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java [beam]

Posted by "colt-snaphade (via GitHub)" <gi...@apache.org>.
colt-snaphade commented on issue #27616:
URL: https://github.com/apache/beam/issues/27616#issuecomment-1973162829

   Hello guys, anyone got solution on this.  I'm also facing the exact same issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org