You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/04 08:53:03 UTC
[GitHub] [beam] gabihodoroaga opened a new issue, #23028: [Bug]: BigQueryIO.Write.Method.STORAGE_WRITE_API fails if you don’t specify the project in the destination.
gabihodoroaga opened a new issue, #23028:
URL: https://github.com/apache/beam/issues/23028
### What happened?
I encountered this error when I updated my BigQuery batch pipeline write method from `FILE_LOADS` to `STORAGE_WRITE_API`
```text
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.RuntimeException:
java.lang.RuntimeException: java.lang.RuntimeException:
com.google.api.gax.rpc.PermissionDeniedException:
io.grpc.StatusRuntimeException: PERMISSION_DENIED: Permission denied: Consumer 'project:null' has been suspended.
```
After a while I realized that the error is generated by the fact that I’m not setting the project name when I set the destination. For example `dataset.table` instead of `project-id:dataset.table`.
This breaks the compatibility with the current pipelines and it will make it harder for people to make the transition to the new BigQuery Storage Write API.
By digging into the source code I found the this situation is handled by the FILE_LOADS by using the project property from the pipeline options
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java:239
```java
if (Strings.isNullOrEmpty(tableReference.getProjectId())) {
BigQueryOptions options = c.getPipelineOptions().as(BigQueryOptions.class);
tableReference.setProjectId(
options.getBigQueryProject() == null
? options.getProject()
: options.getBigQueryProject());
tableDestination = tableDestination.withTableReference(tableReference);
}
```
For the Storage API implementation this can be handled by updating this function and pass the pipeline options as parameter
```java
public String getTableUrn() {
TableReference table = getTableReference();
return String.format(
"projects/%s/datasets/%s/tables/%s",
table.getProjectId(), table.getDatasetId(), table.getTableId());
}
```
I can try to make a PR if this approach is acceptable.
### Issue Priority
Priority: 3
### Issue Component
Component: io-java-gcp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on issue #23028: [Bug]: BigQueryIO.Write.Method.STORAGE_WRITE_API fails if you don’t specify the project in the destination.
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on issue #23028:
URL: https://github.com/apache/beam/issues/23028#issuecomment-1337778578
P.S. It might make more sense to add a `TableDestination::resolveProjectId` method that takes a PipelineOptions parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on issue #23028: [Bug]: BigQueryIO.Write.Method.STORAGE_WRITE_API fails if you don’t specify the project in the destination.
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on issue #23028:
URL: https://github.com/apache/beam/issues/23028#issuecomment-1337662129
This seems like a simple enough fix. @gabihodoroaga you're welcome to open a PR and tag me as a reviewer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #23028: [Bug]: BigQueryIO.Write.Method.STORAGE_WRITE_API fails if you don’t specify the project in the destination.
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #23028:
URL: https://github.com/apache/beam/issues/23028#issuecomment-1335621271
@johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #23028: [Bug]: BigQueryIO.Write.Method.STORAGE_WRITE_API fails if you don’t specify the project in the destination.
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #23028:
URL: https://github.com/apache/beam/issues/23028#issuecomment-1337630839
@ahmedabu98 do you have some context on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem closed issue #23028: [Bug]: BigQueryIO.Write.Method.STORAGE_WRITE_API fails if you don’t specify the project in the destination.
Posted by GitBox <gi...@apache.org>.
pabloem closed issue #23028: [Bug]: BigQueryIO.Write.Method.STORAGE_WRITE_API fails if you don’t specify the project in the destination.
URL: https://github.com/apache/beam/issues/23028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] gabihodoroaga commented on issue #23028: [Bug]: BigQueryIO.Write.Method.STORAGE_WRITE_API fails if you don’t specify the project in the destination.
Posted by GitBox <gi...@apache.org>.
gabihodoroaga commented on issue #23028:
URL: https://github.com/apache/beam/issues/23028#issuecomment-1339807541
Thanks @ahmedabu98 I will open a PR for this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org