You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ChetanRuparel (via GitHub)" <gi...@apache.org> on 2026/09/23 06:20:13 UTC
[PR] [BigQueryIO] Improve error message when BigQuery API is reported as not enabled [beam]
ChetanRuparel opened a new pull request, #40233:
URL: https://github.com/apache/beam/pull/40233
When BigQuery returns a `400 Bad Request` containing `"has not enabled BigQuery"` (which can occur when the project ID is misconfigured, the BigQuery API is disabled, or during a Google Cloud authentication/service outage), `BigQueryServicesImpl` previously surfaced the raw message without actionable troubleshooting guidance.
This change:
- Logs and appends remediation guidance in `InsertBatchofRowsCallable.call()` and `executeWithRetries()` instructing users to verify the project ID, check BigQuery API enablement (`https://console.cloud.google.com/apis/library/bigquery.googleapis.com`), or check the Google Cloud Status Dashboard (`https://status.cloud.google.com/`) for ongoing outages.
- Adds unit test `BigQueryServicesImplTest.testInsertBigQueryNotEnabledErrorLog`.
R: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [BigQueryIO] Improve error message when BigQuery API is reported as not enabled [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #40233:
URL: https://github.com/apache/beam/pull/40233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [BigQueryIO] Improve error message when BigQuery API is reported as not enabled [beam]
Posted by "ChetanRuparel (via GitHub)" <gi...@apache.org>.
ChetanRuparel commented on code in PR #40233:
URL: https://github.com/apache/beam/pull/40233#discussion_r4102258834
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:
##########
@@ -1734,6 +1746,12 @@ static <T> T executeWithRetries(
return request.execute();
} catch (IOException e) {
lastException = e;
+ if (ApiErrorExtractor.INSTANCE.badRequest(e)
+ && e.getMessage() != null
+ && e.getMessage().contains(BIGQUERY_NOT_ENABLED)) {
+ LOG.error(BIGQUERY_NOT_ENABLED_GUIDANCE, e);
+ throw new IOException(errorMessage + " " + BIGQUERY_NOT_ENABLED_GUIDANCE, e);
Review Comment:
Done. Updated `executeWithRetries` to throw `new IOException(e.getMessage() + " " + BIGQUERY_NOT_ENABLED_GUIDANCE, e)` so the actual BigQuery error message is surfaced directly in the raised exception. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [BigQueryIO] Improve error message when BigQuery API is reported as not enabled [beam]
Posted by "ChetanRuparel (via GitHub)" <gi...@apache.org>.
ChetanRuparel commented on code in PR #40233:
URL: https://github.com/apache/beam/pull/40233#discussion_r4102290755
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java:
##########
@@ -1523,6 +1523,63 @@ public void testInsertTimeoutLog() throws Exception {
verifyWriteMetricWasSet("project", "dataset", "table", " no rows present in the request. ", 1);
}
+ /**
+ * Tests that {@link DatasetServiceImpl#insertAll} logs and includes suggested remedy when the
Review Comment:
Done. Added `testExecuteWithRetriesBigQueryNotEnabled` in `BigQueryServicesImplTest` to verify that `executeWithRetries` fails fast without retrying, includes both `e.getMessage()` and `BIGQUERY_NOT_ENABLED_GUIDANCE` in the thrown `IOException`, and omits the caller's `"aborting after 9 retries."` message.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [BigQueryIO] Improve error message when BigQuery API is reported as not enabled [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #40233:
URL: https://github.com/apache/beam/pull/40233#discussion_r4097109705
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:
##########
@@ -1734,6 +1746,12 @@ static <T> T executeWithRetries(
return request.execute();
} catch (IOException e) {
lastException = e;
+ if (ApiErrorExtractor.INSTANCE.badRequest(e)
+ && e.getMessage() != null
+ && e.getMessage().contains(BIGQUERY_NOT_ENABLED)) {
+ LOG.error(BIGQUERY_NOT_ENABLED_GUIDANCE, e);
+ throw new IOException(errorMessage + " " + BIGQUERY_NOT_ENABLED_GUIDANCE, e);
Review Comment:
This includes the caller supported error and actual error from BQ (`e.getMessage()`) might be hidden in a long stack-trace. Can we also make `e.getMessage()` in the exception raised.
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java:
##########
@@ -1523,6 +1523,63 @@ public void testInsertTimeoutLog() throws Exception {
verifyWriteMetricWasSet("project", "dataset", "table", " no rows present in the request. ", 1);
}
+ /**
+ * Tests that {@link DatasetServiceImpl#insertAll} logs and includes suggested remedy when the
Review Comment:
Can you add a unit test for `BigQueryServicesImpl.executeWithRetries` that confirms the proper behavior for the exception raised.
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:
##########
@@ -1734,6 +1746,12 @@ static <T> T executeWithRetries(
return request.execute();
} catch (IOException e) {
lastException = e;
+ if (ApiErrorExtractor.INSTANCE.badRequest(e)
+ && e.getMessage() != null
+ && e.getMessage().contains(BIGQUERY_NOT_ENABLED)) {
+ LOG.error(BIGQUERY_NOT_ENABLED_GUIDANCE, e);
+ throw new IOException(errorMessage + " " + BIGQUERY_NOT_ENABLED_GUIDANCE, e);
+ }
Review Comment:
Also the provided `errorMessage` "aborting after %d retries." will be wrong in this case since we are failing fast after an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [BigQueryIO] Improve error message when BigQuery API is reported as not enabled [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40233:
URL: https://github.com/apache/beam/pull/40233#issuecomment-5792222623
Assigning reviewers:
R: @chamikaramj for label java.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [BigQueryIO] Improve error message when BigQuery API is reported as not enabled [beam]
Posted by "ChetanRuparel (via GitHub)" <gi...@apache.org>.
ChetanRuparel commented on code in PR #40233:
URL: https://github.com/apache/beam/pull/40233#discussion_r4102267699
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:
##########
@@ -1734,6 +1746,12 @@ static <T> T executeWithRetries(
return request.execute();
} catch (IOException e) {
lastException = e;
+ if (ApiErrorExtractor.INSTANCE.badRequest(e)
+ && e.getMessage() != null
+ && e.getMessage().contains(BIGQUERY_NOT_ENABLED)) {
+ LOG.error(BIGQUERY_NOT_ENABLED_GUIDANCE, e);
+ throw new IOException(errorMessage + " " + BIGQUERY_NOT_ENABLED_GUIDANCE, e);
+ }
Review Comment:
Done. Removed `errorMessage` from this fail-fast branch and replaced it with `e.getMessage() + " " + BIGQUERY_NOT_ENABLED_GUIDANCE` so the misleading `"aborting after %d retries."` message is not included. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org