You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "mxtymoshyk (via GitHub)" <gi...@apache.org> on 2026/08/31 13:56:02 UTC

[PR] Treat a 404 from BigQuery table and dataset deletes as success [beam]

mxtymoshyk opened a new pull request, #39934:
URL: https://github.com/apache/beam/pull/39934

   BigQueryIO creates temporary tables and datasets and deletes them when it is done. A delete can succeed at BigQuery and still have its work item fail to commit afterwards. The runner then replays that work item, and the replayed delete gets a 404 because the first attempt already removed the resource.
   
   `DatasetServiceImpl.deleteTable` and `deleteDataset` passed `ALWAYS_RETRY`, so that 404 was retried `MAX_RPC_RETRIES` (9) times with exponential backoff and then rethrown as an `IOException`. In a streaming job the work item is retried forever and the job cannot drain.
   
   Both methods now pass `DONT_RETRY_NOT_FOUND` and swallow an item-not-found error, since "the resource is gone" is the outcome the caller asked for. This mirrors how `getTable` in the same class already handles a 404. Every other status code keeps its existing retry count and failure behaviour.
   
   fixes #24997
   
   ### Notes for reviewers
   
   * The 404 is only swallowed for the two delete calls. `dryRunQuery` and `patchTableDescription` still use `ALWAYS_RETRY`, and no `getX`/`create`/`patch` path changed.
   * `ApiErrorExtractor.itemNotFound` walks `getCause()`, so it still recognises the 404 after `executeWithRetries` wraps it in a new `IOException` -- the same reason the existing `getTable` catch block works.
   * A swallowed 404 is logged at INFO with the table/dataset id, so an operator can still see that the delete found nothing.
   * The two new tests prepare exactly one mock response, so a retry trips the `Verify` in the shared mock request. They assert both "did not throw" and "did not retry".
   * Verified locally on `:sdks:java:io:google-cloud-platform`: `BigQueryServicesImplTest` passes 50/50 with the change. Reverting only the `BigQueryServicesImpl` change makes both new tests fail with a `VerifyException` from the retried request, confirming the tests cover the fix.
   * No prior PR referenced this issue.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Treat a 404 from BigQuery table and dataset deletes as success [beam]

Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on PR #39934:
URL: https://github.com/apache/beam/pull/39934#issuecomment-5682929112

   @mxtymoshyk please rebase so github action can run again. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Treat a 404 from BigQuery table and dataset deletes as success [beam]

Posted by "mxtymoshyk (via GitHub)" <gi...@apache.org>.
mxtymoshyk commented on PR #39934:
URL: https://github.com/apache/beam/pull/39934#issuecomment-5853273404

   @stankiewicz Rebased onto current master and all checks are green now. @ahmedabu98 this is ready for review when you have a moment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Treat a 404 from BigQuery table and dataset deletes as success [beam]

Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz merged PR #39934:
URL: https://github.com/apache/beam/pull/39934


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Treat a 404 from BigQuery table and dataset deletes as success [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #39934:
URL: https://github.com/apache/beam/pull/39934#issuecomment-5637599643

   R: @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Treat a 404 from BigQuery table and dataset deletes as success [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39934:
URL: https://github.com/apache/beam/pull/39934#issuecomment-5637615656

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Treat a 404 from BigQuery table and dataset deletes as success [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #39934:
URL: https://github.com/apache/beam/pull/39934#issuecomment-5637645254

   R: @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Treat a 404 from BigQuery table and dataset deletes as success [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39934:
URL: https://github.com/apache/beam/pull/39934#issuecomment-5481155287

   Assigning reviewers:
   
   R: @kennknowles for label java.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Treat a 404 from BigQuery table and dataset deletes as success [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39934:
URL: https://github.com/apache/beam/pull/39934#issuecomment-5618722492

   Reminder, please take a look at this pr: @kennknowles @stankiewicz 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Treat a 404 from BigQuery table and dataset deletes as success [beam]

Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on PR #39934:
URL: https://github.com/apache/beam/pull/39934#issuecomment-5891601028

   thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org