You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2023/07/10 18:58:38 UTC
[GitHub] [beam] ahmedabu98 opened a new pull request, #27434: [Java] Cleanup BigQuery write tests
ahmedabu98 opened a new pull request, #27434:
URL: https://github.com/apache/beam/pull/27434
Most of our BigQuery write connector tests live in [BigQueryIOWriteTest](https://github.com/ahmedabu98/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java). I took a closer look recently and saw that it's in need of a cleanup. There are some tests that are duplicates, some that could be written more concisely, some have test names that are too vague (some names are even incorrect), some that run with an incorrect write method.
This PR seeks to take care of some of these issues.
This also fixes a long-standing flake `testTriggeredFileLoadsWithTempTablesToExistingNullSchemaTable[1]` ([example run](https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Cron/1482/testReport/org.apache.beam.sdk.io.gcp.bigquery/BigQueryIOWriteTest/testTriggeredFileLoadsWithTempTablesToExistingNullSchemaTable_1_/))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #27434: [Java] Cleanup BigQuery write tests
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #27434:
URL: https://github.com/apache/beam/pull/27434#issuecomment-1642356510
Opened an issue for the race condition here: https://github.com/apache/beam/issues/27561
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #27434: [Java] Cleanup BigQuery write tests
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27434:
URL: https://github.com/apache/beam/pull/27434#issuecomment-1641974137
Reminder, please take a look at this pr: @robertwb @johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #27434: [Java] Cleanup BigQuery write tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #27434:
URL: https://github.com/apache/beam/pull/27434#issuecomment-1642279209
What's the status of this PR? have you been able to reconcile the race condtions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #27434: [Java] Cleanup BigQuery write tests
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #27434:
URL: https://github.com/apache/beam/pull/27434#issuecomment-1629689305
Ahh I think I see what's going on here... there seems to be a race condition between copy jobs at different triggers. Here, the second trigger's copy jobs start running before the first trigger's copy jobs. The second trigger's copy jobs run and succeed in writing data to the table; then the first trigger's copy jobs run (with `WRITE_EMPTY`) and an error is thrown. As mentioned in the previous comment, the first trigger is associated with whatever dispositions are set by the user. All triggers after that are fixed to `WRITE_APPEND` and `CREATE_NEVER`.
Example below: In the following logs, `beam_bq_job_COPY_...510c8e70fecb762c7_00000-0` is copy jobs ID of the first trigger, `beam_bq_job_COPY_...510c8e70fecb762c7_00001-0` is copy jobs ID of the second trigger. The suffix of these names is `<tableHash>_<paneNumber>-<retryIndex>`. We see copy jobs at pane 1 starting earlier than pane 0. Not sure what is causing this to happen.
```
INFO: Job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0 pending. retrying.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
INFO: Load job beam_bq_job_TEMP_TABLE_LOAD_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_05014b97910b4635a89409bd024d057d_1af13eb133f5d1e510c8e70fecb762c7_00001_00000-0 succeeded. Statistics: null
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager waitForDone
INFO: Job beam_bq_job_TEMP_TABLE_LOAD_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_05014b97910b4635a89409bd024d057d_1af13eb133f5d1e510c8e70fecb762c7_00001_00000-0 completed successfully.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager waitForDone
INFO: Waiting for jobs to complete.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager waitForDone
INFO: Waiting for jobs to complete.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager waitForDone
INFO: Job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0 pending. retrying.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.WriteRename lambda$startCopy$57215ec2$1
INFO: Starting copy job for table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=dataset-id, projectId=project-id, tableId=table-id}} using GenericData{classInfo=[jobId, location, projectId], {jobId=beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0, projectId=project-id}}, job id iteration 0
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
INFO: job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0 started
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.WriteRename lambda$startCopy$57215ec2$1
INFO: Starting copy job for table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=dataset-id, projectId=project-id, tableId=table-id}} using GenericData{classInfo=[jobId, location, projectId], {jobId=beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0, projectId=project-id}}, job id iteration 0
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
INFO: job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0 started
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
INFO: Load job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0 succeeded. Statistics: null
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager waitForDone
INFO: Job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0 completed successfully.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers getRetryJobId
INFO: job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0 is failed. Checking the next job id
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers getRetryJobId
INFO: job id beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-1 not found, so retrying with that id
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
WARNING: Load job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0 failed, will retry: {
"errorResult" : { },
"state" : "FAILED"
}. Next job id beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey merged pull request #27434: [Java] Cleanup BigQuery write tests
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey merged PR #27434:
URL: https://github.com/apache/beam/pull/27434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #27434: [Java] Cleanup BigQuery write tests
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #27434:
URL: https://github.com/apache/beam/pull/27434#issuecomment-1629554747
> This also fixes a long-standing flake testTriggeredFileLoadsWithTempTablesToExistingNullSchemaTable[1]
That's awesome! curious about what was wrong and how it is fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #27434: [Java] Cleanup BigQuery write tests
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #27434:
URL: https://github.com/apache/beam/pull/27434#issuecomment-1629658908
>That's awesome! curious about what was wrong and how it is fixed
Thanks! yeah it took a while to figure out because our fake testing service doesn't propagate an error. I tried running the same pipeline a large number of times with a real BigQuery table and ran into the same behavior (copy job stuck retrying) on one occurrence. BQ was giving a "table already exists" error and kept retrying the copy job.
Write disposition defaults to `WRITE_EMPTY`, so I set it to `WRITE_APPEND` and reran many times without running into the same behavior.
We're supposed to already be covering this here (note this test is a streaming pipeline): https://github.com/apache/beam/blob/4c66866aa9544d1796c7c3880192cb57d2a8dcc0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java#L187-L196
ie. we always set it to `WRITE_APPEND` after the first trigger of copy jobs.
Somehow this doesn't always work? and the user-specified disposition continues to be used.
I've run the tests in a few different ways and the common denominator seems to be that this is happening when the table is created beforehand (as opposed to letting the pipeline create it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #27434: [Java] Cleanup BigQuery write tests
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #27434:
URL: https://github.com/apache/beam/pull/27434#issuecomment-1642318157
> have you been able to reconcile the race condtions?
I have not, and that investigation can probably be in a different PR because the purpose of this one is to clean up this test class. The affected test is mitigated by setting WRITE_APPEND. For now, I just need a review on these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #27434: [Java] Cleanup BigQuery write tests
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27434:
URL: https://github.com/apache/beam/pull/27434#issuecomment-1629608561
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @robertwb for label java.
R: @johnjcasey for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #27434: [Java] Cleanup BigQuery write tests
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #27434:
URL: https://github.com/apache/beam/pull/27434#issuecomment-1630743367
I'm observing the same race condition even when the pipeline creates the table. The second copy job has create disposition fixed to `CREATE_NEVER` and fails when it gets triggered first. However, the first copy job is triggered soon after with `CREATE_IF_NEEDED` and creates the table. So the second copy job only really needs to retry once and it isn't very noticeable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org