You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/20 14:43:54 UTC
[GitHub] [beam] steveniemitz opened a new issue, #23306: [Bug]: BigQueryBatchFileLoads in python loses data when using WRITE_TRUNCATE
steveniemitz opened a new issue, #23306:
URL: https://github.com/apache/beam/issues/23306
### What happened?
When using WRITE_TRUNCATE, if the data being loaded is split into multiple temp tables, the copy job that merges all the results together can end up using WRITE_TRUNCATE for each copy job, resulting in only the last copy job "winning", and overwriting all other jobs.
It looks like there was an attempt to handle this here https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L545, but this code assumes that all inputs to TriggerCopyJobs arrive in the same bundle. From what I observed this is not the case, looking at the logs from that step, the log lines from that step have different "work" fields for copy job 1 and 2.
There probably needs to be a GBK before this step in order to make sure that all copy jobs actually are executed in the same unit?
### Issue Priority
Priority: 1
### Issue Component
Component: io-py-gcp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #23306: [Bug]: BigQueryBatchFileLoads in python loses data when using WRITE_TRUNCATE
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #23306:
URL: https://github.com/apache/beam/issues/23306#issuecomment-1263843079
@ahmedabu98 how familiar are you with this? it strikes me as similar to the other BQ changes you've done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb closed issue #23306: [Bug]: BigQueryBatchFileLoads in python loses data when using WRITE_TRUNCATE
Posted by GitBox <gi...@apache.org>.
robertwb closed issue #23306: [Bug]: BigQueryBatchFileLoads in python loses data when using WRITE_TRUNCATE
URL: https://github.com/apache/beam/issues/23306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #23306: [Bug]: BigQueryBatchFileLoads in python loses data when using WRITE_TRUNCATE
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #23306:
URL: https://github.com/apache/beam/issues/23306#issuecomment-1341251288
@ahmedabu98 are you still working on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on issue #23306: [Bug]: BigQueryBatchFileLoads in python loses data when using WRITE_TRUNCATE
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on issue #23306:
URL: https://github.com/apache/beam/issues/23306#issuecomment-1264088894
Yeah it's in the same context, I can take a stab at this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on issue #23306: [Bug]: BigQueryBatchFileLoads in python loses data when using WRITE_TRUNCATE
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on issue #23306:
URL: https://github.com/apache/beam/issues/23306#issuecomment-1341259751
This resurfaced in #24535, work is being done there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on issue #23306: [Bug]: BigQueryBatchFileLoads in python loses data when using WRITE_TRUNCATE
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on issue #23306:
URL: https://github.com/apache/beam/issues/23306#issuecomment-1264088930
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org