You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/08 18:07:38 UTC
[GitHub] [beam] rizenfrmtheashes opened a new issue, #23104: [Bug]: WriteToBigQuery with file_loads and dynamic table destination doesn't load after first File Load
rizenfrmtheashes opened a new issue, #23104:
URL: https://github.com/apache/beam/issues/23104
### What happened?
I was able to reproduce a bug in the WriteToBigQuery function in the apache beam source code for Dataflow Streaming and the File Loads setting. This issue is primarily caused by the impulse nodes feeding into the WaitForBQJobs task used in four separate nodes in the pipeline.
I reported this issue directly to the Customer Support at the Dataflow team at GCP, but after not seeing it being actioned, I'm reporting it here.
[Here is the Google Doc I created for them that walks through reproduction.](https://docs.google.com/document/d/1uIM5JVq0dAh2uDB0HfzQN7PS60U8TsJnalvAvfkfcnM/edit?usp=sharing)
This has been reproduced on both Apache beam 2.32 and 2.40, running on dataflow.
### Issue Priority
Priority: 2
### Issue Component
Component: io-py-gcp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rizenfrmtheashes commented on issue #23104: [Bug]: WriteToBigQuery with file_loads and dynamic table destination doesn't load after first File Load
Posted by GitBox <gi...@apache.org>.
rizenfrmtheashes commented on issue #23104:
URL: https://github.com/apache/beam/issues/23104#issuecomment-1242197859
@ahmedabu98 Oh I absolutely missed this. I'm glad it's getting actioned. I took a look at the PR and it looks good at addressing the issue with the `WaitForBQJobs` single impulse, as well as removing the single impulse dependency on the tasks following. Looking forward to seeing its inclusion in the next release!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on issue #23104: [Bug]: WriteToBigQuery with file_loads and dynamic table destination doesn't load after first File Load
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #23104:
URL: https://github.com/apache/beam/issues/23104#issuecomment-1242183797
Label "awaiting cannot be managed because it does not exist in the repo. Please check your spelling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #23104: [Bug]: WriteToBigQuery with file_loads and dynamic table destination doesn't load after first File Load
Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #23104:
URL: https://github.com/apache/beam/issues/23104#issuecomment-1242187456
.remove-labels 'awaiting triage'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on issue #23104: [Bug]: WriteToBigQuery with file_loads and dynamic table destination doesn't load after first File Load
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on issue #23104:
URL: https://github.com/apache/beam/issues/23104#issuecomment-1245722081
Thanks for taking a look! I'm looking to get it merged before the end of the week, though I'm not sure if it will make version 2.42.0 as that release branch has already been cut.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on issue #23104: [Bug]: WriteToBigQuery with file_loads and dynamic table destination doesn't load after first File Load
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on issue #23104:
URL: https://github.com/apache/beam/issues/23104#issuecomment-1242098027
Thanks @Abacn
Thank you @rizenfrmtheashes for that document, it was really helpful in understanding the underlying issues. I'm working on a solution in #23012 (currently just writing tests and changing other tests that relied on the previous implementation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #23104: [Bug]: WriteToBigQuery with file_loads and dynamic table destination doesn't load after first File Load
Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #23104:
URL: https://github.com/apache/beam/issues/23104#issuecomment-1242090237
CC: @ahmedabu98 who is currently working on fixing this
.remove-labels "awaiting triage"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey closed issue #23104: [Bug]: WriteToBigQuery with file_loads and dynamic table destination doesn't load after first File Load
Posted by GitBox <gi...@apache.org>.
johnjcasey closed issue #23104: [Bug]: WriteToBigQuery with file_loads and dynamic table destination doesn't load after first File Load
URL: https://github.com/apache/beam/issues/23104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org