[GitHub] [beam] tc5613213 opened a new issue, #22808: [Bug]: Dataflow job stuckness related to big query storage api

[GitBox <gi...@apache.org>]

tc5613213 opened a new issue, #22808:
URL: https://github.com/apache/beam/issues/22808

   ### What happened?
   
   Hi team,
   
   My team has a dataflow job using python sdk that writes pubsub messages to multiple big query tables (inside the pipeline it will also create table using bigquery client if destinating table does not exist). Currently we have increasing among of times where the job got stuck with this error message: `operation ongoing for over 189595.35 seconds in state process-msecs in step Stream to BigQuery-ptransform-23  without returning`.
   
   Error logs:
   ` File "/usr/local/lib/python3.7/threading.py", line 890, in _bootstrap
       self._bootstrap_inner()
   
     File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
       self.run()
   
     File "/usr/local/lib/python3.7/site-packages/apache_beam/utils/thread_pool_executor.py", line 53, in run
       self._work_item.run()
   
     File "/usr/local/lib/python3.7/site-packages/apache_beam/utils/thread_pool_executor.py", line 37, in run
       self._future.set_result(self._fn(*self._fn_args, **self._fn_kwargs))
   
     File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in task
       lambda: self.create_worker().do_instruction(request), request)
   
     File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
       response = task()
   
     File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
       lambda: self.create_worker().do_instruction(request), request)
   
     File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 598, in do_instruction
       getattr(request, request_type), request.instruction_id)
   
     File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
       bundle_processor.process_bundle(instruction_id))
   
     File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1004, in process_bundle
       element.data)
   
     File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
       self.output(decoded_value)
   
     File "/usr/local/lib/python3.7/site-packages/apache_beam/transforms/core.py", line 1920, in <lambda>
       wrapper = lambda x: [fn(*x)]
   
     File "/dataflow/template/utils/create_table_insert.py", line 56, in append_rows_pending
       parent=parent, write_stream=write_stream
   
     File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery_storage_v1/services/big_query_write/client.py", line 573, in create_write_stream
       metadata=metadata,
   
     File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
       return wrapped_func(*args, **kwargs)
   
     File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 291, in retry_wrapped_func
       on_error=on_error,
   
     File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 189, in retry_target
       return target()
   
     File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
       return callable_(*args, **kwargs)
   
     File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 945, in __call__
       wait_for_ready, compression)
   
     File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 933, in _blocking
       event = call.next_event()`
       
       
       What is the possible reason for this? 
   
   ### Issue Priority
   
   Priority: 1
   
   ### Issue Component
   
   Component: runner-dataflow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on issue #22808: [Bug]: Dataflow job stuckness related to big query storage api

[GitBox <gi...@apache.org>]

TheNeuralBit commented on issue #22808:
URL: https://github.com/apache/beam/issues/22808#issuecomment-1224279137

   Is `/dataflow/template/utils/create_table_insert.py` your code? I don't think that's a beam file. Could you use BigQueryIO instead? 
   
   I'm going to lower priority on this since as of now it doesn't look like an issue in Beam.
   
   You might also try contacting GCP support since you're having an issue with interfacing multiple GCP products (PubSub, Dataflow, BigQuery). They can inspect your pipeline and provide advice. I'd also suggest sharing `create_table_insert.py`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org