You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "DerRidda (via GitHub)" <gi...@apache.org> on 2024/04/05 13:08:02 UTC

[I] [Bug]: [beam]

DerRidda opened a new issue, #30867:
URL: https://github.com/apache/beam/issues/30867

   ### What happened?
   
   Update of the Python `grpcio` dependency to version 1.62.1 caused Dataflow job stalling, with excessive waits for responses in GRCP Multi-threaded rendezvous probably somewhere in SDK worker. Upstream issue exists here: https://github.com/grpc/grpc/issues/36256
   
   I could determine that at least version 1.60.0 doesn't exhibit this issue.
   
   While this is not an issue with the Beam SDK itself a potential mitigation could be to limit future versions of the SDK to versions known to not be affected until upstream releases a fix.
   
   Tested on SDK versions 2.53.0, 2.55.0, 2.55.1rc2
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [X] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [X] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2048463960

   follow up investigation: ssh into the VM:
   
   ```
   gcloud compute ssh --zone "us-central1-b" "beamapp-valentyn-04101932-04101232-3i3x-harness-3sqg" --project "apache-beam-testing"
   ```
   
   ```
   curl localhost:8081/sdk_status 
   ```
   is unresponsive.
   
   However, we can ssh into the docker container and use pystack:
   
   ```
   valentyn@...harness-3sqg ~ $ docker exec --privileged -it 1142b2c7eb3c /bin/bash
   pip install pystack
   ps -A
   
   root@beamapp-valentyn-04101932-04101232-3i3x-harness-3sqg:/# ps -A
       PID TTY          TIME CMD
         1 ?        00:00:00 boot
        29 ?        00:03:02 python
        30 ?        00:02:46 python
       114 pts/0    00:00:00 bash
       125 pts/0    00:00:00 ps
   
   pystack remote 29
   ```
   
   Which reveals:
   
   ```
   docker exec --privileged -it 1142b2c7eb3c /bin/bash
   root@beamapp-valentyn-04101932-04101232-3i3x-harness-3sqg:/# pystack remote 29
   Traceback for thread 100 (python) [Has the GIL] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 80, in _worker
           work_item.run()
       (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
           result = self.fn(*self.args, **self.kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/batcher.py", line 385, in _flush_rows
           response = self.table.mutate_rows(rows_to_flush)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 724, in mutate_rows
           self.name,
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 160, in name
           table_client = self._instance._client.table_data_client
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 332, in table_data_client
           transport = self._create_gapic_client_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 285, in _create_gapic_client_channel
           channel = grpc_transport.create_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable_v2/services/bigtable/transports/grpc.py", line 217, in create_channel
           return grpc_helpers.create_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 386, in create_channel
           return grpc.secure_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/__init__.py", line 2119, in secure_channel
           return _channel.Channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 2046, in __init__
           self._channel = cygrpc.Channel(
   
   Traceback for thread 99 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 80, in _worker
           work_item.run()
       (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
           result = self.fn(*self.args, **self.kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/batcher.py", line 385, in _flush_rows
           response = self.table.mutate_rows(rows_to_flush)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 724, in mutate_rows
           self.name,
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 160, in name
           table_client = self._instance._client.table_data_client
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 332, in table_data_client
           transport = self._create_gapic_client_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 285, in _create_gapic_client_channel
           channel = grpc_transport.create_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable_v2/services/bigtable/transports/grpc.py", line 217, in create_channel
           return grpc_helpers.create_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 386, in create_channel
           return grpc.secure_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/__init__.py", line 2119, in secure_channel
           return _channel.Channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 2046, in __init__
           self._channel = cygrpc.Channel(
       (Python) File "/usr/local/lib/python3.8/pkgutil.py", line 638, in get_data
           return loader.get_data(resource_name)
       (Python) File "<frozen importlib._bootstrap_external>", line 1033, in get_data
   
   Traceback for thread 98 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 1252, in run
           self.finished.wait(self.interval)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 558, in wait
           signaled = self._cond.wait(timeout)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 306, in wait
           gotit = waiter.acquire(True, timeout)
   
   Traceback for thread 97 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
   
   Traceback for thread 95 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/data_plane.py", line 686, in <lambda>
           target=lambda: self._read_inputs(elements_iterator),
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/data_plane.py", line 669, in _read_inputs
           for elements in elements_iterator:
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 542, in __next__
           return self._next()
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 959, in _next
           _common.wait(self._state.condition.wait, _response_ready)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_common.py", line 156, in wait
           _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_common.py", line 116, in _wait_once
           wait_fn(timeout=timeout)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 306, in wait
           gotit = waiter.acquire(True, timeout)
   
   Traceback for thread 94 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 275, in consume_request_iterator
           request = next(request_iterator)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/data_plane.py", line 608, in _write_outputs
           streams = [self._to_send.get()]
       (Python) File "/usr/local/lib/python3.8/queue.py", line 170, in get
           self.not_empty.wait()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 302, in wait
           waiter.acquire()
   
   Traceback for thread 93 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 1728, in channel_spin
           event = state.channel.next_call_event()
   
   Traceback for thread 92 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 1060, in pull_responses
           for response in responses:
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 542, in __next__
           return self._next()
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 959, in _next
           _common.wait(self._state.condition.wait, _response_ready)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_common.py", line 156, in wait
           _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_common.py", line 116, in _wait_once
           wait_fn(timeout=timeout)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 306, in wait
           gotit = waiter.acquire(True, timeout)
   
   Traceback for thread 91 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 275, in consume_request_iterator
           request = next(request_iterator)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 1050, in request_iter
           request = self._requests.get()
       (Python) File "/usr/local/lib/python3.8/queue.py", line 170, in get
           self.not_empty.wait()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 302, in wait
           waiter.acquire()
   
   Traceback for thread 90 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 1728, in channel_spin
           event = state.channel.next_call_event()
   
   Traceback for thread 89 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/utils/thread_pool_executor.py", line 53, in run
           self._work_item.run()
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/utils/thread_pool_executor.py", line 37, in run
           self._future.set_result(self._fn(*self._fn_args, **self._fn_kwargs))
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 385, in task
           self._execute(
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 311, in _execute
           response = task()
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 386, in <lambda>
           lambda: self.create_worker().do_instruction(request), request)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 650, in do_instruction
           return getattr(self, request_type)(
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 688, in process_bundle
           bundle_processor.process_bundle(instruction_id))
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1113, in process_bundle
           input_op_by_transform_id[element.transform_id].process_encoded(
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 237, in process_encoded
           self.output(decoded_value)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/io/gcp/bigtableio.py", line 156, in process
           self.batcher.mutate(row)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/batcher.py", line 266, in mutate
           self._flush_async()
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/batcher.py", line 340, in _flush_async
           future = self._executor.submit(self._flush_rows, rows_to_flush)
       (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 188, in submit
           self._adjust_thread_count()
       (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 213, in _adjust_thread_count
           t.start()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 857, in start
           self._started.wait()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 558, in wait
           signaled = self._cond.wait(timeout)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 302, in wait
           waiter.acquire()
   
   Traceback for thread 88 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 275, in consume_request_iterator
           request = next(request_iterator)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 258, in get_responses
           response = self._responses.get()
       (Python) File "/usr/local/lib/python3.8/queue.py", line 170, in get
           self.not_empty.wait()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 302, in wait
           waiter.acquire()
   
   Traceback for thread 87 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 1728, in channel_spin
           event = state.channel.next_call_event()
   
   Traceback for thread 83 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 275, in consume_request_iterator
           request = next(request_iterator)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/worker_status.py", line 190, in _get_responses
           response = self._responses.get()
       (Python) File "/usr/local/lib/python3.8/queue.py", line 170, in get
           self.not_empty.wait()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 302, in wait
           waiter.acquire()
   
   Traceback for thread 82 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 1728, in channel_spin
           event = state.channel.next_call_event()
   
   Traceback for thread 81 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/worker_status.py", line 182, in <lambda>
           target=lambda: self._log_lull_in_bundle_processor(
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/worker_status.py", line 232, in _log_lull_in_bundle_processor
           time.sleep(2 * 60)
   
   Traceback for thread 80 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/worker_status.py", line 177, in <lambda>
           target=lambda: self._serve(), name='fn_api_status_handler')
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/worker_status.py", line 198, in _serve
           for request in self._status_stub.WorkerStatus(self._get_responses()):
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 542, in __next__
           return self._next()
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 959, in _next
           _common.wait(self._state.condition.wait, _response_ready)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_common.py", line 156, in wait
           _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_common.py", line 116, in _wait_once
           wait_fn(timeout=timeout)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 306, in wait
           gotit = waiter.acquire(True, timeout)
   
   Traceback for thread 72 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/data_plane.py", line 224, in run
           while not self._finished.wait(next_call - time.time()):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 558, in wait
           signaled = self._cond.wait(timeout)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 306, in wait
           gotit = waiter.acquire(True, timeout)
   
   Traceback for thread 65 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 275, in consume_request_iterator
           request = next(request_iterator)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/log_handler.py", line 182, in _write_log_entries
           log_entries = [self._log_entry_queue.get()]
       (Python) File "/usr/local/lib/python3.8/queue.py", line 170, in get
           self.not_empty.wait()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 302, in wait
           waiter.acquire()
   
   Traceback for thread 63 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 1728, in channel_spin
           event = state.channel.next_call_event()
   
   Traceback for thread 60 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/log_handler.py", line 99, in <lambda>
           target=lambda: self._read_log_control_messages(),
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/log_handler.py", line 216, in _read_log_control_messages
           for _ in log_control_iterator:
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 542, in __next__
           return self._next()
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 959, in _next
           _common.wait(self._state.condition.wait, _response_ready)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_common.py", line 156, in wait
           _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_common.py", line 116, in _wait_once
           wait_fn(timeout=timeout)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 306, in wait
           gotit = waiter.acquire(True, timeout)
   
   Traceback for thread 29 (python) [] (most recent call last):
       (Python) File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
           return _run_code(code, main_globals, None,
       (Python) File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
           exec(code, run_globals)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 367, in <module>
           main(sys.argv)
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 212, in main
           sdk_harness.run()
       (Python) File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 266, in run
           for work_request in self._control_stub.Control(get_responses()):
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 542, in __next__
           return self._next()
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 959, in _next
           _common.wait(self._state.condition.wait, _response_ready)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_common.py", line 156, in wait
           _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_common.py", line 116, in _wait_once
           wait_fn(timeout=timeout)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 306, in wait
           gotit = waiter.acquire(True, timeout)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2046219427

   also what are the exact errors you are seeing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2040640597

   Ok. Thank you very much for reporting the issue, please let us know if you have more information, that might also help grpcio maintainers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2048032874

   I am observing the pattern that the jobs you start with Beam 2.55.0 SDK have many errors like "Unable to retrieve status info from SDK harness". This is definitely concerning. The message means that Beam SDK process running inside a container, aka SDK harness,  becomes unresponsive to SDK Status requests from the runner; eventually, the runner terminates the unresponsive SDK, and worker might restart.
   
   These errors appear fairly early in pipeline execution. 
   
   Dataflow workers serve the SDK status page on localhost:8081/sdk_status, and it can be queried manually via: gcloud compute ssh --zone "xx-somezone-z" "some-dataflow-gce-worker-01300848-wqox-harness-bvf7" --project "some-project-id" --command "curl localhost:8081/sdk_status". 
   
   Would it be possible to take a closer look at the differences between 2.55.0 and 2.53.0 setup that you have to narrow down the exact change that increases instances of these errors? For example: upgrading/downgrading a dependency X and doing nothing else increases/decreases instances of this error.  I'll also try to repro this issue myself. 
   
   > It also seems to manifest in the Google Cloud Console Dataflow Job viewer UI locking up in the browser until the browser considers tab unresponsive while a fixed job stays responsive..
   
   That is likely an unrelated UI issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2048591086

   looks like `bigtableio_it_test.py::BigtableIOWriteTest::test_bigtable_write` was recently re-enabled: https://github.com/apache/beam/commit/33a0bfb5d374fae99b1f19405aba341a0716b1c0 , and it was known to be flaky before, so it might be a separate issue specific to bigtable client. We should investigate it but it would also help to confirm that this stuckness is the same as what we have in https://github.com/apache/beam/issues/30867#issuecomment-2047005415 . 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2057396240

   I am not certain they are the same bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "DerRidda (via GitHub)" <gi...@apache.org>.
DerRidda commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2055821059

   @tvalentyn Sorry I didn't reply sooner but I can't really siphon of the time to repro this issue more on Dataflow. Seeing your latest comments in the grpcio bug report I assume you have what you need now, though?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2040544276

   Thanks.
   
   ```
   (py38) :python$ docker run --rm -it --entrypoint=/bin/sh apache/beam_python3.10_sdk:2.54.0  -c "pip list | grep grpcio" 
   grpcio                          1.60.0
   grpcio-status                   1.60.0
   (py38) :python$ docker run --rm -it --entrypoint=/bin/sh apache/beam_python3.10_sdk:2.55.0  -c "pip list | grep grpcio" 
   grpcio                          1.62.0
   grpcio-status                   1.62.0
   (py38) :python$ docker run --rm -it --entrypoint=/bin/sh apache/beam_python3.10_sdk:2.55.1  -c "pip list | grep grpcio" 
   grpcio                          1.62.0
   grpcio-status                   1.62.0
   ```
   
   > I could determine that at least version 1.60.0 doesn't exhibit this issue.
   
   Double checking, is this still the case?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2058072150

   There is a confirmed issue https://github.com/googleapis/python-bigtable/issues/949 that affects google-api-core and grpcio library, which caused a regression in Apache Beam 2.55.0.  It will be fixed in the upcoming release of grpcio and mitigated in Beam 2.56.0.
   
   For affected 2.55.0 users, any of the following  mitigations should help:
   - downgrade to apache-beam==2.54.0 or below
   
   Install any of the the following dependency combinations in the Beam pipeline runtime environment (For example, you can use a --requirements_file pipeline option):
   
   - upgrade grpcio and grpcio-status to version 1.63.0 or above (currently only 1.63.0rc1 is available) OR
   - downgrade google-api-core to version 2.16.2 or below  OR
   - downgrade grpcio and grpcio-status to 1.58.0 or below.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2039855556

   @tvalentyn FYI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2048473295

   TLDR is that a thread that executes bigtable/transports/grpc.create_channel() later calls into likely a python extension  cygrpc.Channel(), which holds GIL indefinitely, so other threads cannot run, and we hence SDK is not responsive on /sdk_status RPC calls. https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact also explains this failure mode
    
   ```
   Traceback for thread 100 (python) [Has the GIL] (most recent call last):
   ...
           transport = self._create_gapic_client_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 285, in _create_gapic_client_channel
           channel = grpc_transport.create_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable_v2/services/bigtable/transports/grpc.py", line 217, in create_channel
           return grpc_helpers.create_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 386, in create_channel
           return grpc.secure_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/__init__.py", line 2119, in secure_channel
           return _channel.Channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 2046, in __init__
           self._channel = cygrpc.Channel(
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2048596523

   @DerRidda would it be possible for you to re-run your job using Beam 2.55 SDK, then find a stuck worker VM, and retrieve stacktraces with pystack as I did in: https://github.com/apache/beam/issues/30867#issuecomment-2048463960 ?  Note: you might have to use Dataflow Classic, I am not certain if SSHing into workers is possible with Dataflow Prime.
   
   To find a Stuck VM, look for "Unable to retrieve status info from SDK harness." logs, then find which worker emits those logs by expanding the logging entry in Cloud Logging. you might see something like:
   
   ```
   {
   insertId: "7918190538793288800:164250:0:18926"
   jsonPayload: {
   line: "fnapi_harness_status_service.cc:212"
   message: "Unable to retrieve status info from SDK harness sdk-0-0_sibling_1 within allowed time."
   thread: "115"
   }
   labels: {
   compute.googleapis.com/resource_id: "7918190538793288800"
   compute.googleapis.com/resource_name: "df-<some_identifier>-harness-9zz3"
   compute.googleapis.com/resource_type: "instance"
   ```
   
   the `df-<some_identifier>-harness-9zz3` would be the VM name
   
   Then, SSH to that VM from UI or via a gcloud command, log into the running python container in a privileged mode and run pystack. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2048144737

   I see an instance of "Unable to retrieve status info from SDK harness"  integration test executions of some beam tests, for example   apache_beam.examples.cookbook.bigtableio_it_test.BigtableIOWriteTest , it might be reproducible in other beam tests as well; we should be able to use these for repro.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2048454045

   I was able to repro the issue with a couple of execution of an integration test:
   
   ```
   python -m pytest  -o log_cli=True -o log_level=Info apache_beam/examples/cookbook/bigtableio_it_test.py::BigtableIOWriteTest::test_bigtable_write   --test-pipeline-options='--runner=TestDataflowRunner --project=apache-beam-testing --temp_location=valentyn-testing  --region=us-central1 --wait_until_finish_duration=36000000' --timeout=36000
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2059740318

   > @tvalentyn is the fix here basically just disallowing 1.59.0-1.62.1 in
   
   We have at least two changes in that depend on new versions of GRPCIO, we'd have to roll them back. I am talking to GRPC IO folks if they could help with a patch release.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "DerRidda (via GitHub)" <gi...@apache.org>.
DerRidda commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2059239874

   So, I removed all other dependency pins and just updated grpcio to 1.63.0rc1 (didn't even do grpcio-status in the beginning) on SDK version 2.55.1 with [the patch ](https://github.com/apache/beam/pull/30679) manually applied to rule that out as a potential reason for pipeline stalling.
   
   So far I can no longer repro the issue, my job performs normally under load. I now also added the grpcio-status pin and updated in place, will check once the job is back to expected full scaling but I don't expect issues here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #30867: [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow
URL: https://github.com/apache/beam/issues/30867


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2048477298

   With `--native-all`, pystack adds a bit details, and also shows that other threads are waiting on getting GIL:
   
   ```
   root@beamapp-valentyn-04101932-04101232-3i3x-harness-3sqg:/# pystack remote 29 --native-all
   Traceback for thread 100 (python) [Has the GIL] (most recent call last):
       (C) File "Python/thread_pthread.h", line 232, in pythread_wrapper (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "./Modules/_threadmodule.c", line 1002, in t_bootstrap (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 80, in _worker
           work_item.run()
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
           result = self.fn(*self.args, **self.kwargs)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/batcher.py", line 385, in _flush_rows
           response = self.table.mutate_rows(rows_to_flush)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 724, in mutate_rows
           self.name,
       (C) File "Objects/descrobject.c", line 1496, in property_descr_get (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 160, in name
           table_client = self._instance._client.table_data_client
       (C) File "Objects/descrobject.c", line 1496, in property_descr_get (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 332, in table_data_client
           transport = self._create_gapic_client_channel(
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 285, in _create_gapic_client_channel
           channel = grpc_transport.create_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable_v2/services/bigtable/transports/grpc.py", line 217, in create_channel
           return grpc_helpers.create_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 386, in create_channel
           return grpc.secure_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/__init__.py", line 2119, in secure_channel
           return _channel.Channel(
       (C) File "Objects/typeobject.c", line 6790, in slot_tp_init (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 2046, in __init__
           self._channel = cygrpc.Channel(
       (C) File "???", line 0, in syscall (/usr/lib/x86_64-linux-gnu/libc.so.6)
   
   Traceback for thread 99 (python) [Waiting for the GIL] (most recent call last):
       (C) File "Python/thread_pthread.h", line 232, in pythread_wrapper (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "./Modules/_threadmodule.c", line 1002, in t_bootstrap (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
           self._target(*self._args, **self._kwargs)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 80, in _worker
           work_item.run()
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
           result = self.fn(*self.args, **self.kwargs)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/batcher.py", line 385, in _flush_rows
           response = self.table.mutate_rows(rows_to_flush)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 724, in mutate_rows
           self.name,
       (C) File "Objects/descrobject.c", line 1496, in property_descr_get (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 160, in name
           table_client = self._instance._client.table_data_client
       (C) File "Objects/descrobject.c", line 1496, in property_descr_get (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 332, in table_data_client
           transport = self._create_gapic_client_channel(
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 285, in _create_gapic_client_channel
           channel = grpc_transport.create_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable_v2/services/bigtable/transports/grpc.py", line 217, in create_channel
           return grpc_helpers.create_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 386, in create_channel
           return grpc.secure_channel(
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/__init__.py", line 2119, in secure_channel
           return _channel.Channel(
       (C) File "Objects/typeobject.c", line 6790, in slot_tp_init (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 2046, in __init__
           self._channel = cygrpc.Channel(
       (Python) File "/usr/local/lib/python3.8/pkgutil.py", line 638, in get_data
           return loader.get_data(resource_name)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "<frozen importlib._bootstrap_external>", line 1033, in get_data
       (C) File "./Modules/_io/clinic/bufferedio.c.h", line 174, in _io__Buffered_read (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "./Modules/_io/bufferedio.c", line 891, in _io__Buffered_read_impl (inlined) (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "./Modules/_io/bufferedio.c", line 1541, in _bufferedreader_read_all (inlined) (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "./Modules/_io/fileio.c", line 706, in _io_FileIO_readall_impl (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Python/ceval_gil.h", line 206, in take_gil (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Python/condvar.h", line 73, in PyCOND_TIMEDWAIT (inlined) (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "???", line 0, in pthread_cond_timedwait (/usr/lib/x86_64-linux-gnu/libc.so.6)
   
   Traceback for thread 98 (python) [Waiting for the GIL] (most recent call last):
       (C) File "Python/thread_pthread.h", line 232, in pythread_wrapper (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "./Modules/_threadmodule.c", line 1002, in t_bootstrap (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
           self._bootstrap_inner()
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
           self.run()
       (C) File "Objects/call.c", line 284, in function_code_fastcall (/usr/local/lib/libpython3.8.so.1.0)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 1252, in run
           self.finished.wait(self.interval)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 558, in wait
           signaled = self._cond.wait(timeout)
       (Python) File "/usr/local/lib/python3.8/threading.py", line 306, in wait
           gotit = waiter.acquire(True, timeout)
       (C) File "./Modules/_threadmodule.c", line 146, in lock_PyThread_acquire_lock (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "./Modules/_threadmodule.c", line 64, in acquire_timed (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Python/ceval_gil.h", line 206, in take_gil (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "Python/condvar.h", line 73, in PyCOND_TIMEDWAIT (inlined) (/usr/local/lib/libpython3.8.so.1.0)
       (C) File "???", line 0, in pthread_cond_timedwait (/usr/lib/x86_64-linux-gnu/libc.so.6)
   
   ```  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "DerRidda (via GitHub)" <gi...@apache.org>.
DerRidda commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2040607446

   > Thanks.
   > 
   > ```
   > (py38) :python$ docker run --rm -it --entrypoint=/bin/sh apache/beam_python3.10_sdk:2.53.0  -c "pip list | grep grpcio" 
   > grpcio                          1.59.3
   > grpcio-status                   1.59.3
   > (py38) :python$ docker run --rm -it --entrypoint=/bin/sh apache/beam_python3.10_sdk:2.54.0  -c "pip list | grep grpcio" 
   > grpcio                          1.60.0
   > grpcio-status                   1.60.0
   > (py38) :python$ docker run --rm -it --entrypoint=/bin/sh apache/beam_python3.10_sdk:2.55.0  -c "pip list | grep grpcio" 
   > grpcio                          1.62.0
   > grpcio-status                   1.62.0
   > (py38) :python$ docker run --rm -it --entrypoint=/bin/sh apache/beam_python3.10_sdk:2.55.1  -c "pip list | grep grpcio" 
   > grpcio                          1.62.0
   > grpcio-status                   1.62.0
   > ```
   > 
   > > I could determine that at least version 1.60.0 doesn't exhibit this issue.
   > 
   > Double checking, is this still the case?
   
   Actually still happens on 1.60.0 for me. It just looked fixed for a brief moment.
   The known good version of my pipeline probably uses 1.59.3 as I changed the SDK container image to be fully custom as well in the update so even though the project local deps were showing 1.60.0 it never got installed into the image of the known good service. I will investigate more on Monday.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "DerRidda (via GitHub)" <gi...@apache.org>.
DerRidda commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2047005415

   @tvalentyn Job ID: 2024-04-03_08_14_02-12227946365357908481 in europe-west4
   I saw failures to obtain SDK worker status which might actually be the symptom of https://github.com/apache/beam/pull/30679 but when I manually applied the fix for that I still saw completely stuck workers, sometimes without seeing any feedback from the Dataflow backend, just nothing happening. Maybe occasionally an SDK worker went away again without much for me to go on in terms of the immediately visible Dataflow worker logs or parsing through Logs Explorer more in depth.
   
   It also seems to manifest in the Google Cloud Console Dataflow Job viewer UI locking up in the browser until the browser considers tab unresponsive while a fixed job stays responsive..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2046067630

   Can you share Dataflow Job IDs where you've seen this error?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2057505210

   we can fix the bug we are currently investigating and then come back to your issue and see if you still reproduce it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2040558042

   cc: @damccorm 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2059261252

   @tvalentyn is the fix here basically just disallowing 1.59.0-1.62.1 in https://github.com/apache/beam/blob/e59d313c3b17aaf45176ed5035068f01858809a0/sdks/python/setup.py#L368 ? Branch cut is tomorrow, so I'd love to get this in today if we can


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2059697427

   @DerRidda Great to hear! grpcio-status change is to make `pip check` happy for the downgrade scenario.
   @damccorm yes, a PR  is coming shortly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2048620271

   given that across all our test suite I only see the error in Bitgtable tests reliably, I filed a separate issue: https://github.com/apache/beam/issues/30927.
   
   Let's try to get separate stacktraces for stuck processes for the pipelines discussed earlier in this bug.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "DerRidda (via GitHub)" <gi...@apache.org>.
DerRidda commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2046007949

   @tvalentyn Some further investigation might point at a combination of the protobuf library version with grpcio: https://github.com/grpc/grpc/issues/36256#issuecomment-2046006931


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: GRCPIO 1.62.1 deadlocks in Beam SDK worker with Dataflow [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30867:
URL: https://github.com/apache/beam/issues/30867#issuecomment-2048048914

   one other thing I would try: use Beam 2.55.0 setup but downgrade protobuf to "protbuf==3.20.3"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org