You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tvalentyn (via GitHub)" <gi...@apache.org> on 2024/04/10 23:57:17 UTC
[I] [Failing Test]: BigtableIOWriteTest::test_bigtable_write is about 50% flaky [beam]
tvalentyn opened a new issue, #30927:
URL: https://github.com/apache/beam/issues/30927
### What happened?
From https://github.com/apache/beam/issues/30867#issuecomment-2048454045 and below, it appears that BigTable client initialization sometimes gets stuck and holding GIL indefinitely in:
```
Traceback for thread 100 (python) [Has the GIL] (most recent call last):
(Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
(Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
(Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
(Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 80, in _worker
work_item.run()
(Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
(Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/batcher.py", line 385, in _flush_rows
response = self.table.mutate_rows(rows_to_flush)
(Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 724, in mutate_rows
self.name,
(Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 160, in name
table_client = self._instance._client.table_data_client
(Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 332, in table_data_client
transport = self._create_gapic_client_channel(
(Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 285, in _create_gapic_client_channel
channel = grpc_transport.create_channel(
(Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable_v2/services/bigtable/transports/grpc.py", line 217, in create_channel
return grpc_helpers.create_channel(
(Python) File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 386, in create_channel
return grpc.secure_channel(
(Python) File "/usr/local/lib/python3.8/site-packages/grpc/__init__.py", line 2119, in secure_channel
return _channel.Channel(
(Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 2046, in __init__
self._channel = cygrpc.Channel(
```
Repro:
```
python -m pytest -o log_cli=True -o log_level=Info apache_beam/examples/cookbook/bigtableio_it_test.py::BigtableIOWriteTest::test_bigtable_write --test-pipeline-options='--runner=TestDataflowRunner --project=apache-beam-testing --temp_location=valentyn-testing --region=us-central1 --wait_until_finish_duration=36000000' --timeout=36000
```
I wasn't able to repro the error locally outside of Dataflow context.
cc: @mutianf FYI given that BT client is involved. It might be a wider issue, as discussed in https://github.com/apache/beam/issues/30867 and https://github.com/grpc/grpc/issues/36256.
### Issue Failure
Failure: Test is flaky
### Issue Priority
Priority: 2 (backlog / disabled test but we think the product is healthy)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: BigtableIOWriteTest::test_bigtable_write is about 50% flaky [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #30927:
URL: https://github.com/apache/beam/issues/30927#issuecomment-2049395180
@mutianf Can you check this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: BigtableIOWriteTest::test_bigtable_write is about 50% flaky [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30927:
URL: https://github.com/apache/beam/issues/30927#issuecomment-2070970289
fixed in https://github.com/apache/beam/pull/31044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: BigtableIOWriteTest::test_bigtable_write is about 50% flaky [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30927:
URL: https://github.com/apache/beam/issues/30927#issuecomment-2052729627
gdb backtrace of the stuck that thread holds up GIL: https://github.com/grpc/grpc/issues/36256#issuecomment-2052729097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: BigtableIOWriteTest::test_bigtable_write is about 50% flaky [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #30927: [Failing Test]: BigtableIOWriteTest::test_bigtable_write is about 50% flaky
URL: https://github.com/apache/beam/issues/30927
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org