You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "chamikaramj (via GitHub)" <gi...@apache.org> on 2023/11/14 00:31:22 UTC
[I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
chamikaramj opened a new issue, #29424:
URL: https://github.com/apache/beam/issues/29424
### What happened?
I recently upgraded to Mac OS Sonoma and seems like x-lang jobs are getting stuck during job submission when run with this OS.
The job seems to be getting stuck when connecting to Java expansion service using gRPC (see below for the stack-trace). I tried following scenarios:
* Running job with Beam HEAD on Mac OS Sonoma - job gets stuck
* Running job with Beam 2.51.0 and dependencies constrained to ```https://github.com/apache/beam/blob/release-2.51.0/sdks/python/container/py38/base_image_requirements.txt``` on Mac OS Sonoma - job gets stuck
* Running job with Beam HEAD on Linux (Debian) - job runs fine
```
INFO:apache_beam.utils.subprocess_server:INFO: Registering external transforms: [beam:transform:org.apache.beam:kafka_read_with_metadata:v1, beam:transform:org.apache.beam:kafka_read_without_metadata:v1, beam:transform:org.apache.beam:kafka_write:v1, beam:external:java:generate_sequence:v1]
INFO:apache_beam.utils.subprocess_server:
INFO:apache_beam.utils.subprocess_server:Registered transforms:
INFO:apache_beam.utils.subprocess_server: beam:transform:org.apache.beam:kafka_read_with_metadata:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@71075444
INFO:apache_beam.utils.subprocess_server: beam:transform:org.apache.beam:kafka_read_without_metadata:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@4f32a3ad
INFO:apache_beam.utils.subprocess_server: beam:transform:org.apache.beam:kafka_write:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@6b695b06
INFO:apache_beam.utils.subprocess_server: beam:external:java:generate_sequence:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@4d1bf319
INFO:apache_beam.utils.subprocess_server:
INFO:apache_beam.utils.subprocess_server:Registered SchemaTransformProviders:
INFO:apache_beam.utils.subprocess_server: beam:schematransform:org.apache.beam:kafka_read:v1
INFO:apache_beam.utils.subprocess_server: beam:schematransform:org.apache.beam:kafka_write:v1
INFO:apache_beam.utils.subprocess_server:Nov 13, 2023 4:23:36 PM org.apache.beam.sdk.expansion.service.ExpansionService expand
INFO:apache_beam.utils.subprocess_server:INFO: Expanding 'WriteToKafka(beam:transform:org.apache.beam:kafka_write:v1)' with URN 'beam:transform:org.apache.beam:kafka_write:v1'
INFO:apache_beam.utils.subprocess_server:Dependencies list: {}
INFO:apache_beam.utils.subprocess_server:Nov 13, 2023 4:23:36 PM org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader payloadToConfig
INFO:apache_beam.utils.subprocess_server:WARNING: Configuration class 'org.apache.beam.sdk.io.kafka.KafkaIO$Write$External$Configuration' has no schema registered. Attempting to construct with setter approach.
INFO:apache_beam.utils.subprocess_server:Nov 13, 2023 4:23:36 PM org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader payloadToConfig
INFO:apache_beam.utils.subprocess_server:WARNING: Configuration class 'org.apache.beam.sdk.io.kafka.KafkaIO$Write$External$Configuration' has no schema registered. Attempting to construct with setter approach.
/Users/chamikara/testing/xlang_job_stuck/env_2_51_0_with_constraints/lib/python3.8/site-packages/apache_beam/transforms/external.py:725: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self._expansion_service, pipeline.options)
```
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1809350608
cc: @tvalentyn @johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1828401866
Thanks for rootcausing and fixing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1816417668
@Polber has experienced and debugged this for a bit too -- even found a hacky workaround, I think he could expand more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1816887046
Thanks. This makes me think that the issue might be bug related to Python subprocess module's interaction with the Mac OS Sonoma.
For the record, I did confirm that removing usage of PIPE **does not** solve the issue (basically re-introducing https://github.com/apache/beam/commit/081986e0e96e36ad9d00c0712622f08feeded228).
For now, my hypothesis is that this a a bug related to Mac OS Sonoma and Python. Mac OS Sonoma was just released two months ago. So bugs can be expected here I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1823501440
Yes, seems like x-lang auto-expansion service is currently broken for Sonoma. As a workaround, Sonoma users can start a manual expansion service till we have the fix in Beam 2.53.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1823499002
Seems like closing the gRPC channel started for the expansion service fixes the issue: https://github.com/apache/beam/pull/29524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1810416490
cc @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1811640777
I tried running py-spy on the stuck process and it produced the following stack-trace.
```
Thread 0x7FF85E4B6B40 (idle): "MainThread"
_execute_child (subprocess.py:1639)
__init__ (subprocess.py:858)
run (subprocess.py:493)
check_output (subprocess.py:415)
_run_subprocess_ignore_stderr (auth/_cloud_sdk.py:88)
get_project_id (auth/_cloud_sdk.py:106)
_get_gcloud_sdk_credentials (auth/_default.py:242)
<lambda> (auth/_default.py:651)
default (auth/_default.py:657)
_get_credentials_with_retrys (apache_beam/internal/gcp/auth.py:173)
wrapper (apache_beam/utils/retry.py:275)
_get_service_credentials (apache_beam/internal/gcp/auth.py:153)
get_service_credentials (apache_beam/internal/gcp/auth.py:136)
get_service_credentials (apache_beam/internal/gcp/auth.py:82)
__init__ (apache_beam/runners/dataflow/internal/apiclient.py:500)
run_pipeline (apache_beam/runners/dataflow/dataflow_runner.py:471)
run (apache_beam/pipeline.py:585)
__exit__ (apache_beam/pipeline.py:608)
run (apache_beam/examples/kafkataxi/kafka_taxi.py:94)
<module> (apache_beam/examples/kafkataxi/kafka_taxi.py:169)
_run_code (runpy.py:87)
_run_module_as_main (runpy.py:194)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1816800051
@chamikaramj I still don't know the root cause, but I found a way to sort of "trick" it into working (the hacky workaround Bruno mentioned)
If you are running in PyCharm or any IDE with a debugger, put a breakpoint on line 133 of `apache_beam/subprocess/subprocess_server.py` which is the return statement for `start_process()`. Then, run the xlang pipeline in debug mode, let the process stop on that breakpoint, then resume the process (it will stop on that breakpoint twice per xlang transform). It should run fine after resuming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1823435323
I'm running into the same issue, and having trouble finding a smaller repro, but can confirm it is hanging on seemingly random attempts to fork subprocesses.
Does this mean cross language is broken for all users on the newest version of OS-X?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1811625208
I added some logging and seems like the job is actually getting stuck when trying to retrieve local credentials at following line (past expansion service).
https://github.com/apache/beam/blob/a687c7d41df5d25917cbef28b437fda8e0ebffb7/sdks/python/apache_beam/internal/gcp/auth.py#L172
So this is unlikely to be an x-lang only issue.
I tried to create a local reproduction from a Python script outside the pipeline but that script retrieved local credentials correctly. So I don't have a small reproduction for this yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1816911100
I can also confirm that my xlang environment was working perfectly fine all last week until I updated to Sonoma.
The issue is complicated, because I tried reproducing by calling a similar codepath in a test script and could not trigger the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1829131539
@chamikaramj Great work, thank you! Really curious to see how you got to the bottom of this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1811685829
So strangely it seems to be getting stuck trying to get the project using gcloud at following location.
https://github.com/googleapis/google-auth-library-python/blob/6c610a564e833545837991f473b98139740b8866/google/auth/_cloud_sdk.py#L106
The command:
('gcloud', 'config', 'get', 'project')
I upgrade gcloud to the latest version but it didn't fix the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj closed issue #29424: [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma
URL: https://github.com/apache/beam/issues/29424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1830481632
Just a local easy reproduction (that makes the process stuck) and going through the code for any issues during expansion service creation :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org