You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "chamikaramj (via GitHub)" <gi...@apache.org> on 2023/11/14 00:31:22 UTC

[I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

chamikaramj opened a new issue, #29424:
URL: https://github.com/apache/beam/issues/29424

   ### What happened?
   
   I recently upgraded to Mac OS Sonoma and seems like x-lang jobs are getting stuck during job submission when run with this OS.
   
   The job seems to be getting stuck when connecting to Java expansion service using gRPC (see below for the stack-trace). I tried following scenarios:
   
   * Running job with Beam HEAD on Mac OS Sonoma - job gets stuck
   * Running job with Beam 2.51.0 and dependencies constrained to ```https://github.com/apache/beam/blob/release-2.51.0/sdks/python/container/py38/base_image_requirements.txt``` on Mac OS Sonoma - job gets stuck
   * Running job with Beam HEAD on Linux (Debian) - job runs fine
   
   ```
   INFO:apache_beam.utils.subprocess_server:INFO: Registering external transforms: [beam:transform:org.apache.beam:kafka_read_with_metadata:v1, beam:transform:org.apache.beam:kafka_read_without_metadata:v1, beam:transform:org.apache.beam:kafka_write:v1, beam:external:java:generate_sequence:v1]
   INFO:apache_beam.utils.subprocess_server:
   INFO:apache_beam.utils.subprocess_server:Registered transforms:
   INFO:apache_beam.utils.subprocess_server:	beam:transform:org.apache.beam:kafka_read_with_metadata:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@71075444
   INFO:apache_beam.utils.subprocess_server:	beam:transform:org.apache.beam:kafka_read_without_metadata:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@4f32a3ad
   INFO:apache_beam.utils.subprocess_server:	beam:transform:org.apache.beam:kafka_write:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@6b695b06
   INFO:apache_beam.utils.subprocess_server:	beam:external:java:generate_sequence:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@4d1bf319
   INFO:apache_beam.utils.subprocess_server:
   INFO:apache_beam.utils.subprocess_server:Registered SchemaTransformProviders:
   INFO:apache_beam.utils.subprocess_server:	beam:schematransform:org.apache.beam:kafka_read:v1
   INFO:apache_beam.utils.subprocess_server:	beam:schematransform:org.apache.beam:kafka_write:v1
   INFO:apache_beam.utils.subprocess_server:Nov 13, 2023 4:23:36 PM org.apache.beam.sdk.expansion.service.ExpansionService expand
   INFO:apache_beam.utils.subprocess_server:INFO: Expanding 'WriteToKafka(beam:transform:org.apache.beam:kafka_write:v1)' with URN 'beam:transform:org.apache.beam:kafka_write:v1'
   INFO:apache_beam.utils.subprocess_server:Dependencies list: {}
   INFO:apache_beam.utils.subprocess_server:Nov 13, 2023 4:23:36 PM org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader payloadToConfig
   INFO:apache_beam.utils.subprocess_server:WARNING: Configuration class 'org.apache.beam.sdk.io.kafka.KafkaIO$Write$External$Configuration' has no schema registered. Attempting to construct with setter approach.
   INFO:apache_beam.utils.subprocess_server:Nov 13, 2023 4:23:36 PM org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader payloadToConfig
   INFO:apache_beam.utils.subprocess_server:WARNING: Configuration class 'org.apache.beam.sdk.io.kafka.KafkaIO$Write$External$Configuration' has no schema registered. Attempting to construct with setter approach.
   /Users/chamikara/testing/xlang_job_stuck/env_2_51_0_with_constraints/lib/python3.8/site-packages/apache_beam/transforms/external.py:725: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
     self._expansion_service, pipeline.options)
   ```
   
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1809350608

   cc: @tvalentyn @johnjcasey 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1828401866

   Thanks for rootcausing and fixing this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1816417668

   @Polber has experienced and debugged this for a bit too -- even found a hacky workaround, I think he could expand more
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1816887046

   Thanks. This makes me think that the issue might be bug related to Python subprocess module's interaction with the Mac OS Sonoma.
   
   For the record, I did confirm that removing usage of PIPE **does not** solve the issue (basically re-introducing https://github.com/apache/beam/commit/081986e0e96e36ad9d00c0712622f08feeded228).
   
   For now, my hypothesis is that this a a bug related to Mac OS Sonoma and Python. Mac OS Sonoma was just released two months ago. So bugs can be expected here I guess.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1823501440

   Yes, seems like x-lang auto-expansion service is currently broken for Sonoma. As a workaround, Sonoma users can start a manual expansion service till we have the fix in Beam 2.53.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1823499002

   Seems like closing the gRPC channel started for the expansion service fixes the issue: https://github.com/apache/beam/pull/29524


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1810416490

   cc @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1811640777

   I tried running py-spy on the stuck process and it produced the following stack-trace.
   
   ```
   Thread 0x7FF85E4B6B40 (idle): "MainThread"
       _execute_child (subprocess.py:1639)
       __init__ (subprocess.py:858)
       run (subprocess.py:493)
       check_output (subprocess.py:415)
       _run_subprocess_ignore_stderr (auth/_cloud_sdk.py:88)
       get_project_id (auth/_cloud_sdk.py:106)
       _get_gcloud_sdk_credentials (auth/_default.py:242)
       <lambda> (auth/_default.py:651)
       default (auth/_default.py:657)
       _get_credentials_with_retrys (apache_beam/internal/gcp/auth.py:173)
       wrapper (apache_beam/utils/retry.py:275)
       _get_service_credentials (apache_beam/internal/gcp/auth.py:153)
       get_service_credentials (apache_beam/internal/gcp/auth.py:136)
       get_service_credentials (apache_beam/internal/gcp/auth.py:82)
       __init__ (apache_beam/runners/dataflow/internal/apiclient.py:500)
       run_pipeline (apache_beam/runners/dataflow/dataflow_runner.py:471)
       run (apache_beam/pipeline.py:585)
       __exit__ (apache_beam/pipeline.py:608)
       run (apache_beam/examples/kafkataxi/kafka_taxi.py:94)
       <module> (apache_beam/examples/kafkataxi/kafka_taxi.py:169)
       _run_code (runpy.py:87)
       _run_module_as_main (runpy.py:194)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1816800051

   @chamikaramj I still don't know the root cause, but I found a way to sort of "trick" it into working (the hacky workaround Bruno mentioned)
   
   If you are running in PyCharm or any IDE with a debugger, put a breakpoint on line 133 of `apache_beam/subprocess/subprocess_server.py` which is the return statement for `start_process()`. Then, run the xlang pipeline in debug mode, let the process stop on that breakpoint, then resume the process (it will stop on that breakpoint twice per xlang transform). It should run fine after resuming.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1823435323

   I'm running into the same issue, and having trouble finding a smaller repro, but can confirm it is hanging on seemingly random attempts to fork subprocesses. 
   
   Does this mean cross language is broken for all users on the newest version of OS-X?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1811625208

   I added some logging and seems like the job is actually getting stuck when trying to retrieve local credentials at following line (past expansion service).
   
   https://github.com/apache/beam/blob/a687c7d41df5d25917cbef28b437fda8e0ebffb7/sdks/python/apache_beam/internal/gcp/auth.py#L172
   
   So this is unlikely to be an x-lang only issue.
   
   I tried to create a local reproduction from a Python script outside the pipeline but that script retrieved local credentials correctly. So I don't have a small reproduction for this yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1816911100

   I can also confirm that my xlang environment was working perfectly fine all last week until I updated to Sonoma.
   
   The issue is complicated, because I tried reproducing by calling a similar codepath in a test script and could not trigger the behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1829131539

   @chamikaramj Great work, thank you! Really curious to see how you got to the bottom of this 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1811685829

   So strangely it seems to be getting stuck trying to get the project using gcloud at following location.
   
   https://github.com/googleapis/google-auth-library-python/blob/6c610a564e833545837991f473b98139740b8866/google/auth/_cloud_sdk.py#L106
   
   The command:
   ('gcloud', 'config', 'get', 'project')
   
   I upgrade gcloud to the latest version but it didn't fix the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj closed issue #29424: [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma 
URL: https://github.com/apache/beam/issues/29424


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python x-lang jobs seems to be getting stuck when run from Mac OS Sonoma [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #29424:
URL: https://github.com/apache/beam/issues/29424#issuecomment-1830481632

   Just a local easy reproduction (that makes the process stuck) and going through the code for any issues during expansion service creation :)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org