[GitHub] [beam] y1chi commented on pull request #14877: [BEAM-12395] Add support to enable google cloud profiler for python S…

[GitBox <gi...@apache.org>]

y1chi commented on pull request #14877:
URL: https://github.com/apache/beam/pull/14877#issuecomment-877338303


   > Could you describe the benefit of adding Cloud Profiler? Does this enable automatic instrumentation of Beam pipelines running on GCP? My team would actually benefit from having that feature :)
   > 
   > FWIW, adding `google-cloud-profiler` makes using apache-beam within Docker more difficult.
   > 
   > `google-cloud-profiler` doesn't provide wheels, so upgrading `apache-beam` from 2.30.0 to 2.31.0 requires adding a compiler toolchain to the Docker build. To avoid bloating the final Docker image, that means setting up multi-stage builds in the Dockerfile. Multi-stage builds to build a Python library is a little obscure. The Cloud Profiler docs does have [good documentation for it](https://cloud.google.com/profiler/docs/profiling-python#running_with_linux_alpine), but it's a new step all Beam Docker users will need to deal with, and it's a bit annoying to jump through that hoop for a transitive dependency.
   > 
   > It would have been nice to have this change flagged in the release notes (https://github.com/apache/beam/releases/tag/v2.31.0).
   > 
   > I guess the _ideal_ solution here is for google-cloud-profiler to distribute wheels... I filed [GoogleCloudPlatform/cloud-profiler-python#98](https://github.com/GoogleCloudPlatform/cloud-profiler-python/issues/98)
   
   Yes, this will enable automatic instrumentation of beam pipeline on GCP when the pipeline options has '--experiments=enable_google_cloud_profiler', I'll add that to release note. The package should have been pre-installed if running GCP. Didn't notice that this will introduce complexity to docker user, thanks for creating the issue for google-cloud-profiler to distribute the wheel. We can remove the dependency from base requirement and maybe have user to provide that dependency through requirements if they indeed need this functionality.
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org