You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "damccorm (via GitHub)" <gi...@apache.org> on 2025/06/30 17:17:27 UTC
[I] [Feature Request]: Add set of ML Beam containers [beam]
damccorm opened a new issue, #35487:
URL: https://github.com/apache/beam/issues/35487
### What would you like to happen?
Full context here - https://docs.google.com/document/d/1JcVFJsPbVvtvaYdGi-DzWy9PIIYJhL7LwWGEXt2NZMk/edit?tab=t.0 - we should add some containers for running Beam ML jobs.
Steps:
- [ ] Create requirements files for these containers. These should be auto-generated like we do for other containers today - https://github.com/apache/beam/blob/master/sdks/python/container/run_generate_requirements.sh - #35484
- [ ] Automatically stage and publish CPU ML containers alongside our existing containers. This should be able to mostly reuse the existing Dockerfile with a different set of requirements
- [ ] Automatically stage and publish GPU ML containers alongside our existing containers. This will require a different Dockerfile and likely will require building in a GPU environment (real or simulated)
- [ ] Allow users to specify `--sdk_container_image=ml` or `--sdk_container_image=gpu` and auto-resolve to the correct container.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add set of ML Beam containers [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #35487:
URL: https://github.com/apache/beam/issues/35487#issuecomment-3564314970
I still need to add `Allow users to specify --sdk_container_image=ml or --sdk_container_image=gpu and auto-resolve to the correct container.` - so I think we can leave it open for now (publishing the containers is only part of the task here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add set of ML Beam containers [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #35487:
URL: https://github.com/apache/beam/issues/35487#issuecomment-3564301354
ummm one issue can only have one milestone. Please check ML container publish status manually when release candidate containers are built. cc: @shunping @Amar3tto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add set of ML Beam containers [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #35487:
URL: https://github.com/apache/beam/issues/35487#issuecomment-3564298181
I understand this has been done? Though Beam 2.69 ML containers had some script issues. Put this on Beam 2.70 milestone to check again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add set of ML Beam containers [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #35487:
URL: https://github.com/apache/beam/issues/35487#issuecomment-4288655408
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org