You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "rakeshcusat (via GitHub)" <gi...@apache.org> on 2025/06/15 21:05:07 UTC
[PR] BEAM-9626: Make Pymongo package optional - 20259 [beam]
rakeshcusat opened a new pull request, #35297:
URL: https://github.com/apache/beam/pull/35297
More information could be found here:
[BEAM-9626](https://issues.apache.org/jira/browse/BEAM-9626)
https://github.com/apache/beam/issues/20259
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Move out optional packages from core dependencies - 20259 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35297:
URL: https://github.com/apache/beam/pull/35297#issuecomment-3249107732
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Make Pymongo package optional - 20259 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35297:
URL: https://github.com/apache/beam/pull/35297#issuecomment-2974728550
Assigning reviewers:
R: @liferoad for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Make Pymongo package optional - 20259 [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35297:
URL: https://github.com/apache/beam/pull/35297#discussion_r2150080688
##########
sdks/python/setup.py:
##########
@@ -378,7 +378,6 @@ def get_portability_package_data():
'numpy>=1.14.3,<2.3.0', # Update pyproject.toml as well.
'objsize>=0.6.1,<0.8.0',
'packaging>=22.0',
- 'pymongo>=3.8.0,<5.0.0',
Review Comment:
I like this change, thanks. Up to you on whether you want to take this on, but I think we would benefit from doing this with more dependencies. Specifically, the following are used very lightly and are not needed for core functionality:
- hdfs (would need a new extra)
- jsonschema (just used in yaml, could be moved to that extra)
- pydot (just used in interactive, could be moved to that extra)
- redis
> I also feel that we should update the [CHANGES.md](https://github.com/apache/beam/blob/master/CHANGES.md) file to mention this change.
It is breaking, so I agree - could you please do this as part of this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Move out optional packages from core dependencies - 20259 [beam]
Posted by "rakeshcusat (via GitHub)" <gi...@apache.org>.
rakeshcusat commented on code in PR #35297:
URL: https://github.com/apache/beam/pull/35297#discussion_r2153681386
##########
sdks/python/setup.py:
##########
@@ -391,10 +388,8 @@ def get_portability_package_data():
# 3. Exclude protobuf 4 versions that leak memory, see:
# https://github.com/apache/beam/issues/28246
'protobuf>=3.20.3,<6.0.0.dev0,!=4.0.*,!=4.21.*,!=4.22.0,!=4.23.*,!=4.24.*', # pylint: disable=line-too-long
- 'pydot>=1.2.0,<2',
Review Comment:
@damccorm what is the easiest way to find the mapping of test environment and the tox.ini section? This could help me to easily identify the broken environment and fix it. Otherwise, it will take longer time.
Also, I realized that `jsonschema` is part of core and can't be moved to extra section of `yaml`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Make Pymongo package optional - 20259 [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35297:
URL: https://github.com/apache/beam/pull/35297#discussion_r2152843557
##########
CHANGES.md:
##########
@@ -76,7 +76,10 @@
## Breaking Changes
-* X behavior was changed ([#X](https://github.com/apache/beam/issues/X)).
+* X behavior was changed ([#X](https://github.com/apache/beam/issues/X)).=======
+* Yapf version upgraded to 0.43.0 for formatting (Python) ([#34801](https://github.com/apache/beam/pull/34801/)).
+* Extra packages moved out of `install_required` to individual extra sections (`hdfs`, `mongodb`, `redis`) as part of [#35297](https://github.com/apache/beam/pull/35297)
Review Comment:
Could we mention that the others have been moved to separate (existing) extras as well?
##########
sdks/python/setup.py:
##########
@@ -391,10 +388,8 @@ def get_portability_package_data():
# 3. Exclude protobuf 4 versions that leak memory, see:
# https://github.com/apache/beam/issues/28246
'protobuf>=3.20.3,<6.0.0.dev0,!=4.0.*,!=4.21.*,!=4.22.0,!=4.23.*,!=4.24.*', # pylint: disable=line-too-long
- 'pydot>=1.2.0,<2',
Review Comment:
Looks like some of these are causing failures because the appropriate extras aren't installed when running the test - could you take a look? (or feel free to revert/simplify to a smaller set of dependencies)
##########
sdks/python/setup.py:
##########
@@ -391,10 +388,8 @@ def get_portability_package_data():
# 3. Exclude protobuf 4 versions that leak memory, see:
# https://github.com/apache/beam/issues/28246
'protobuf>=3.20.3,<6.0.0.dev0,!=4.0.*,!=4.21.*,!=4.22.0,!=4.23.*,!=4.24.*', # pylint: disable=line-too-long
- 'pydot>=1.2.0,<2',
Review Comment:
I see you started on this, I think its probably just adding interactive/yaml to a few tox suites
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Move out optional packages from core dependencies - 20259 [beam]
Posted by "rakeshcusat (via GitHub)" <gi...@apache.org>.
rakeshcusat commented on code in PR #35297:
URL: https://github.com/apache/beam/pull/35297#discussion_r2153271740
##########
CHANGES.md:
##########
@@ -76,7 +76,10 @@
## Breaking Changes
-* X behavior was changed ([#X](https://github.com/apache/beam/issues/X)).
+* X behavior was changed ([#X](https://github.com/apache/beam/issues/X)).=======
+* Yapf version upgraded to 0.43.0 for formatting (Python) ([#34801](https://github.com/apache/beam/pull/34801/)).
+* Extra packages moved out of `install_required` to individual extra sections (`hdfs`, `mongodb`, `redis`) as part of [#35297](https://github.com/apache/beam/pull/35297)
Review Comment:
Good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Move out optional packages from core dependencies - 20259 [beam]
Posted by "rakeshcusat (via GitHub)" <gi...@apache.org>.
rakeshcusat commented on code in PR #35297:
URL: https://github.com/apache/beam/pull/35297#discussion_r2153273295
##########
sdks/python/setup.py:
##########
@@ -391,10 +388,8 @@ def get_portability_package_data():
# 3. Exclude protobuf 4 versions that leak memory, see:
# https://github.com/apache/beam/issues/28246
'protobuf>=3.20.3,<6.0.0.dev0,!=4.0.*,!=4.21.*,!=4.22.0,!=4.23.*,!=4.24.*', # pylint: disable=line-too-long
- 'pydot>=1.2.0,<2',
Review Comment:
yes, I am not done with this change yet. I will update the tox suites soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Move out optional packages from core dependencies - 20259 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #35297: BEAM-9626: Move out optional packages from core dependencies - 20259
URL: https://github.com/apache/beam/pull/35297
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Move out optional packages from core dependencies - 20259 [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #35297:
URL: https://github.com/apache/beam/pull/35297#issuecomment-2987126116
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/35297?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 54.51%. Comparing base [(`cecfa61`)](https://app.codecov.io/gh/apache/beam/commit/cecfa6128ceaecf3b8d02a8a42788b3505595845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`ccc51b0`)](https://app.codecov.io/gh/apache/beam/commit/ccc51b0acee6b3f04be8a0a3262bec0d9d591e28?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 37 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #35297 +/- ##
==========================================
Coverage 54.50% 54.51%
Complexity 1559 1559
==========================================
Files 1035 1036 +1
Lines 161595 161757 +162
Branches 1139 1139
==========================================
+ Hits 88084 88175 +91
- Misses 71380 71451 +71
Partials 2131 2131
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/35297/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/apache/beam/pull/35297/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.83% <100.00%> (-0.06%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/35297?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Move out optional packages from core dependencies - 20259 [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35297:
URL: https://github.com/apache/beam/pull/35297#discussion_r2155141138
##########
sdks/python/setup.py:
##########
@@ -391,10 +388,8 @@ def get_portability_package_data():
# 3. Exclude protobuf 4 versions that leak memory, see:
# https://github.com/apache/beam/issues/28246
'protobuf>=3.20.3,<6.0.0.dev0,!=4.0.*,!=4.21.*,!=4.22.0,!=4.23.*,!=4.24.*', # pylint: disable=line-too-long
- 'pydot>=1.2.0,<2',
Review Comment:
> would this line (Task :sdks:python:test-suites:tox:py312:testPython312 ) in the log good enough to identify the environment? or would you choose a different strategy to identify the environment?
Yeah, that's what I usually use, and then you can look at the gradle task. An alternative at the failed test log, you can see something like:
```
________________________ YamlMappingTest.test_validate _________________________
[gw1] linux -- Python 3.9.22 /runner/_work/beam/beam/sdks/python/test-suites/tox/py39/build/srcs/sdks/python/target/.tox-py39-cloud/py39-cloud/bin/python
...
```
Tox will create a folder with the task name (in this case py39-cloud) as part of the path name. So in this case, the task is py39-cloud - https://github.com/apache/beam/blob/9d75411a3bd085351e6283b556e067fa4e467b35/sdks/python/tox.ini#L99
You'll also see this task show up elsewhere in the logs (e.g. tox-py39-cloud shows up a bunch of places). An llm probably could figure it out too, but I haven't tried it yet :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Move out optional packages from core dependencies - 20259 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35297:
URL: https://github.com/apache/beam/pull/35297#issuecomment-3224038505
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Make Pymongo package optional - 20259 [beam]
Posted by "rakeshcusat (via GitHub)" <gi...@apache.org>.
rakeshcusat commented on PR #35297:
URL: https://github.com/apache/beam/pull/35297#issuecomment-2974693705
PTAL @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Make Pymongo package optional - 20259 [beam]
Posted by "rakeshcusat (via GitHub)" <gi...@apache.org>.
rakeshcusat commented on code in PR #35297:
URL: https://github.com/apache/beam/pull/35297#discussion_r2151412550
##########
sdks/python/setup.py:
##########
@@ -378,7 +378,6 @@ def get_portability_package_data():
'numpy>=1.14.3,<2.3.0', # Update pyproject.toml as well.
'objsize>=0.6.1,<0.8.0',
'packaging>=22.0',
- 'pymongo>=3.8.0,<5.0.0',
Review Comment:
Thank you for highlighting others. I was just thinking about Redis. I can take care of them in this changeset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BEAM-9626: Move out optional packages from core dependencies - 20259 [beam]
Posted by "rakeshcusat (via GitHub)" <gi...@apache.org>.
rakeshcusat commented on code in PR #35297:
URL: https://github.com/apache/beam/pull/35297#discussion_r2153691810
##########
sdks/python/setup.py:
##########
@@ -391,10 +388,8 @@ def get_portability_package_data():
# 3. Exclude protobuf 4 versions that leak memory, see:
# https://github.com/apache/beam/issues/28246
'protobuf>=3.20.3,<6.0.0.dev0,!=4.0.*,!=4.21.*,!=4.22.0,!=4.23.*,!=4.24.*', # pylint: disable=line-too-long
- 'pydot>=1.2.0,<2',
Review Comment:
would this line (`Task :sdks:python:test-suites:tox:py312:testPython312` ) in the log good enough to identify the environment? or would you choose a different strategy to identify the environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org