You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/11/09 20:12:55 UTC

[GitHub] [beam] AnandInguva opened a new pull request, #24069: [Python]Set pickle library at the Pipeline creation stage

AnandInguva opened a new pull request, #24069:
URL: https://github.com/apache/beam/pull/24069

   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309566529

   if it happens, it is also possible that we may have to set this flag in multiple places (which is less ideal)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309328724

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn merged pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
tvalentyn merged PR #24069:
URL: https://github.com/apache/beam/pull/24069


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1310466534

   R: @tvalentyn


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309340895

   > This is great if it works.
   > 
   > If I recall correctly, I tried a similar fix in the past and it did not work.
   > 
   > The way to know if you got it right (where I didn't) is to raise an exception in dill_pickler on any call. Then run the pipeline. If the dill pickler is never called then you know this workaround worked.
   > 
   > I think what was happening is before the pipeline **init** method got called, some of the transforms were pickling themselves.
   
   I can run some pipelines with raised exception on dill, on portable runners to see if it works


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1310679166

   Thanks, @AnandInguva !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309602175

   or we can do it during the instantiation of pipeline options. Let me try that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309565670

   I don't know off the top of my head when that can happen. But it's possible to detect that this happens and print a warning. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309327612

   R: @ryanthompson591 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309362379

   I was able to run word count example with this change using cloudpickle along with raised exception on the dill dumps call. @tvalentyn do you suspect pickling happens before beam.Pipeline init call?
   ![image](https://user-images.githubusercontent.com/34158215/200939327-e680f4e3-0aef-422a-a55f-59f7981a2ad0.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309620140

   I wouldn't do that in PipelineOptions


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309659519

   I think this line snippet would be sufficient. I ran the wordcount example and one RunInference example using cloudpickle . Worked fine. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309665292

   # [Codecov](https://codecov.io/gh/apache/beam/pull/24069?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#24069](https://codecov.io/gh/apache/beam/pull/24069?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ff96665) into [master](https://codecov.io/gh/apache/beam/commit/539fa9159ffc116b2e79e6de2804dfdd1c1e4722?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (539fa91) will **decrease** coverage by `0.00%`.
   > The diff coverage is `100.00%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #24069      +/-   ##
   ==========================================
   - Coverage   73.30%   73.30%   -0.01%     
   ==========================================
     Files         714      714              
     Lines       96418    96420       +2     
   ==========================================
     Hits        70679    70679              
   - Misses      24418    24420       +2     
     Partials     1321     1321              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.19% <100.00%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/24069?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/24069/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `92.27% <100.00%> (+0.02%)` | :arrow_up: |
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/24069/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/24069/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.47% <0.00%> (-1.39%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/24069/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.50% <0.00%> (-1.27%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/24069/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.42% <0.00%> (+0.12%)` | :arrow_up: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/24069/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.89% <0.00%> (+0.13%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/24069/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `93.43% <0.00%> (+0.38%)` | :arrow_up: |
   | [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/24069/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `75.51% <0.00%> (+2.04%)` | :arrow_up: |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
ryanthompson591 commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309696538

   Where I was seeing it was PTransformWithSideInputs.
   
   https://github.com/apache/beam/blob/4522f4ce916223afb3c4edd02a7be6018d0ce5a4/sdks/python/apache_beam/transforms/ptransform.py#L864
   
   If your code happens before that is called then I think we are golden.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309735945

   > Where I was seeing it was PTransformWithSideInputs.
   > 
   > https://github.com/apache/beam/blob/4522f4ce916223afb3c4edd02a7be6018d0ce5a4/sdks/python/apache_beam/transforms/ptransform.py#L864
   > 
   > If your code happens before that is called then I think we are golden.
   
   Do you have an example that we can test it with?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #24069: [Python]Set pickle library at the Pipeline creation stage

Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #24069:
URL: https://github.com/apache/beam/pull/24069#issuecomment-1309737222

   ParDo instantiates from this class and in the wordcount example, we use a ParDo in the pipeline. So this should work.
   https://github.com/apache/beam/blob/4522f4ce916223afb3c4edd02a7be6018d0ce5a4/sdks/python/apache_beam/transforms/core.py#L1386


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org