You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/22 18:54:12 UTC
[GitHub] [beam] yeandy opened a new pull request, #22001: Move model class outside of run() function in pytorch_language_modeling example
yeandy opened a new pull request, #22001:
URL: https://github.com/apache/beam/pull/22001
`dill` was bumped to 0.3.5.1 last month. With this version, there were no issues pickling the classes defined in `pytorch_language_modeling.py` during development. However, since we only recently reverted `dill` back to `0.3.1.1`, and haven't had a chance to test `pytorch_language_modeling.py` till `apache_beam==2.40.0rc1` was released, we didn't realize that dill version `0.3.1.1` has issues with pickling some of the classes. See error [here](https://gist.github.com/yeandy/4c51243213e24eef9417ff386b24cd0e)
**Solution**: Move the wrapper class outside of the `run()` function. This example now works for the current dill package `dill==0.3.1.1`.
Note: I tested a hacky solution by adding the following code to `pytorch_language_modeling.py`.
```
from apache_beam.internal import pickler
pickler.set_library('cloudpickle')
```
But this is not good practice, so I went with the change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #22001:
URL: https://github.com/apache/beam/pull/22001#issuecomment-1163490232
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #22001:
URL: https://github.com/apache/beam/pull/22001#issuecomment-1163509424
# [Codecov](https://codecov.io/gh/apache/beam/pull/22001?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#22001](https://codecov.io/gh/apache/beam/pull/22001?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (56c6820) into [master](https://codecov.io/gh/apache/beam/commit/45bebebdafe7c8df707259ca11c5633aebdb3933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (45bebeb) will **decrease** coverage by `0.00%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #22001 +/- ##
==========================================
- Coverage 73.99% 73.99% -0.01%
==========================================
Files 703 703
Lines 92934 92934
==========================================
- Hits 68769 68765 -4
- Misses 22899 22903 +4
Partials 1266 1266
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.57% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/22001?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...am/examples/inference/pytorch\_language\_modeling.py](https://codecov.io/gh/apache/beam/pull/22001/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvaW5mZXJlbmNlL3B5dG9yY2hfbGFuZ3VhZ2VfbW9kZWxpbmcucHk=) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/22001/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `93.05% <0.00%> (-0.39%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/22001/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.30% <0.00%> (-0.38%)` | :arrow_down: |
| [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/22001/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `94.02% <0.00%> (+0.74%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/22001?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/22001?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [45bebeb...56c6820](https://codecov.io/gh/apache/beam/pull/22001?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22001:
URL: https://github.com/apache/beam/pull/22001#issuecomment-1163495547
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #22001:
URL: https://github.com/apache/beam/pull/22001#issuecomment-1163490231
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #22001:
URL: https://github.com/apache/beam/pull/22001#issuecomment-1163490227
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on PR #22001:
URL: https://github.com/apache/beam/pull/22001#issuecomment-1164529832
if you intend to try to CP this into the release, you will need another PR for the release branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn merged pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
tvalentyn merged PR #22001:
URL: https://github.com/apache/beam/pull/22001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #22001:
URL: https://github.com/apache/beam/pull/22001#issuecomment-1163490235
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #22001:
URL: https://github.com/apache/beam/pull/22001#issuecomment-1163490229
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yeandy commented on pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
yeandy commented on PR #22001:
URL: https://github.com/apache/beam/pull/22001#issuecomment-1163494533
R: @ryanthompson591 @AnandInguva
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yeandy commented on pull request #22001: Move model class outside of run() function in pytorch_language_modeling example
Posted by GitBox <gi...@apache.org>.
yeandy commented on PR #22001:
URL: https://github.com/apache/beam/pull/22001#issuecomment-1164541729
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org