You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/02/28 19:32:10 UTC

[GitHub] [beam] ryanthompson591 opened a new pull request #16970: [BEAM-13982] A base class for run inference

ryanthompson591 opened a new pull request #16970:
URL: https://github.com/apache/beam/pull/16970


   Make a base class for a transform that run inferences.
   
   It will measure metrics, load models (using beam.shared) and batch inferences and run them.
   
   
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e00f05f) into [master](https://codecov.io/gh/apache/beam/commit/a1a63219780c22da8e3524cd9d4452c9431aabd6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1a6321) will **increase** coverage by `0.03%`.
   > The diff coverage is `92.18%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.03%   74.06%   +0.03%     
   ==========================================
     Files         672      673       +1     
     Lines       88284    88423     +139     
   ==========================================
   + Hits        65358    65494     +136     
   - Misses      21806    21809       +3     
     Partials     1120     1120              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.66% <92.18%> (+0.03%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.18% <92.18%> (ø)` | |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `90.00% <0.00%> (-1.00%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.96% <0.00%> (ø)` | |
   | [.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==) | `98.90% <0.00%> (+0.03%)` | :arrow_up: |
   | [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `94.91% <0.00%> (+0.05%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.64% <0.00%> (+0.24%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/external.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbC5weQ==) | `80.00% <0.00%> (+0.27%)` | :arrow_up: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.85% <0.00%> (+1.42%)` | :arrow_up: |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.36% <0.00%> (+1.63%)` | :arrow_up: |
   | ... and [1 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a1a6321...e00f05f](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0386e4f) into [master](https://codecov.io/gh/apache/beam/commit/62a661071b7db15e71d236abe68e15582e8997c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (62a6610) will **increase** coverage by `0.02%`.
   > The diff coverage is `92.18%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   73.94%   73.97%   +0.02%     
   ==========================================
     Files         669      670       +1     
     Lines       88158    88315     +157     
   ==========================================
   + Hits        65191    65329     +138     
   - Misses      21855    21874      +19     
     Partials     1112     1112              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.64% <92.18%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.18% <92.18%> (ø)` | |
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.37% <0.00%> (-4.66%)` | :arrow_down: |
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.68% <0.00%> (-2.44%)` | :arrow_down: |
   | [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `87.58% <0.00%> (-1.01%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `80.62% <0.00%> (-0.78%)` | :arrow_down: |
   | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.85% <0.00%> (-0.51%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/gcp/auth.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvZ2NwL2F1dGgucHk=) | `81.03% <0.00%> (-0.45%)` | :arrow_down: |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.32% <0.00%> (-0.26%)` | :arrow_down: |
   | [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `77.27% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.58% <0.00%> (-0.11%)` | :arrow_down: |
   | ... and [14 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [62a6610...0386e4f](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b91a5c7) into [master](https://codecov.io/gh/apache/beam/commit/a1a63219780c22da8e3524cd9d4452c9431aabd6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1a6321) will **increase** coverage by `0.07%`.
   > The diff coverage is `92.30%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.03%   74.10%   +0.07%     
   ==========================================
     Files         672      675       +3     
     Lines       88284    88554     +270     
   ==========================================
   + Hits        65358    65621     +263     
   - Misses      21806    21813       +7     
     Partials     1120     1120              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.68% <92.30%> (+0.05%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.30% <92.30%> (ø)` | |
   | [...apache\_beam/runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | `74.57% <0.00%> (-1.31%)` | :arrow_down: |
   | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.85% <0.00%> (-0.51%)` | :arrow_down: |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `87.90% <0.00%> (-0.42%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.09% <0.00%> (-0.31%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.90% <0.00%> (-0.16%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.96% <0.00%> (ø)` | |
   | [...ache\_beam/runners/dataflow/dataflow\_job\_service.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19qb2Jfc2VydmljZS5weQ==) | `62.16% <0.00%> (ø)` | |
   | [.../apache\_beam/runners/interactive/dataproc/types.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy90eXBlcy5weQ==) | `93.33% <0.00%> (ø)` | |
   | ... and [14 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a1a6321...b91a5c7](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b91a5c7) into [master](https://codecov.io/gh/apache/beam/commit/ca47cd83670a77507abcf5c54429966743af3ec0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ca47cd8) will **increase** coverage by `0.11%`.
   > The diff coverage is `92.30%`.
   
   > :exclamation: Current head b91a5c7 differs from pull request most recent head c4e1733. Consider uploading reports for the commit c4e1733 to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   73.98%   74.10%   +0.11%     
   ==========================================
     Files         674      675       +1     
     Lines       88558    88554       -4     
   ==========================================
   + Hits        65521    65621     +100     
   + Misses      21916    21813     -103     
   + Partials     1121     1120       -1     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.68% <92.30%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.30% <92.30%> (ø)` | |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `96.72% <0.00%> (-1.64%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.09% <0.00%> (-0.73%)` | :arrow_down: |
   | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.85% <0.00%> (-0.51%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/exec/fn.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbi5nbw==) | `69.26% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/typex/class.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2NsYXNzLmdv) | `85.38% <0.00%> (-0.12%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/util/reflectx/calls.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvY2FsbHMuZ28=) | `0.00% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/graphx/translate.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3RyYW5zbGF0ZS5nbw==) | `28.82% <0.00%> (+0.01%)` | :arrow_up: |
   | [sdks/go/pkg/beam/core/funcx/fn.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L2ZuLmdv) | `57.57% <0.00%> (+0.03%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `90.13% <0.00%> (+0.14%)` | :arrow_up: |
   | ... and [4 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [ca47cd8...c4e1733](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821740330



##########
File path: sdks/python/apache_beam/ml/inference/base_test.py
##########
@@ -0,0 +1,134 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from typing import Any
+from typing import Iterable
+import unittest
+
+import apache_beam as beam
+import apache_beam.ml.inference.base as base
+from apache_beam.metrics.metric import MetricsFilter
+from apache_beam.ml.inference.api import PredictionResult
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.testing.test_pipeline import TestPipeline
+
+
+class MockModel:

Review comment:
       Done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (37b6198) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.93%`.
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 37b6198 differs from pull request most recent head 5fd7fe0. Consider uploading reports for the commit 5fd7fe0 to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.62%   +8.93%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52253    -9442     
   + Misses      19898    10229    -9669     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.62% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.23% <0.00%> (-0.14%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.17% <0.00%> (-0.08%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/util/hooks/hooks.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaG9va3MvaG9va3MuZ28=) | | |
   | [sdks/go/pkg/beam/core/runtime/xlangx/namespace.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L25hbWVzcGFjZS5nbw==) | | |
   | [sdks/go/pkg/beam/option.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9vcHRpb24uZ28=) | | |
   | ... and [202 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...5fd7fe0](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821162358



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)
+    predictions = [
+        PredictionResult(e, r) for e, r in zip(examples, inference_generator)
+    ]
+    inference_latency = self._clock.get_current_time_in_microseconds(

Review comment:
       IDK, I think the code is pretty clear on its own in this case.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (37b6198) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.93%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.62%   +8.93%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52253    -9442     
   + Misses      19898    10229    -9669     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.62% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.23% <0.00%> (-0.14%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.17% <0.00%> (-0.08%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/metrics/sampler.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3Mvc2FtcGxlci5nbw==) | | |
   | [sdks/go/pkg/beam/metrics.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9tZXRyaWNzLmdv) | | |
   | [sdks/go/pkg/beam/schema.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9zY2hlbWEuZ28=) | | |
   | ... and [202 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...37b6198](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818640533



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.

Review comment:
       https://issues.apache.org/jira/browse/BEAM-14044
   
   Go ahead and take this when we are ready.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821762052



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    self._metrics_collector = MetricsCollector(
+        model_loader.get_metrics_namespace())
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()

Review comment:
       removed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (37b6198) into [master](https://codecov.io/gh/apache/beam/commit/62a661071b7db15e71d236abe68e15582e8997c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (62a6610) will **increase** coverage by `9.68%`.
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 37b6198 differs from pull request most recent head b888735. Consider uploading reports for the commit b888735 to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   73.94%   83.62%   +9.68%     
   ==========================================
     Files         669      453     -216     
     Lines       88158    62482   -25676     
   ==========================================
   - Hits        65191    52253   -12938     
   + Misses      21855    10229   -11626     
   + Partials     1112        0    -1112     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | go | `?` | |
   | python | `83.62% <ø> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `81.42% <0.00%> (-7.15%)` | :arrow_down: |
   | [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | `73.22% <0.00%> (-0.90%)` | :arrow_down: |
   | [.../internal/clients/dataflow/dataflow\_v1b3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9jbGllbnRzL2RhdGFmbG93L2RhdGFmbG93X3YxYjNfY2xpZW50LnB5) | `71.26% <0.00%> (-0.27%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.17% <0.00%> (-0.22%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `95.23% <0.00%> (-0.18%)` | :arrow_down: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.23% <0.00%> (-0.14%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.59% <0.00%> (-0.12%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `95.07% <0.00%> (-0.07%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <0.00%> (-0.04%)` | :arrow_down: |
   | [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.52% <0.00%> (-0.03%)` | :arrow_down: |
   | ... and [235 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [62a6610...b888735](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r840622351



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""

Review comment:
       I did, let me know if you like my changes.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818631338



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+

Review comment:
       I didn't add those metrics here, because I'm not tracking them. For example, some are not really relevant until we support side loading new models.
   
   How about we punt for now and I make a jira FR for those?
   https://issues.apache.org/jira/browse/BEAM-14043
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r819919945



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)

Review comment:
       The time saved is probably negligible, but should we put `start_time` and `inference_latency` around just `inference_generator`, instead of also around the manipulations w/ the keys and wrapping of `PredictionResult`? Technically, the operations w/ keys and `PredictionResult` are not related to the pure inference call.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

tvalentyn commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821110920



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+

Review comment:
       looks like it's 'run-inference'. Tracking issues as subtasks of BEAM-13970 works too.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

tvalentyn commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821314513



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)
+    predictions = [
+        PredictionResult(e, r) for e, r in zip(examples, inference_generator)
+    ]
+    inference_latency = self._clock.get_current_time_in_microseconds(

Review comment:
       +1, prefer to use comments to explain _why_ the code is doing something that's not obvious; no need to add comments to explain _what_ the code is doing (that's typically a sign of an unclear code that needs to be reorganized).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e00f05f) into [master](https://codecov.io/gh/apache/beam/commit/a1a63219780c22da8e3524cd9d4452c9431aabd6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1a6321) will **increase** coverage by `0.03%`.
   > The diff coverage is `92.18%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.03%   74.06%   +0.03%     
   ==========================================
     Files         672      673       +1     
     Lines       88284    88423     +139     
   ==========================================
   + Hits        65358    65494     +136     
   - Misses      21806    21809       +3     
     Partials     1120     1120              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.66% <92.18%> (+0.03%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.18% <92.18%> (ø)` | |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `90.00% <0.00%> (-1.00%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.96% <0.00%> (ø)` | |
   | [.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==) | `98.90% <0.00%> (+0.03%)` | :arrow_up: |
   | [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `94.91% <0.00%> (+0.05%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.64% <0.00%> (+0.24%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/external.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbC5weQ==) | `80.00% <0.00%> (+0.27%)` | :arrow_up: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.85% <0.00%> (+1.42%)` | :arrow_up: |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.36% <0.00%> (+1.63%)` | :arrow_up: |
   | ... and [1 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a1a6321...e00f05f](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0386e4f) into [master](https://codecov.io/gh/apache/beam/commit/62a661071b7db15e71d236abe68e15582e8997c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (62a6610) will **increase** coverage by `0.02%`.
   > The diff coverage is `92.18%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   73.94%   73.97%   +0.02%     
   ==========================================
     Files         669      670       +1     
     Lines       88158    88315     +157     
   ==========================================
   + Hits        65191    65329     +138     
   - Misses      21855    21874      +19     
     Partials     1112     1112              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.64% <92.18%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.18% <92.18%> (ø)` | |
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.37% <0.00%> (-4.66%)` | :arrow_down: |
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.68% <0.00%> (-2.44%)` | :arrow_down: |
   | [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `87.58% <0.00%> (-1.01%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `80.62% <0.00%> (-0.78%)` | :arrow_down: |
   | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.85% <0.00%> (-0.51%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/gcp/auth.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvZ2NwL2F1dGgucHk=) | `81.03% <0.00%> (-0.45%)` | :arrow_down: |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.32% <0.00%> (-0.26%)` | :arrow_down: |
   | [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `77.27% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.58% <0.00%> (-0.11%)` | :arrow_down: |
   | ... and [14 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [62a6610...0386e4f](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r840870284



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any
+from typing import Iterable
+from typing import Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+class InferenceRunner():
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[Any]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+  def get_num_bytes(self, batch: Any) -> int:
+    """Returns the number of bytes of data for a batch."""
+    return len(pickle.dumps(batch))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class ModelLoader():
+  """Has the ability to load an ML model."""
+  def load_model(self) -> Any:
+    """Loads and initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_inference_runner(self) -> InferenceRunner:
+    """Returns an implementation of InferenceRunner for this model."""
+    raise NotImplementedError(type(self))
+
+
+def _unbatch(maybe_keyed_batches: Tuple[Any, Any]):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class RunInference(beam.PTransform):
+  """An extensible transform for running inferences."""
+  def __init__(self, model_loader: ModelLoader, clock=None):
+    self._model_loader = model_loader
+    self._clock = clock
+
+  # TODO(BEAM-14208): Add batch_size back off in the case there
+  # are functional reasons large batch sizes cannot be handled.
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO(BEAM-14044): Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(shared.Shared(), self._model_loader, self._clock))

Review comment:
       Sorry, pushed that commit.
   
   While instantiating in the DoFn is simpler.  TFX-BSL was specifically passing it in. I think they base it on the shared sample code which explicitly passes in the shared object.
   
   https://beam.apache.org/releases/pydoc/2.32.0/apache_beam.utils.shared.html
   
   It contains this sentence in the doc:
   "Real-world uses will typically involve using a side-input to a DoFn to initialize the shared resource in a way that can’t be done with just its constructor."
   
   My understanding is that keeping it as a side input allows the DoFn to be cleaned up and respawned without recreating the shared handle. Am I mistaken?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] commented on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2cb876c) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.92%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.61%   +8.92%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52247    -9448     
   + Misses      19898    10235    -9663     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.61% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.01% <0.00%> (-1.39%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `80.39% <0.00%> (-0.86%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.48% <0.00%> (-0.76%)` | :arrow_down: |
   | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.85% <0.00%> (-0.51%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.90% <0.00%> (-0.16%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/coderx/int.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2ludC5nbw==) | | |
   | ... and [204 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...2cb876c](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r819923088



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace

Review comment:
       Will this be a future ticket? Or is it just a reminder for this PR? Maybe we could use`run_inference_metrics`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821867353



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    self._metrics_collector = MetricsCollector(
+        model_loader.get_metrics_namespace())
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    # Process supports both keyed data, and example only data.
+    # First keys and samples are separated (if there are keys)
+    has_keys = isinstance(batch[0], tuple)
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+
+    start_time = self._clock.get_current_time_in_microseconds()
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)
+    predictions = [
+        PredictionResult(e, r) for e, r in zip(examples, inference_generator)
+    ]
+
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+
+    # keys will be recombined with their predictions in the RunInferenceImpl PTransform.
+    yield keys, predictions
+
+  def finish_bundle(self):
+    self._metrics_collector.update_metrics_with_cache()
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._clock = clock
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO(BEAM-14044): Hook into the batching DoFn APIs.
+        | beam.BatchElements(min_batch_size=2)
+        | beam.ParDo(
+            RunInferenceDoFn(
+                self._model_loader, self._inference_runner, clock=self._clock))
+        | beam.FlatMap(_unbatch))
+
+
+def _is_darwin() -> bool:
+  return sys.platform == 'darwin'
+
+
+def _get_current_process_memory_in_bytes():
+  """Returns memory usage in bytes."""
+
+  if resource is not None:

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (37b6198) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.93%`.
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 37b6198 differs from pull request most recent head 1867029. Consider uploading reports for the commit 1867029 to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.62%   +8.93%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52253    -9442     
   + Misses      19898    10229    -9669     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.62% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.23% <0.00%> (-0.14%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.17% <0.00%> (-0.08%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/encoding.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9lbmNvZGluZy5nbw==) | | |
   | [sdks/go/pkg/beam/core/graph/fn.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2ZuLmdv) | | |
   | [sdks/go/pkg/beam/core/util/reflectx/types.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvdHlwZXMuZ28=) | | |
   | ... and [202 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...1867029](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821160506



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None

Review comment:
       Yeah, I added a few comments. The TFX one's logic is a little tricky to follow. Hopefully how keys are separated and recombined make sense now with a few more comments.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (37b6198) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.93%`.
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 37b6198 differs from pull request most recent head a1573da. Consider uploading reports for the commit a1573da to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.62%   +8.93%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52253    -9442     
   + Misses      19898    10229    -9669     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.62% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.23% <0.00%> (-0.14%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.17% <0.00%> (-0.08%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/util/protox/protox.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcHJvdG94L3Byb3RveC5nbw==) | | |
   | [...ks/go/pkg/beam/core/runtime/coderx/coderx.shims.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2NvZGVyeC5zaGltcy5nbw==) | | |
   | [sdks/go/pkg/beam/transforms/stats/min.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21pbi5nbw==) | | |
   | ... and [202 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...a1573da](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818643826



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:

Review comment:
       How about doing that change as part of https://issues.apache.org/jira/browse/BEAM-14044. I suppose this sort of feature would depend a lot on how we implement batching more specifically per framework.  Let me add Robert as a reviewer at this point and see what he thinks we should do for batching.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818005185



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.

Review comment:
       +1, please assign this to me




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818895779



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(self._model_loader, self._inference_runner))
+        | beam.FlatMap(_unbatch))
+
+
+def _is_darwin() -> bool:
+  return sys.platform == 'darwin'
+
+
+def _get_current_process_memory_in_bytes():
+  """Returns memory usage in bytes."""
+
+  if resource is not None:
+    usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    if _is_darwin():
+      return usage
+    return usage * 1024
+  else:
+    logging.warning(
+        'Resource module is not available for current platform, '
+        'memory usage cannot be fetched.')
+  return 0
+
+
+def _is_windows() -> bool:
+  return platform.system() == 'Windows' or os.name == 'nt'
+
+
+def _is_cygwin() -> bool:
+  return platform.system().startswith('CYGWIN_NT')
+
+
+class _Clock(object):

Review comment:
       Is it necessary to define our own clock abstraction? The solution in BatchElements is just to provide a kwarg allowing tests to provide a mock to override `time.time`: https://github.com/apache/beam/blob/06e7c201dbdd8fe37d308b0bad2b1684e85e1dc7/sdks/python/apache_beam/transforms/util.py#L626-L633
   
   If we do need this (maybe for the fine-grained clock?) it would be nice to make it a general Beam solution as this is a common need.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821162358



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)
+    predictions = [
+        PredictionResult(e, r) for e, r in zip(examples, inference_generator)
+    ]
+    inference_latency = self._clock.get_current_time_in_microseconds(

Review comment:
       IDK, I think the code is pretty clear on its own in this case.  If we want we can break it into a function, like:
   
   def _compute_metrics(start_time, batch):
     ....
   
   but IMO it's not needed at this point.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821157956



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.

Review comment:
       don't delete just mark duplicate.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e00f05f) into [master](https://codecov.io/gh/apache/beam/commit/a1a63219780c22da8e3524cd9d4452c9431aabd6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1a6321) will **increase** coverage by `0.03%`.
   > The diff coverage is `92.18%`.
   
   > :exclamation: Current head e00f05f differs from pull request most recent head 9a73b7d. Consider uploading reports for the commit 9a73b7d to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.03%   74.06%   +0.03%     
   ==========================================
     Files         672      673       +1     
     Lines       88284    88423     +139     
   ==========================================
   + Hits        65358    65494     +136     
   - Misses      21806    21809       +3     
     Partials     1120     1120              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.66% <92.18%> (+0.03%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.18% <92.18%> (ø)` | |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `90.00% <0.00%> (-1.00%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.96% <0.00%> (ø)` | |
   | [.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==) | `98.90% <0.00%> (+0.03%)` | :arrow_up: |
   | [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `94.91% <0.00%> (+0.05%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.64% <0.00%> (+0.24%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/external.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbC5weQ==) | `80.00% <0.00%> (+0.27%)` | :arrow_up: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.85% <0.00%> (+1.42%)` | :arrow_up: |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.36% <0.00%> (+1.63%)` | :arrow_up: |
   | ... and [1 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a1a6321...9a73b7d](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] robertwb commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

robertwb commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r840851135



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any
+from typing import Iterable
+from typing import Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+class InferenceRunner():
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[Any]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+  def get_num_bytes(self, batch: Any) -> int:
+    """Returns the number of bytes of data for a batch."""
+    return len(pickle.dumps(batch))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class ModelLoader():
+  """Has the ability to load an ML model."""
+  def load_model(self) -> Any:
+    """Loads and initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_inference_runner(self) -> InferenceRunner:
+    """Returns an implementation of InferenceRunner for this model."""
+    raise NotImplementedError(type(self))
+
+
+def _unbatch(maybe_keyed_batches: Tuple[Any, Any]):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class RunInference(beam.PTransform):
+  """An extensible transform for running inferences."""
+  def __init__(self, model_loader: ModelLoader, clock=None):
+    self._model_loader = model_loader
+    self._clock = clock
+
+  # TODO(BEAM-14208): Add batch_size back off in the case there
+  # are functional reasons large batch sizes cannot be handled.
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO(BEAM-14044): Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(shared.Shared(), self._model_loader, self._clock))

Review comment:
       Not done? I think we should not have `shared.Shared()` be an argument to the DoFn, but rather construct it in the DoFn's constructor. If we want to make it an argument for testing, one can put it as an optional one after clock.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] robertwb commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

robertwb commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r840845214



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""

Review comment:
       Thanks, this will be helpful. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (37b6198) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.93%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.62%   +8.93%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52253    -9442     
   + Misses      19898    10229    -9669     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.62% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.23% <0.00%> (-0.14%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.17% <0.00%> (-0.08%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/option.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9vcHRpb24uZ28=) | | |
   | [sdks/go/pkg/beam/transforms/stats/mean.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21lYW4uZ28=) | | |
   | [sdks/go/pkg/beam/metrics.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9tZXRyaWNzLmdv) | | |
   | ... and [202 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...37b6198](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r833463846



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any, Iterable, Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+class InferenceRunner():
+  """Implements running inferences for a framework."""
+
+  def run_inference(self, batch: Any, model: Any) -> Iterable[Any]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+  def get_num_bytes(self, batch: Any) -> int:
+    """Returns the number of bytes of data for a batch."""
+    return len(pickle.dumps(batch))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class ModelLoader():
+  """Has the ability to load an ML model."""
+
+  def load_model(self) -> Any:
+    """Loads and initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_inference_runner(self) -> InferenceRunner:
+    """Returns an implementation of InferenceRunner for this model."""
+    raise NotImplementedError(type(self))
+
+
+def _unbatch(maybe_keyed_batches: Tuple[Any, Any]):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class RunInference(beam.PTransform):
+  """An extensible transform for running inferences."""
+
+  def __init__(self, model_loader: ModelLoader, clock=None):
+    self._model_loader = model_loader
+    self._clock = clock
+
+  # TODO: Add batch_size back off in the case there are functional
+  # reasons large batch sizes cannot be handled.
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (pcoll
+            | beam.BatchElements()
+            | beam.ParDo(
+                RunInferenceDoFn(shared.Shared(), self._model_loader,
+                                 self._clock))
+            | beam.FlatMap(_unbatch))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(namespace,
+                                          'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(namespace,
+                                          'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(self, examples_count: int, examples_byte_size: int,
+             latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  """A DoFn implementation generic to frameworks."""
+
+  def __init__(self,
+               shared_handle: shared.Shared,
+               model_loader: ModelLoader,
+               clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = model_loader.get_inference_runner()
+    self._shared_model_handle = shared_handle
+    self._metrics_collector = MetricsCollector(
+        self._inference_runner.get_metrics_namespace())
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    # TODO: Investigate releasing model.

Review comment:
       Can we add a corresponding JIRA for this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9a73b7d) into [master](https://codecov.io/gh/apache/beam/commit/a1a63219780c22da8e3524cd9d4452c9431aabd6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1a6321) will **increase** coverage by `0.03%`.
   > The diff coverage is `92.30%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.03%   74.06%   +0.03%     
   ==========================================
     Files         672      673       +1     
     Lines       88284    88425     +141     
   ==========================================
   + Hits        65358    65495     +137     
   - Misses      21806    21810       +4     
     Partials     1120     1120              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.65% <92.30%> (+0.03%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.30% <92.30%> (ø)` | |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.90% <0.00%> (-0.16%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.96% <0.00%> (ø)` | |
   | [.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==) | `98.90% <0.00%> (+0.03%)` | :arrow_up: |
   | [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `94.91% <0.00%> (+0.05%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (+0.12%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/external.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbC5weQ==) | `80.00% <0.00%> (+0.27%)` | :arrow_up: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.85% <0.00%> (+1.42%)` | :arrow_up: |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.36% <0.00%> (+1.63%)` | :arrow_up: |
   | ... and [1 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a1a6321...9a73b7d](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9a73b7d) into [master](https://codecov.io/gh/apache/beam/commit/a1a63219780c22da8e3524cd9d4452c9431aabd6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1a6321) will **increase** coverage by `0.03%`.
   > The diff coverage is `92.30%`.
   
   > :exclamation: Current head 9a73b7d differs from pull request most recent head b91a5c7. Consider uploading reports for the commit b91a5c7 to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.03%   74.06%   +0.03%     
   ==========================================
     Files         672      673       +1     
     Lines       88284    88425     +141     
   ==========================================
   + Hits        65358    65495     +137     
   - Misses      21806    21810       +4     
     Partials     1120     1120              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.65% <92.30%> (+0.03%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.30% <92.30%> (ø)` | |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.90% <0.00%> (-0.16%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.96% <0.00%> (ø)` | |
   | [.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==) | `98.90% <0.00%> (+0.03%)` | :arrow_up: |
   | [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `94.91% <0.00%> (+0.05%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (+0.12%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/external.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbC5weQ==) | `80.00% <0.00%> (+0.27%)` | :arrow_up: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.85% <0.00%> (+1.42%)` | :arrow_up: |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.36% <0.00%> (+1.63%)` | :arrow_up: |
   | ... and [1 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a1a6321...b91a5c7](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r840625167



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any
+from typing import Iterable
+from typing import Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+class InferenceRunner():
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[Any]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+  def get_num_bytes(self, batch: Any) -> int:
+    """Returns the number of bytes of data for a batch."""
+    return len(pickle.dumps(batch))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class ModelLoader():
+  """Has the ability to load an ML model."""
+  def load_model(self) -> Any:
+    """Loads and initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_inference_runner(self) -> InferenceRunner:
+    """Returns an implementation of InferenceRunner for this model."""
+    raise NotImplementedError(type(self))
+
+
+def _unbatch(maybe_keyed_batches: Tuple[Any, Any]):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class RunInference(beam.PTransform):
+  """An extensible transform for running inferences."""
+  def __init__(self, model_loader: ModelLoader, clock=None):
+    self._model_loader = model_loader
+    self._clock = clock
+
+  # TODO(BEAM-14208): Add batch_size back off in the case there
+  # are functional reasons large batch sizes cannot be handled.
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO(BEAM-14044): Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(shared.Shared(), self._model_loader, self._clock))

Review comment:
       Done, I'm going to put clock last, since that is the mostly testing var. I think the reason shared could be optional would only be in the case of a remote service. But even in that case shared is still used by this transform, which ideally should be the main entry point.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r840633104



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any
+from typing import Iterable
+from typing import Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+class InferenceRunner():
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[Any]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+  def get_num_bytes(self, batch: Any) -> int:
+    """Returns the number of bytes of data for a batch."""
+    return len(pickle.dumps(batch))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class ModelLoader():
+  """Has the ability to load an ML model."""
+  def load_model(self) -> Any:
+    """Loads and initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_inference_runner(self) -> InferenceRunner:
+    """Returns an implementation of InferenceRunner for this model."""
+    raise NotImplementedError(type(self))
+
+
+def _unbatch(maybe_keyed_batches: Tuple[Any, Any]):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class RunInference(beam.PTransform):
+  """An extensible transform for running inferences."""
+  def __init__(self, model_loader: ModelLoader, clock=None):
+    self._model_loader = model_loader
+    self._clock = clock
+
+  # TODO(BEAM-14208): Add batch_size back off in the case there
+  # are functional reasons large batch sizes cannot be handled.
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO(BEAM-14044): Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(shared.Shared(), self._model_loader, self._clock))
+        | beam.FlatMap(_unbatch))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache

Review comment:
       I am not sure. This is a relic of the TFX-BSL runinference transform.  I will track it so that we can correctly comment it or remove it in a separate PR.
   
   https://issues.apache.org/jira/browse/BEAM-13970
   
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b91a5c7) into [master](https://codecov.io/gh/apache/beam/commit/ca47cd83670a77507abcf5c54429966743af3ec0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ca47cd8) will **increase** coverage by `0.11%`.
   > The diff coverage is `92.30%`.
   
   > :exclamation: Current head b91a5c7 differs from pull request most recent head fbefc60. Consider uploading reports for the commit fbefc60 to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   73.98%   74.10%   +0.11%     
   ==========================================
     Files         674      675       +1     
     Lines       88558    88554       -4     
   ==========================================
   + Hits        65521    65621     +100     
   + Misses      21916    21813     -103     
   + Partials     1121     1120       -1     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.68% <92.30%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.30% <92.30%> (ø)` | |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `96.72% <0.00%> (-1.64%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.09% <0.00%> (-0.73%)` | :arrow_down: |
   | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.85% <0.00%> (-0.51%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/exec/fn.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbi5nbw==) | `69.26% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/typex/class.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2NsYXNzLmdv) | `85.38% <0.00%> (-0.12%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/util/reflectx/calls.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvY2FsbHMuZ28=) | `0.00% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/graphx/translate.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3RyYW5zbGF0ZS5nbw==) | `28.82% <0.00%> (+0.01%)` | :arrow_up: |
   | [sdks/go/pkg/beam/core/funcx/fn.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L2ZuLmdv) | `57.57% <0.00%> (+0.03%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `90.13% <0.00%> (+0.14%)` | :arrow_up: |
   | ... and [4 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [ca47cd8...fbefc60](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1057203820


   R: yeandy


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818687411



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.

Review comment:
       I made one [yesterday](https://issues.apache.org/jira/browse/BEAM-14037) but forgot to update this comment 😆  I don't think I have delete permissions to get rid of this though




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r819923238



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None

Review comment:
       Just for a little more understanding of logic, can we add a comment above like "Separating examples from keys"? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

tvalentyn commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821248609



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):

Review comment:
       s/Any/Tuple[Any, Any] ?
   

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""

Review comment:
       ```suggestion
       """Loads and initializes a model for processing."""
   ```

##########
File path: sdks/python/apache_beam/ml/inference/base_test.py
##########
@@ -0,0 +1,134 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from typing import Any
+from typing import Iterable
+import unittest
+
+import apache_beam as beam
+import apache_beam.ml.inference.base as base
+from apache_beam.metrics.metric import MetricsFilter
+from apache_beam.ml.inference.api import PredictionResult
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.testing.test_pipeline import TestPipeline
+
+
+class MockModel:

Review comment:
       nit: sounds like a FakeModel rather than MockModel, ditto below.

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')

Review comment:
       Is my understanding correct that batching / batch size will be encapsulated from user? If so, would it make sense to collect prorated performance metrics per example instead of per batch?

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)
+    predictions = [
+        PredictionResult(e, r) for e, r in zip(examples, inference_generator)
+    ]
+    inference_latency = self._clock.get_current_time_in_microseconds(

Review comment:
       +1, no need to add comments to explain what the code is doing (that's a sign of an unclear code).

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    self._metrics_collector = MetricsCollector(
+        model_loader.get_metrics_namespace())
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()

Review comment:
       nit: it's a no-op.

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(self._model_loader, self._inference_runner))
+        | beam.FlatMap(_unbatch))
+
+
+def _is_darwin() -> bool:
+  return sys.platform == 'darwin'
+
+
+def _get_current_process_memory_in_bytes():
+  """Returns memory usage in bytes."""
+
+  if resource is not None:
+    usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    if _is_darwin():
+      return usage
+    return usage * 1024
+  else:
+    logging.warning(
+        'Resource module is not available for current platform, '
+        'memory usage cannot be fetched.')
+  return 0
+
+
+def _is_windows() -> bool:

Review comment:
       we make our unit tests run on all major operating systems through github actions

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    self._metrics_collector = MetricsCollector(
+        model_loader.get_metrics_namespace())
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    # Process supports both keyed data, and example only data.
+    # First keys and samples are separated (if there are keys)
+    has_keys = isinstance(batch[0], tuple)
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+
+    start_time = self._clock.get_current_time_in_microseconds()
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)
+    predictions = [
+        PredictionResult(e, r) for e, r in zip(examples, inference_generator)
+    ]
+
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+
+    # keys will be recombined with their predictions in the RunInferenceImpl PTransform.
+    yield keys, predictions
+
+  def finish_bundle(self):
+    self._metrics_collector.update_metrics_with_cache()
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._clock = clock
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO(BEAM-14044): Hook into the batching DoFn APIs.
+        | beam.BatchElements(min_batch_size=2)
+        | beam.ParDo(
+            RunInferenceDoFn(
+                self._model_loader, self._inference_runner, clock=self._clock))
+        | beam.FlatMap(_unbatch))
+
+
+def _is_darwin() -> bool:
+  return sys.platform == 'darwin'
+
+
+def _get_current_process_memory_in_bytes():
+  """Returns memory usage in bytes."""
+
+  if resource is not None:

Review comment:
       You should have
   ```
   try:
     import resource
   except ImportError:
     resource = None
   ```
   to make this condition work




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2cb876c) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.92%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.61%   +8.92%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52247    -9448     
   + Misses      19898    10235    -9663     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.61% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.01% <0.00%> (-1.39%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `80.39% <0.00%> (-0.86%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.48% <0.00%> (-0.76%)` | :arrow_down: |
   | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.85% <0.00%> (-0.51%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.90% <0.00%> (-0.16%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/util/harnessopts/cache.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2hhcm5lc3NvcHRzL2NhY2hlLmdv) | | |
   | ... and [204 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...2cb876c](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821157406



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)

Review comment:
       I'm indifferent mostly I can see the advantages of both. I can move it down for now though and we can move it later if we are unhappy.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821866198



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')

Review comment:
       It's an interesting and good idea. I followed what TFX does because I anticipate having their code use our base implementation.  However, users that are interested in per example should be able to see that metric with a simple division.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1071935338


   We're working with the tfx bsl team to see what we can share.  So this is
   on pause a little.
   
   On Thu, Mar 17, 2022, 9:04 PM Ahmet Altay ***@***.***> wrote:
   
   > @ryanthompson591 <https://github.com/ryanthompson591> - is this ready for
   > another round of review?
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/pull/16970#issuecomment-1071918329>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ABS23CVBHB5GIYLGLYWNS7DVAPJDRANCNFSM5PR7LZGQ>
   > .
   > Triage notifications on the go with GitHub Mobile for iOS
   > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
   > or Android
   > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
   >
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b888735) into [master](https://codecov.io/gh/apache/beam/commit/62a661071b7db15e71d236abe68e15582e8997c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (62a6610) will **increase** coverage by `0.02%`.
   > The diff coverage is `92.18%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   73.94%   73.97%   +0.02%     
   ==========================================
     Files         669      670       +1     
     Lines       88158    88305     +147     
   ==========================================
   + Hits        65191    65324     +133     
   - Misses      21855    21869      +14     
     Partials     1112     1112              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.65% <92.18%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.18% <92.18%> (ø)` | |
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.37% <0.00%> (-4.66%)` | :arrow_down: |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `96.72% <0.00%> (-1.64%)` | :arrow_down: |
   | [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `87.58% <0.00%> (-1.01%)` | :arrow_down: |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.32% <0.00%> (-0.26%)` | :arrow_down: |
   | [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `77.27% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.58% <0.00%> (-0.11%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (-0.08%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.39% <0.00%> (ø)` | |
   | [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.51% <0.00%> (ø)` | |
   | ... and [4 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [62a6610...b888735](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b91a5c7) into [master](https://codecov.io/gh/apache/beam/commit/a1a63219780c22da8e3524cd9d4452c9431aabd6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1a6321) will **increase** coverage by `0.07%`.
   > The diff coverage is `92.30%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.03%   74.10%   +0.07%     
   ==========================================
     Files         672      675       +3     
     Lines       88284    88554     +270     
   ==========================================
   + Hits        65358    65621     +263     
   - Misses      21806    21813       +7     
     Partials     1120     1120              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.68% <92.30%> (+0.05%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.30% <92.30%> (ø)` | |
   | [...apache\_beam/runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | `74.57% <0.00%> (-1.31%)` | :arrow_down: |
   | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.85% <0.00%> (-0.51%)` | :arrow_down: |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `87.90% <0.00%> (-0.42%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.09% <0.00%> (-0.31%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.90% <0.00%> (-0.16%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.96% <0.00%> (ø)` | |
   | [.../apache\_beam/runners/interactive/dataproc/types.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy90eXBlcy5weQ==) | `93.33% <0.00%> (ø)` | |
   | [...ache\_beam/runners/dataflow/dataflow\_job\_service.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19qb2Jfc2VydmljZS5weQ==) | `62.16% <0.00%> (ø)` | |
   | ... and [14 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a1a6321...b91a5c7](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0386e4f) into [master](https://codecov.io/gh/apache/beam/commit/a1a63219780c22da8e3524cd9d4452c9431aabd6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1a6321) will **decrease** coverage by `0.05%`.
   > The diff coverage is `92.18%`.
   
   > :exclamation: Current head 0386e4f differs from pull request most recent head e00f05f. Consider uploading reports for the commit e00f05f to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   - Coverage   74.03%   73.97%   -0.06%     
   ==========================================
     Files         672      670       -2     
     Lines       88284    88315      +31     
   ==========================================
   - Hits        65358    65329      -29     
   - Misses      21806    21874      +68     
   + Partials     1120     1112       -8     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.64% <92.18%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.18% <92.18%> (ø)` | |
   | [sdks/go/pkg/beam/io/datastoreio/datastore.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhc3RvcmVpby9kYXRhc3RvcmUuZ28=) | `16.03% <0.00%> (-51.78%)` | :arrow_down: |
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.68% <0.00%> (-2.44%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `80.62% <0.00%> (-0.78%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/pipelinex/replace.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3JlcGxhY2UuZ28=) | `65.94% <0.00%> (-0.72%)` | :arrow_down: |
   | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.85% <0.00%> (-0.51%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/util/reflectx/tags.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvdGFncy5nbw==) | `0.00% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/util/jsonx/jsonx.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvanNvbngvanNvbnguZ28=) | | |
   | [sdks/go/pkg/beam/options/jobopts/options.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9vcHRpb25zL2pvYm9wdHMvb3B0aW9ucy5nbw==) | | |
   | [sdks/go/pkg/beam/options/jobopts/stringSlice.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9vcHRpb25zL2pvYm9wdHMvc3RyaW5nU2xpY2UuZ28=) | | |
   | ... and [4 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a1a6321...e00f05f](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

tvalentyn commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821314513



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)
+    predictions = [
+        PredictionResult(e, r) for e, r in zip(examples, inference_generator)
+    ]
+    inference_latency = self._clock.get_current_time_in_microseconds(

Review comment:
       +1, no need to add comments to explain what the code is doing (that's typically a sign of an unclear code that needs to be reorganized).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821862548



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

tvalentyn commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821935947



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')

Review comment:
       will batch size be constant for entire execution ? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r840656241



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any
+from typing import Iterable
+from typing import Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+class InferenceRunner():
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[Any]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+  def get_num_bytes(self, batch: Any) -> int:
+    """Returns the number of bytes of data for a batch."""
+    return len(pickle.dumps(batch))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class ModelLoader():
+  """Has the ability to load an ML model."""
+  def load_model(self) -> Any:
+    """Loads and initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_inference_runner(self) -> InferenceRunner:
+    """Returns an implementation of InferenceRunner for this model."""
+    raise NotImplementedError(type(self))
+
+
+def _unbatch(maybe_keyed_batches: Tuple[Any, Any]):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class RunInference(beam.PTransform):
+  """An extensible transform for running inferences."""
+  def __init__(self, model_loader: ModelLoader, clock=None):
+    self._model_loader = model_loader
+    self._clock = clock
+
+  # TODO(BEAM-14208): Add batch_size back off in the case there
+  # are functional reasons large batch sizes cannot be handled.
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO(BEAM-14044): Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(shared.Shared(), self._model_loader, self._clock))
+        | beam.FlatMap(_unbatch))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  """A DoFn implementation generic to frameworks."""
+  def __init__(
+      self,
+      shared_handle: shared.Shared,
+      model_loader: ModelLoader,
+      clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = model_loader.get_inference_runner()
+    self._shared_model_handle = shared_handle
+    self._metrics_collector = MetricsCollector(
+        self._inference_runner.get_metrics_namespace())
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (

Review comment:
       Done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9a73b7d) into [master](https://codecov.io/gh/apache/beam/commit/a1a63219780c22da8e3524cd9d4452c9431aabd6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1a6321) will **increase** coverage by `0.03%`.
   > The diff coverage is `92.30%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.03%   74.06%   +0.03%     
   ==========================================
     Files         672      673       +1     
     Lines       88284    88425     +141     
   ==========================================
   + Hits        65358    65495     +137     
   - Misses      21806    21810       +4     
     Partials     1120     1120              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.65% <92.30%> (+0.03%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.30% <92.30%> (ø)` | |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.90% <0.00%> (-0.16%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.96% <0.00%> (ø)` | |
   | [.../apache\_beam/options/pipeline\_options\_validator.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zX3ZhbGlkYXRvci5weQ==) | `98.90% <0.00%> (+0.03%)` | :arrow_up: |
   | [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `94.91% <0.00%> (+0.05%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (+0.12%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/external.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbC5weQ==) | `80.00% <0.00%> (+0.27%)` | :arrow_up: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.85% <0.00%> (+1.42%)` | :arrow_up: |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.36% <0.00%> (+1.63%)` | :arrow_up: |
   | ... and [1 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a1a6321...9a73b7d](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aaltay commented on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

aaltay commented on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1071918329


   @ryanthompson591 - is this ready for another round of review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r817956018



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:

Review comment:
       Do we want `Iterable[PredictionResult]` instead of `List[PredictionResult]`?

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult

Review comment:
       I have `api.py`. Do we we want with the `s`? 

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:

Review comment:
       Robert had something like 
   ```
   ExampleBatchType = TypeVar('ExampleBatchType')
   InferenceBatchType = TypeVar('InferenceBatchType')
    
   class InferenceRunner(Generic[ExampleBatchType, InferenceBatchType]):
     def to_impl_batch(batch: Iterable[Any] -> ExampleBatchType):
       raise NotImplementedError(type(self))
    
     def from_impl_batch(inferences: InferenceBatchType) -> Iterable[Any]:
       raise NotImplementedError(type(self))
    
     def run_inference(self, batch : ExampleBatchType) -> InferenceBatchType:
       raise NotImplementedError(type(self))
   ```
   I'm guessing it's not necessary to have the type variables if you're not converting to and from batches? But how do we structure this if we want to leverage Batched DoFn in the future? Thought it's probably not necessary to do the batching explicitly if we can leverage something like `@DoFn.yields_batches`? @TheNeuralBit what are your thoughts?

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(self._model_loader, self._inference_runner))
+        | beam.FlatMap(_unbatch))
+
+
+def _is_darwin() -> bool:
+  return sys.platform == 'darwin'
+
+
+def _get_current_process_memory_in_bytes():
+  """Returns memory usage in bytes."""
+
+  if resource is not None:
+    usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    if _is_darwin():
+      return usage
+    return usage * 1024
+  else:
+    logging.warning(
+        'Resource module is not available for current platform, '
+        'memory usage cannot be fetched.')
+  return 0
+
+
+def _is_windows() -> bool:
+  return platform.system() == 'Windows' or os.name == 'nt'
+
+
+def _is_cygwin() -> bool:
+  return platform.system().startswith('CYGWIN_NT')
+
+
+class _Clock(object):

Review comment:
       Do we want a test for Clock?

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):

Review comment:
       ```suggestion
     def load_model(self) -> InferenceRunner:
   ```

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(self._model_loader, self._inference_runner))
+        | beam.FlatMap(_unbatch))
+
+
+def _is_darwin() -> bool:
+  return sys.platform == 'darwin'
+
+
+def _get_current_process_memory_in_bytes():
+  """Returns memory usage in bytes."""
+
+  if resource is not None:
+    usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    if _is_darwin():
+      return usage
+    return usage * 1024
+  else:
+    logging.warning(
+        'Resource module is not available for current platform, '
+        'memory usage cannot be fetched.')
+  return 0
+
+
+def _is_windows() -> bool:

Review comment:
       Do we want test for OS?

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.

Review comment:
       Let's file a JIRA for this

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+

Review comment:
       What about `num_failed_inferences`, `num_model_loads`, `num_model_versions`, and `curr_model_version`? I think the latter two are relevant to side-inputs, which we don't have yet.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1058163858


   R: @tvalentyn @robertwb
   CC: @kerrydc


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818645909



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(self._model_loader, self._inference_runner))
+        | beam.FlatMap(_unbatch))
+
+
+def _is_darwin() -> bool:
+  return sys.platform == 'darwin'
+
+
+def _get_current_process_memory_in_bytes():
+  """Returns memory usage in bytes."""
+
+  if resource is not None:
+    usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    if _is_darwin():
+      return usage
+    return usage * 1024
+  else:
+    logging.warning(
+        'Resource module is not available for current platform, '
+        'memory usage cannot be fetched.')
+  return 0
+
+
+def _is_windows() -> bool:
+  return platform.system() == 'Windows' or os.name == 'nt'
+
+
+def _is_cygwin() -> bool:
+  return platform.system().startswith('CYGWIN_NT')
+
+
+class _Clock(object):

Review comment:
       Since the clock is very environment specific its a little harder to unit test. However, I do think I should change this up (it's based on the TFX version) and make this mockable so we can write better unit tests for the metrics.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818646535



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(self._model_loader, self._inference_runner))
+        | beam.FlatMap(_unbatch))
+
+
+def _is_darwin() -> bool:
+  return sys.platform == 'darwin'
+
+
+def _get_current_process_memory_in_bytes():
+  """Returns memory usage in bytes."""
+
+  if resource is not None:
+    usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    if _is_darwin():
+      return usage
+    return usage * 1024
+  else:
+    logging.warning(
+        'Resource module is not available for current platform, '
+        'memory usage cannot be fetched.')
+  return 0
+
+
+def _is_windows() -> bool:

Review comment:
       I mean, that is environment specific, so I haven't seen this sort of small function unit tested often.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2cb876c) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.92%`.
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 2cb876c differs from pull request most recent head 37b6198. Consider uploading reports for the commit 37b6198 to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.61%   +8.92%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52247    -9448     
   + Misses      19898    10235    -9663     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.61% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.01% <0.00%> (-1.39%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `80.39% <0.00%> (-0.86%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.48% <0.00%> (-0.76%)` | :arrow_down: |
   | [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.85% <0.00%> (-0.51%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.90% <0.00%> (-0.16%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/util/hooks/hooks.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaG9va3MvaG9va3MuZ28=) | | |
   | ... and [204 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...37b6198](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1057209917


   R: @TheNeuralBit and @tvalentyn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1057293214


   CC: @KevinGG 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818631705



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:

Review comment:
       Yeah, I like Iterable, it is more generic. Done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (37b6198) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.93%`.
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 37b6198 differs from pull request most recent head 5fe7edf. Consider uploading reports for the commit 5fe7edf to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.62%   +8.93%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52253    -9442     
   + Misses      19898    10229    -9669     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.62% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.23% <0.00%> (-0.14%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.17% <0.00%> (-0.08%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/graphx/tree.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3RyZWUuZ28=) | | |
   | [sdks/go/pkg/beam/core/runtime/exec/sideinput.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zaWRlaW5wdXQuZ28=) | | |
   | [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | | |
   | ... and [202 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...5fe7edf](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818859202



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inferences = self._inference_runner.run_inference(examples, self._model)
+    inference_latency = self._clock.get_current_time_in_microseconds(
+    ) - start_time
+    num_bytes = sys.getsizeof(batch)
+    num_elements = len(batch)
+    self._metrics_collector.update(
+        num_elements, sys.getsizeof(batch), inference_latency)
+    yield keys, [PredictionResult(e, r) for e, r in zip(examples, inferences)]
+
+
+class RunInferenceImpl(beam.PTransform):
+  def __init__(self, model_loader, inference_runner):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO: Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(self._model_loader, self._inference_runner))
+        | beam.FlatMap(_unbatch))
+
+
+def _is_darwin() -> bool:
+  return sys.platform == 'darwin'
+
+
+def _get_current_process_memory_in_bytes():
+  """Returns memory usage in bytes."""
+
+  if resource is not None:
+    usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    if _is_darwin():
+      return usage
+    return usage * 1024
+  else:
+    logging.warning(
+        'Resource module is not available for current platform, '
+        'memory usage cannot be fetched.')
+  return 0
+
+
+def _is_windows() -> bool:
+  return platform.system() == 'Windows' or os.name == 'nt'
+
+
+def _is_cygwin() -> bool:
+  return platform.system().startswith('CYGWIN_NT')
+
+
+class _Clock(object):

Review comment:
       Agreed, I'd like to be able to do basic validation for Clock.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1071935338


   We're working with the tfx bsl team to see what we can share.  So this is
   on pause a little.
   
   On Thu, Mar 17, 2022, 9:04 PM Ahmet Altay ***@***.***> wrote:
   
   > @ryanthompson591 <https://github.com/ryanthompson591> - is this ready for
   > another round of review?
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/pull/16970#issuecomment-1071918329>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ABS23CVBHB5GIYLGLYWNS7DVAPJDRANCNFSM5PR7LZGQ>
   > .
   > Triage notifications on the go with GitHub Mobile for iOS
   > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
   > or Android
   > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
   >
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

tvalentyn commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821314513



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)
+    predictions = [
+        PredictionResult(e, r) for e, r in zip(examples, inference_generator)
+    ]
+    inference_latency = self._clock.get_current_time_in_microseconds(

Review comment:
       +1, no need to add comments to explain _what_ the code is doing (that's typically a sign of an unclear code that needs to be reorganized).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b888735) into [master](https://codecov.io/gh/apache/beam/commit/62a661071b7db15e71d236abe68e15582e8997c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (62a6610) will **increase** coverage by `0.02%`.
   > The diff coverage is `92.18%`.
   
   > :exclamation: Current head b888735 differs from pull request most recent head 0386e4f. Consider uploading reports for the commit 0386e4f to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   73.94%   73.97%   +0.02%     
   ==========================================
     Files         669      670       +1     
     Lines       88158    88305     +147     
   ==========================================
   + Hits        65191    65324     +133     
   - Misses      21855    21869      +14     
     Partials     1112     1112              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.65% <92.18%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.18% <92.18%> (ø)` | |
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.37% <0.00%> (-4.66%)` | :arrow_down: |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `96.72% <0.00%> (-1.64%)` | :arrow_down: |
   | [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `87.58% <0.00%> (-1.01%)` | :arrow_down: |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.32% <0.00%> (-0.26%)` | :arrow_down: |
   | [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `77.27% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.58% <0.00%> (-0.11%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (-0.08%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.39% <0.00%> (ø)` | |
   | [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.51% <0.00%> (ø)` | |
   | ... and [4 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [62a6610...0386e4f](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (37b6198) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.93%`.
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 37b6198 differs from pull request most recent head eb8cf56. Consider uploading reports for the commit eb8cf56 to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.62%   +8.93%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52253    -9442     
   + Misses      19898    10229    -9669     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.62% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.23% <0.00%> (-0.14%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.17% <0.00%> (-0.08%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/xlangx/expand.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuZC5nbw==) | | |
   | [sdks/go/pkg/beam/core/typex/fulltype.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2Z1bGx0eXBlLmdv) | | |
   | [...o/pkg/beam/io/rtrackers/offsetrange/offsetrange.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9ydHJhY2tlcnMvb2Zmc2V0cmFuZ2Uvb2Zmc2V0cmFuZ2UuZ28=) | | |
   | ... and [202 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...eb8cf56](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818622443



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult

Review comment:
       No I think api.py is better.  Changed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r819923706



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)
+    predictions = [
+        PredictionResult(e, r) for e, r in zip(examples, inference_generator)
+    ]
+    inference_latency = self._clock.get_current_time_in_microseconds(

Review comment:
       Just for a little more understanding of logic, can we add a new line above, and a comment above like "Compute and save metrics"? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r818857971



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+

Review comment:
       Sounds good, thanks!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (37b6198) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.93%`.
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 37b6198 differs from pull request most recent head 5504687. Consider uploading reports for the commit 5504687 to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.62%   +8.93%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52253    -9442     
   + Misses      19898    10229    -9669     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.62% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.23% <0.00%> (-0.14%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.17% <0.00%> (-0.08%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/exec/decode.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kZWNvZGUuZ28=) | | |
   | [.../go/pkg/beam/runners/dataflow/dataflowlib/fixup.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2ZpeHVwLmdv) | | |
   | [sdks/go/pkg/beam/io/filesystem/local/local.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL2xvY2FsL2xvY2FsLmdv) | | |
   | ... and [202 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...5504687](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

tvalentyn commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821087251



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,194 @@
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import List
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.apis import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> List[PredictionResult]:
+    """Runs inferences on a batch of examples and returns a list of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+

Review comment:
       Which label are you using to track the runinference work in Jira? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821798777



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace
+    self._metrics_collector = MetricsCollector('default_namespace')
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (
+          (end_time - start_time) / _MILLISECOND_TO_MICROSECOND)
+      self._metrics_collector.model_byte_size_cache = (
+          memory_after - memory_before)
+      return model
+
+    return self._shared_model_handle.acquire(load)
+
+  def setup(self):
+    super().setup()
+    self._model = self._load_model()
+
+  def process(self, batch):
+    has_keys = isinstance(batch[0], tuple)
+    start_time = self._clock.get_current_time_in_microseconds()
+    if has_keys:
+      examples = [example for _, example in batch]
+      keys = [key for key, _ in batch]
+    else:
+      examples = batch
+      keys = None
+    inference_generator = self._inference_runner.run_inference(
+        examples, self._model)
+    predictions = [
+        PredictionResult(e, r) for e, r in zip(examples, inference_generator)
+    ]
+    inference_latency = self._clock.get_current_time_in_microseconds(

Review comment:
       👍 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ryanthompson591 commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

ryanthompson591 commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r821156953



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  def __init__(self, model_loader, inference_runner, clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = inference_runner
+    self._shared_model_handle = shared.Shared()
+    # TODO: Compute a good metrics namespace

Review comment:
       I think we can put this in one of the framework defined classes. TFX has a namespace where they define a lot of things about the implementation details. Currently their namespace to reads bulkinference.  The name space they define is very model specific.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (37b6198) into [master](https://codecov.io/gh/apache/beam/commit/1c579f9524a35bcf9a3a7b758359c8f7aeabf5fe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c579f9) will **increase** coverage by `8.93%`.
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 37b6198 differs from pull request most recent head 11b9488. Consider uploading reports for the commit 11b9488 to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16970/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   74.69%   83.62%   +8.93%     
   ==========================================
     Files         657      453     -204     
     Lines       82593    62482   -20111     
   ==========================================
   - Hits        61695    52253    -9442     
   + Misses      19898    10229    -9669     
   + Partials     1000        0    -1000     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.62% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.98% <0.00%> (-0.30%)` | :arrow_down: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.23% <0.00%> (-0.14%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (-0.13%)` | :arrow_down: |
   | [...thon/apache\_beam/io/aws/clients/s3/boto3\_client.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL2NsaWVudHMvczMvYm90bzNfY2xpZW50LnB5) | `14.39% <0.00%> (-0.13%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.17% <0.00%> (-0.08%)` | :arrow_down: |
   | [...n/apache\_beam/examples/complete/game/user\_score.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlLnB5) | `62.79% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/xlangx/registry.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3JlZ2lzdHJ5Lmdv) | | |
   | [sdks/go/pkg/beam/io/xlang/schemaio/schemaio.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby94bGFuZy9zY2hlbWFpby9zY2hlbWFpby5nbw==) | | |
   | [sdks/go/pkg/beam/util.go](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsLmdv) | | |
   | ... and [202 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c579f9...11b9488](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1054615571


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16970](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b888735) into [master](https://codecov.io/gh/apache/beam/commit/62a661071b7db15e71d236abe68e15582e8997c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (62a6610) will **increase** coverage by `0.02%`.
   > The diff coverage is `92.18%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16970      +/-   ##
   ==========================================
   + Coverage   73.94%   73.97%   +0.02%     
   ==========================================
     Files         669      670       +1     
     Lines       88158    88305     +147     
   ==========================================
   + Hits        65191    65324     +133     
   - Misses      21855    21869      +14     
     Partials     1112     1112              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.65% <92.18%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `92.18% <92.18%> (ø)` | |
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.37% <0.00%> (-4.66%)` | :arrow_down: |
   | [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `96.72% <0.00%> (-1.64%)` | :arrow_down: |
   | [...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5) | `87.58% <0.00%> (-1.01%)` | :arrow_down: |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.32% <0.00%> (-0.26%)` | :arrow_down: |
   | [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `77.27% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.58% <0.00%> (-0.11%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `63.63% <0.00%> (-0.08%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.39% <0.00%> (ø)` | |
   | [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.51% <0.00%> (ø)` | |
   | ... and [4 more](https://codecov.io/gh/apache/beam/pull/16970/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [62a6610...b888735](https://codecov.io/gh/apache/beam/pull/16970?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r833463729



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any, Iterable, Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+class InferenceRunner():
+  """Implements running inferences for a framework."""
+
+  def run_inference(self, batch: Any, model: Any) -> Iterable[Any]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+  def get_num_bytes(self, batch: Any) -> int:
+    """Returns the number of bytes of data for a batch."""
+    return len(pickle.dumps(batch))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class ModelLoader():
+  """Has the ability to load an ML model."""
+
+  def load_model(self) -> Any:
+    """Loads and initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_inference_runner(self) -> InferenceRunner:
+    """Returns an implementation of InferenceRunner for this model."""
+    raise NotImplementedError(type(self))
+
+
+def _unbatch(maybe_keyed_batches: Tuple[Any, Any]):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class RunInference(beam.PTransform):
+  """An extensible transform for running inferences."""
+
+  def __init__(self, model_loader: ModelLoader, clock=None):
+    self._model_loader = model_loader
+    self._clock = clock
+
+  # TODO: Add batch_size back off in the case there are functional

Review comment:
       Can we add a corresponding JIRA for this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] robertwb commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

robertwb commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r833491143



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import platform
+import resource
+import sys
+import time
+from typing import Any
+from typing import Iterable
+
+import apache_beam as beam
+from apache_beam.utils import shared
+from apache_beam.ml.inference.api import PredictionResult
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+def _unbatch(maybe_keyed_batches: Any):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class ModelLoader:
+  """Has the ability to load an ML model."""
+  def load_model(self):
+    """Loads an initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class InferenceRunner:
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[PredictionResult]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')

Review comment:
       No, batch size is not in general constant.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] robertwb commented on a change in pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

robertwb commented on a change in pull request #16970:
URL: https://github.com/apache/beam/pull/16970#discussion_r840097231



##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any
+from typing import Iterable
+from typing import Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000

Review comment:
       Use underscores like 1_000_000 to make this easier to read.

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any
+from typing import Iterable
+from typing import Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+class InferenceRunner():
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[Any]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+  def get_num_bytes(self, batch: Any) -> int:
+    """Returns the number of bytes of data for a batch."""
+    return len(pickle.dumps(batch))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class ModelLoader():
+  """Has the ability to load an ML model."""
+  def load_model(self) -> Any:
+    """Loads and initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_inference_runner(self) -> InferenceRunner:
+    """Returns an implementation of InferenceRunner for this model."""
+    raise NotImplementedError(type(self))
+
+
+def _unbatch(maybe_keyed_batches: Tuple[Any, Any]):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class RunInference(beam.PTransform):
+  """An extensible transform for running inferences."""
+  def __init__(self, model_loader: ModelLoader, clock=None):
+    self._model_loader = model_loader
+    self._clock = clock
+
+  # TODO(BEAM-14208): Add batch_size back off in the case there
+  # are functional reasons large batch sizes cannot be handled.
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO(BEAM-14044): Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(shared.Shared(), self._model_loader, self._clock))

Review comment:
       Let's put the shared argument last, and make it optional (similar to clock).

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any
+from typing import Iterable
+from typing import Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+class InferenceRunner():
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[Any]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+  def get_num_bytes(self, batch: Any) -> int:
+    """Returns the number of bytes of data for a batch."""
+    return len(pickle.dumps(batch))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class ModelLoader():
+  """Has the ability to load an ML model."""
+  def load_model(self) -> Any:
+    """Loads and initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_inference_runner(self) -> InferenceRunner:
+    """Returns an implementation of InferenceRunner for this model."""
+    raise NotImplementedError(type(self))
+
+
+def _unbatch(maybe_keyed_batches: Tuple[Any, Any]):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class RunInference(beam.PTransform):
+  """An extensible transform for running inferences."""
+  def __init__(self, model_loader: ModelLoader, clock=None):
+    self._model_loader = model_loader
+    self._clock = clock
+
+  # TODO(BEAM-14208): Add batch_size back off in the case there
+  # are functional reasons large batch sizes cannot be handled.
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO(BEAM-14044): Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(shared.Shared(), self._model_loader, self._clock))
+        | beam.FlatMap(_unbatch))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache
+    self.load_model_latency_milli_secs_cache = None
+    self.model_byte_size_cache = None
+
+  def update_metrics_with_cache(self):
+    if self.load_model_latency_milli_secs_cache is not None:
+      self._load_model_latency_milli_secs.update(
+          self.load_model_latency_milli_secs_cache)
+      self.load_model_latency_milli_secs_cache = None
+    if self.model_byte_size_cache is not None:
+      self._model_byte_size.update(self.model_byte_size_cache)
+      self.model_byte_size_cache = None
+
+  def update(
+      self,
+      examples_count: int,
+      examples_byte_size: int,
+      latency_micro_secs: int):
+    self._inference_batch_latency_micro_secs.update(latency_micro_secs)
+    self._inference_counter.inc(examples_count)
+    self._inference_request_batch_size.update(examples_count)
+    self._inference_request_batch_byte_size.update(examples_byte_size)
+
+
+class RunInferenceDoFn(beam.DoFn):
+  """A DoFn implementation generic to frameworks."""
+  def __init__(
+      self,
+      shared_handle: shared.Shared,
+      model_loader: ModelLoader,
+      clock=None):
+    self._model_loader = model_loader
+    self._inference_runner = model_loader.get_inference_runner()
+    self._shared_model_handle = shared_handle
+    self._metrics_collector = MetricsCollector(
+        self._inference_runner.get_metrics_namespace())
+    self._clock = clock
+    if not clock:
+      self._clock = _ClockFactory.make_clock()
+    self._model = None
+
+  def _load_model(self):
+    def load():
+      """Function for constructing shared LoadedModel."""
+      memory_before = _get_current_process_memory_in_bytes()
+      start_time = self._clock.get_current_time_in_microseconds()
+      model = self._model_loader.load_model()
+      end_time = self._clock.get_current_time_in_microseconds()
+      memory_after = _get_current_process_memory_in_bytes()
+      self._metrics_collector.load_model_latency_milli_secs_cache = (

Review comment:
       Rather than setting attributes directly, can you make this update_load_model_metrics or something? 

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""
+
+import logging
+import os
+import pickle
+import platform
+import sys
+import time
+from typing import Any
+from typing import Iterable
+from typing import Tuple
+
+import apache_beam as beam
+from apache_beam.utils import shared
+
+try:
+  # pylint: disable=g-import-not-at-top
+  import resource
+except ImportError:
+  resource = None
+
+_MILLISECOND_TO_MICROSECOND = 1000
+_MICROSECOND_TO_NANOSECOND = 1000
+_SECOND_TO_MICROSECOND = 1000000
+
+
+class InferenceRunner():
+  """Implements running inferences for a framework."""
+  def run_inference(self, batch: Any, model: Any) -> Iterable[Any]:
+    """Runs inferences on a batch of examples and returns an Iterable of Predictions."""
+    raise NotImplementedError(type(self))
+
+  def get_num_bytes(self, batch: Any) -> int:
+    """Returns the number of bytes of data for a batch."""
+    return len(pickle.dumps(batch))
+
+  def get_metrics_namespace(self) -> str:
+    """Returns a namespace for metrics collected by the RunInference transform."""
+    return 'RunInference'
+
+
+class ModelLoader():
+  """Has the ability to load an ML model."""
+  def load_model(self) -> Any:
+    """Loads and initializes a model for processing."""
+    raise NotImplementedError(type(self))
+
+  def get_inference_runner(self) -> InferenceRunner:
+    """Returns an implementation of InferenceRunner for this model."""
+    raise NotImplementedError(type(self))
+
+
+def _unbatch(maybe_keyed_batches: Tuple[Any, Any]):
+  keys, results = maybe_keyed_batches
+  if keys:
+    return zip(keys, results)
+  else:
+    return results
+
+
+class RunInference(beam.PTransform):
+  """An extensible transform for running inferences."""
+  def __init__(self, model_loader: ModelLoader, clock=None):
+    self._model_loader = model_loader
+    self._clock = clock
+
+  # TODO(BEAM-14208): Add batch_size back off in the case there
+  # are functional reasons large batch sizes cannot be handled.
+  def expand(self, pcoll: beam.PCollection) -> beam.PCollection:
+    return (
+        pcoll
+        # TODO(BEAM-14044): Hook into the batching DoFn APIs.
+        | beam.BatchElements()
+        | beam.ParDo(
+            RunInferenceDoFn(shared.Shared(), self._model_loader, self._clock))
+        | beam.FlatMap(_unbatch))
+
+
+class MetricsCollector:
+  """A metrics collector that tracks ML related performance and memory usage."""
+  def __init__(self, namespace: str):
+    # Metrics
+    self._inference_counter = beam.metrics.Metrics.counter(
+        namespace, 'num_inferences')
+    self._inference_request_batch_size = beam.metrics.Metrics.distribution(
+        namespace, 'inference_request_batch_size')
+    self._inference_request_batch_byte_size = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_request_batch_byte_size'))
+    # Batch inference latency in microseconds.
+    self._inference_batch_latency_micro_secs = (
+        beam.metrics.Metrics.distribution(
+            namespace, 'inference_batch_latency_micro_secs'))
+    self._model_byte_size = beam.metrics.Metrics.distribution(
+        namespace, 'model_byte_size')
+    # Model load latency in milliseconds.
+    self._load_model_latency_milli_secs = beam.metrics.Metrics.distribution(
+        namespace, 'load_model_latency_milli_secs')
+
+    # Metrics cache

Review comment:
       What is the goal of the metric cache? Is it because we can't update metrics in setup? If so, can you add a comment? 

##########
File path: sdks/python/apache_beam/ml/inference/base.py
##########
@@ -0,0 +1,252 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An extensible run inference transform."""

Review comment:
       It's probably worth expanding this at least a little bit, naming the main classes and usage. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aaltay commented on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

aaltay commented on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1071918329


   @ryanthompson591 - is this ready for another round of review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] yeandy edited a comment on pull request #16970: [BEAM-13982] A base class for run inference

Posted by GitBox <gi...@apache.org>.

yeandy edited a comment on pull request #16970:
URL: https://github.com/apache/beam/pull/16970#issuecomment-1057209917


   CC: @TheNeuralBit and @tvalentyn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org