You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/13 01:27:56 UTC
[GitHub] [beam] y1chi opened a new pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
y1chi opened a new pull request #12562:
URL: https://github.com/apache/beam/pull/12562
… sdk worker
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-696271535
# [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=h1) Report
> Merging [#12562](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e164d170eb6b5ec1dddd99f09e79dfb0147b84ae?el=desc) will **increase** coverage by `47.83%`.
> The diff coverage is `6.00%`.
[](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12562 +/- ##
===========================================
+ Coverage 34.47% 82.30% +47.83%
===========================================
Files 684 455 -229
Lines 81483 54583 -26900
Branches 9180 0 -9180
===========================================
+ Hits 28090 44927 +16837
+ Misses 52972 9656 -43316
+ Partials 421 0 -421
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `28.12% <6.00%> (ø)` | |
| [examples/complete/juliaset/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvX19pbml0X18ucHk=) | | |
| [examples/cookbook/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvY29va2Jvb2svX19pbml0X18ucHk=) | | |
| [examples/complete/game/user\_score\_it\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvY29tcGxldGUvZ2FtZS91c2VyX3Njb3JlX2l0X3Rlc3QucHk=) | | |
| [testing/load\_tests/load\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-dGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdC5weQ==) | | |
| [examples/complete/game/game\_stats.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvY29tcGxldGUvZ2FtZS9nYW1lX3N0YXRzLnB5) | | |
| [runners/interactive/interactive\_runner\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-cnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXJfdGVzdC5weQ==) | | |
| [transforms/environments.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9lbnZpcm9ubWVudHMucHk=) | | |
| [examples/snippets/util\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvdXRpbF90ZXN0LnB5) | | |
| [examples/snippets/snippets\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHNfdGVzdC5weQ==) | | |
| ... and [1130 more](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=footer). Last update [e44c3c2...30fae7a](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] angoenka commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
angoenka commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-698526591
Can you please take a look at failing tests.
We can merge it after they are healthy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] angoenka commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
angoenka commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-698526591
Can you please take a look at failing tests.
We can merge it after they are healthy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-696271535
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-696271535
# [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request head (`BEAM-10200-2@30fae7a`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-head-commit).
> The diff coverage is `n/a`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] angoenka merged pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
angoenka merged pull request #12562:
URL: https://github.com/apache/beam/pull/12562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] angoenka commented on a change in pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
angoenka commented on a change in pull request #12562:
URL: https://github.com/apache/beam/pull/12562#discussion_r491759030
##########
File path: sdks/python/apache_beam/utils/profiler.py
##########
@@ -44,59 +42,91 @@
class Profile(object):
- """cProfile wrapper context for saving and logging profiler results."""
+ """cProfile and Heapy wrapper context for saving and logging profiler
+ results."""
SORTBY = 'cumulative'
def __init__(
self,
- profile_id,
- profile_location=None,
- log_results=False,
- file_copy_fn=None,
- time_prefix='%Y-%m-%d_%H_%M_%S-'):
+ profile_id, # type: str
+ profile_location=None, # type: Optional[str]
+ log_results=False, # type: bool
+ file_copy_fn=None, # type: Optional[Callable[[str, str], None]]
+ time_prefix='%Y-%m-%d_%H_%M_%S-', # type: str
+ enable_cpu_profiling=False, # type: bool
+ enable_memory_profiling=False, # type: bool
+ ):
+ """Creates a Profile object.
+
+ Args:
+ profile_id: Unique id of the profiling session.
+ profile_location: The file location where the profiling results will be
+ stored.
+ log_results: Log the result to console if true.
+ file_copy_fn: Lambda function for copying files.
+ time_prefix: Format of the timestamp prefix in profiling result files.
+ enable_cpu_profiling: CPU profiler will be enabled during the profiling
+ session.
+ enable_memory_profiling: Memory profiler will be enabled during the
+ profiling session, the profiler only records the newly allocated objects
+ in this session.
+ """
self.stats = None
self.profile_id = str(profile_id)
self.profile_location = profile_location
self.log_results = log_results
self.file_copy_fn = file_copy_fn or self.default_file_copy_fn
self.time_prefix = time_prefix
self.profile_output = None
+ self.enable_cpu_profiling = enable_cpu_profiling
+ self.enable_memory_profiling = enable_memory_profiling
def __enter__(self):
_LOGGER.info('Start profiling: %s', self.profile_id)
- self.profile = cProfile.Profile()
- self.profile.enable()
+ if self.enable_cpu_profiling:
+ self.profile = cProfile.Profile()
+ self.profile.enable()
+ if self.enable_memory_profiling:
+ try:
+ from guppy import hpy
+ self.hpy = hpy()
+ self.hpy.setrelheap()
+ except ImportError:
Review comment:
Let's log the import failure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
y1chi commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-674134380
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi removed a comment on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
y1chi removed a comment on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-674134204
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-696271535
# [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=h1) Report
> Merging [#12562](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e164d170eb6b5ec1dddd99f09e79dfb0147b84ae?el=desc) will **increase** coverage by `47.83%`.
> The diff coverage is `6.00%`.
[](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12562 +/- ##
===========================================
+ Coverage 34.47% 82.30% +47.83%
===========================================
Files 684 455 -229
Lines 81483 54583 -26900
Branches 9180 0 -9180
===========================================
+ Hits 28090 44927 +16837
+ Misses 52972 9656 -43316
+ Partials 421 0 -421
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `28.12% <6.00%> (ø)` | |
| [io/gcp/dicomclient.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-aW8vZ2NwL2RpY29tY2xpZW50LnB5) | | |
| [runners/job/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-cnVubmVycy9qb2IvX19pbml0X18ucHk=) | | |
| [dataframe/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZGF0YWZyYW1lL19faW5pdF9fLnB5) | | |
| [...pets/transforms/elementwise/withtimestamps\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS93aXRodGltZXN0YW1wc190ZXN0LnB5) | | |
| [runners/portability/flink\_runner.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-cnVubmVycy9wb3J0YWJpbGl0eS9mbGlua19ydW5uZXIucHk=) | | |
| [examples/complete/tfidf\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvY29tcGxldGUvdGZpZGZfdGVzdC5weQ==) | | |
| [runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-cnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | | |
| [runners/worker/statecache.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-cnVubmVycy93b3JrZXIvc3RhdGVjYWNoZS5weQ==) | | |
| [coders/observable\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-Y29kZXJzL29ic2VydmFibGVfdGVzdC5weQ==) | | |
| ... and [1130 more](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=footer). Last update [e44c3c2...b29fdcc](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-696271535
# [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=h1) Report
> Merging [#12562](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e164d170eb6b5ec1dddd99f09e79dfb0147b84ae?el=desc) will **increase** coverage by `47.83%`.
> The diff coverage is `6.00%`.
[](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12562 +/- ##
===========================================
+ Coverage 34.47% 82.30% +47.83%
===========================================
Files 684 455 -229
Lines 81483 54583 -26900
Branches 9180 0 -9180
===========================================
+ Hits 28090 44927 +16837
+ Misses 52972 9656 -43316
+ Partials 421 0 -421
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `28.12% <6.00%> (ø)` | |
| [testing/pipeline\_verifiers.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-dGVzdGluZy9waXBlbGluZV92ZXJpZmllcnMucHk=) | | |
| [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | | |
| [runners/interactive/pipeline\_fragment.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-cnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9mcmFnbWVudC5weQ==) | | |
| [dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZGF0YWZyYW1lL3NjaGVtYXMucHk=) | | |
| [utils/subprocess\_server\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-dXRpbHMvc3VicHJvY2Vzc19zZXJ2ZXJfdGVzdC5weQ==) | | |
| [runners/worker/sideinputs.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-cnVubmVycy93b3JrZXIvc2lkZWlucHV0cy5weQ==) | | |
| [examples/complete/game/leader\_board.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvY29tcGxldGUvZ2FtZS9sZWFkZXJfYm9hcmQucHk=) | | |
| [transforms/util.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy91dGlsLnB5) | | |
| [io/filebasedsink\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-aW8vZmlsZWJhc2Vkc2lua190ZXN0LnB5) | | |
| ... and [1130 more](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=footer). Last update [e44c3c2...30fae7a](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] angoenka commented on a change in pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
angoenka commented on a change in pull request #12562:
URL: https://github.com/apache/beam/pull/12562#discussion_r491759030
##########
File path: sdks/python/apache_beam/utils/profiler.py
##########
@@ -44,59 +42,91 @@
class Profile(object):
- """cProfile wrapper context for saving and logging profiler results."""
+ """cProfile and Heapy wrapper context for saving and logging profiler
+ results."""
SORTBY = 'cumulative'
def __init__(
self,
- profile_id,
- profile_location=None,
- log_results=False,
- file_copy_fn=None,
- time_prefix='%Y-%m-%d_%H_%M_%S-'):
+ profile_id, # type: str
+ profile_location=None, # type: Optional[str]
+ log_results=False, # type: bool
+ file_copy_fn=None, # type: Optional[Callable[[str, str], None]]
+ time_prefix='%Y-%m-%d_%H_%M_%S-', # type: str
+ enable_cpu_profiling=False, # type: bool
+ enable_memory_profiling=False, # type: bool
+ ):
+ """Creates a Profile object.
+
+ Args:
+ profile_id: Unique id of the profiling session.
+ profile_location: The file location where the profiling results will be
+ stored.
+ log_results: Log the result to console if true.
+ file_copy_fn: Lambda function for copying files.
+ time_prefix: Format of the timestamp prefix in profiling result files.
+ enable_cpu_profiling: CPU profiler will be enabled during the profiling
+ session.
+ enable_memory_profiling: Memory profiler will be enabled during the
+ profiling session, the profiler only records the newly allocated objects
+ in this session.
+ """
self.stats = None
self.profile_id = str(profile_id)
self.profile_location = profile_location
self.log_results = log_results
self.file_copy_fn = file_copy_fn or self.default_file_copy_fn
self.time_prefix = time_prefix
self.profile_output = None
+ self.enable_cpu_profiling = enable_cpu_profiling
+ self.enable_memory_profiling = enable_memory_profiling
def __enter__(self):
_LOGGER.info('Start profiling: %s', self.profile_id)
- self.profile = cProfile.Profile()
- self.profile.enable()
+ if self.enable_cpu_profiling:
+ self.profile = cProfile.Profile()
+ self.profile.enable()
+ if self.enable_memory_profiling:
+ try:
+ from guppy import hpy
+ self.hpy = hpy()
+ self.hpy.setrelheap()
+ except ImportError:
Review comment:
Let's log the import failure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
y1chi commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-696269645
> This also change the behavior from periodic heap dump to i think on demand heap dump.
> Shall we also summarize how it should be used?
Do you mean the deletion of MemoryReporter? I don't think it is used anywhere but we can also keep it if it is useful for users.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
y1chi commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-698529465
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
y1chi commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-698529465
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
y1chi commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-673661881
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-696271535
# [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request head (`BEAM-10200-2@30fae7a`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-head-commit).
> The diff coverage is `n/a`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
y1chi commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-696269645
> This also change the behavior from periodic heap dump to i think on demand heap dump.
> Shall we also summarize how it should be used?
Do you mean the deletion of MemoryReporter? I don't think it is used anywhere but we can also keep it if it is useful for users.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
y1chi commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-701672888
@angoenka I think this PR can be merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-696271535
# [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=h1) Report
> Merging [#12562](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e164d170eb6b5ec1dddd99f09e79dfb0147b84ae?el=desc) will **increase** coverage by `47.83%`.
> The diff coverage is `6.00%`.
[](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12562 +/- ##
===========================================
+ Coverage 34.47% 82.30% +47.83%
===========================================
Files 684 455 -229
Lines 81483 54583 -26900
Branches 9180 0 -9180
===========================================
+ Hits 28090 44927 +16837
+ Misses 52972 9656 -43316
+ Partials 421 0 -421
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `28.12% <6.00%> (ø)` | |
| [examples/complete/juliaset/setup.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvc2V0dXAucHk=) | | |
| [examples/snippets/snippets\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHNfdGVzdF9weTMucHk=) | | |
| [testing/load\_tests/sideinput\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-dGVzdGluZy9sb2FkX3Rlc3RzL3NpZGVpbnB1dF90ZXN0LnB5) | | |
| [io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-aW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | | |
| [io/aws/clients/s3/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-aW8vYXdzL2NsaWVudHMvczMvX19pbml0X18ucHk=) | | |
| [runners/interactive/testing/mock\_ipython.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-cnVubmVycy9pbnRlcmFjdGl2ZS90ZXN0aW5nL21vY2tfaXB5dGhvbi5weQ==) | | |
| [examples/complete/game/hourly\_team\_score\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvY29tcGxldGUvZ2FtZS9ob3VybHlfdGVhbV9zY29yZV90ZXN0LnB5) | | |
| [coders/standard\_coders\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-Y29kZXJzL3N0YW5kYXJkX2NvZGVyc190ZXN0LnB5) | | |
| [transforms/external\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9leHRlcm5hbF90ZXN0X3B5My5weQ==) | | |
| ... and [1130 more](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=footer). Last update [e44c3c2...30fae7a](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-696271535
# [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=h1) Report
> Merging [#12562](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e164d170eb6b5ec1dddd99f09e79dfb0147b84ae?el=desc) will **increase** coverage by `47.83%`.
> The diff coverage is `6.00%`.
[](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12562 +/- ##
===========================================
+ Coverage 34.47% 82.30% +47.83%
===========================================
Files 684 455 -229
Lines 81483 54583 -26900
Branches 9180 0 -9180
===========================================
+ Hits 28090 44927 +16837
+ Misses 52972 9656 -43316
+ Partials 421 0 -421
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `28.12% <6.00%> (ø)` | |
| [internal/http\_client\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-aW50ZXJuYWwvaHR0cF9jbGllbnRfdGVzdC5weQ==) | | |
| [...ners/portability/flink\_uber\_jar\_job\_server\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-cnVubmVycy9wb3J0YWJpbGl0eS9mbGlua191YmVyX2phcl9qb2Jfc2VydmVyX3Rlc3QucHk=) | | |
| [transforms/sql.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-dHJhbnNmb3Jtcy9zcWwucHk=) | | |
| [examples/snippets/snippets\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvc25pcHBldHNfdGVzdC5weQ==) | | |
| [io/hadoopfilesystem.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-aW8vaGFkb29wZmlsZXN5c3RlbS5weQ==) | | |
| [...les/snippets/transforms/aggregation/latest\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-ZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9sYXRlc3RfdGVzdC5weQ==) | | |
| [version.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-dmVyc2lvbi5weQ==) | | |
| [coders/avro\_record.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-Y29kZXJzL2F2cm9fcmVjb3JkLnB5) | | |
| [testing/load\_tests/co\_group\_by\_key\_test.py](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree#diff-dGVzdGluZy9sb2FkX3Rlc3RzL2NvX2dyb3VwX2J5X2tleV90ZXN0LnB5) | | |
| ... and [1130 more](https://codecov.io/gh/apache/beam/pull/12562/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=footer). Last update [e44c3c2...30fae7a](https://codecov.io/gh/apache/beam/pull/12562?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on pull request #12562: [BEAM-10200] Respect profile_memory option and add memory profiler to…
Posted by GitBox <gi...@apache.org>.
y1chi commented on pull request #12562:
URL: https://github.com/apache/beam/pull/12562#issuecomment-674134204
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org