You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/10/12 23:17:10 UTC
[GitHub] [beam] KevinGG opened a new pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
KevinGG opened a new pull request #13080:
URL: https://github.com/apache/beam/pull/13080
1. Replaced `capture` with `record` in top level APIs accessible by
users.
2. Fixed some formatting in the loggings.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidyan74 commented on a change in pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
davidyan74 commented on a change in pull request #13080:
URL: https://github.com/apache/beam/pull/13080#discussion_r503608190
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -52,98 +53,116 @@
class Options(interactive_options.InteractiveOptions):
"""Options that guide how Interactive Beam works."""
@property
- def enable_capture_replay(self):
- """Whether replayable source data capture should be replayed for multiple
- PCollection evaluations and pipeline runs as long as the data captured is
+ def enable_record_replay(self):
Review comment:
enable_recording_replay. Basically, if "capture" is used as a noun, change it to "recording" instead of "record", since "record" used as a noun usually means an individual record instead of the act of recording. Please check all occurrences.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on a change in pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #13080:
URL: https://github.com/apache/beam/pull/13080#discussion_r503622541
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -91,23 +91,23 @@ def recordable_sources(self):
return self.capture_control._capturable_sources
@property
- def record_duration(self):
- """The data record of sources ends as soon as the background caching job
+ def recording_duration(self):
+ """The data recording of sources ends as soon as the background caching job
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on a change in pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #13080:
URL: https://github.com/apache/beam/pull/13080#discussion_r503617273
##########
File path: sdks/python/apache_beam/runners/interactive/background_caching_job.py
##########
@@ -19,21 +19,21 @@
For internal use only; no backwards-compatibility guarantees.
-A background caching job is a job that captures events for all capturable
+A background caching job is a job that records events for all recordable
sources of a given pipeline. With Interactive Beam, one such job is started when
a pipeline run happens (which produces a main job in contrast to the background
caching job) and meets the following conditions:
- #. The pipeline contains capturable sources, configured through
- interactive_beam.options.capturable_sources.
+ #. The pipeline contains recordable sources, configured through
+ interactive_beam.options.recordable_sources.
#. No such background job is running.
#. No such background job has completed successfully and the cached events are
- still valid (invalidated when capturable sources change in the pipeline).
+ still valid (invalidated when recordable sources change in the pipeline).
Once started, the background caching job runs asynchronously until it hits some
-capture limit configured in interactive_beam.options. Meanwhile, the main job
+record limit configured in interactive_beam.options. Meanwhile, the main job
Review comment:
Switched from `record` to `recording`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #13080:
URL: https://github.com/apache/beam/pull/13080#issuecomment-707393985
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidyan74 commented on a change in pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
davidyan74 commented on a change in pull request #13080:
URL: https://github.com/apache/beam/pull/13080#discussion_r503617854
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -91,23 +91,23 @@ def recordable_sources(self):
return self.capture_control._capturable_sources
@property
- def record_duration(self):
- """The data record of sources ends as soon as the background caching job
+ def recording_duration(self):
+ """The data recording of sources ends as soon as the background caching job
Review comment:
background source recording job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidyan74 commented on a change in pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
davidyan74 commented on a change in pull request #13080:
URL: https://github.com/apache/beam/pull/13080#discussion_r503607479
##########
File path: sdks/python/apache_beam/runners/interactive/background_caching_job.py
##########
@@ -19,21 +19,21 @@
For internal use only; no backwards-compatibility guarantees.
-A background caching job is a job that captures events for all capturable
+A background caching job is a job that records events for all recordable
Review comment:
"Background caching job" -> "Background source recording job". Please check all occurrences.
##########
File path: sdks/python/apache_beam/runners/interactive/background_caching_job.py
##########
@@ -56,7 +56,7 @@ class BackgroundCachingJob(object):
"""A simple abstraction that controls necessary components of a timed and
space limited background caching job.
- A background caching job successfully completes source data capture in 2
+ A background caching job successfully completes source data record in 2
Review comment:
recording
##########
File path: sdks/python/apache_beam/runners/interactive/background_caching_job.py
##########
@@ -165,9 +165,9 @@ def is_background_caching_job_needed(user_pipeline):
# If this is True, we can invalidate a previous done/running job if there is
# one.
cache_changed = is_source_to_cache_changed(user_pipeline)
- # When capture replay is disabled, cache is always needed for capturable
+ # When record replay is disabled, cache is always needed for recordable
Review comment:
recording
##########
File path: sdks/python/apache_beam/runners/interactive/background_caching_job.py
##########
@@ -165,9 +165,9 @@ def is_background_caching_job_needed(user_pipeline):
# If this is True, we can invalidate a previous done/running job if there is
# one.
cache_changed = is_source_to_cache_changed(user_pipeline)
- # When capture replay is disabled, cache is always needed for capturable
+ # When record replay is disabled, cache is always needed for recordable
# sources (if any).
- if need_cache and not ie.current_env().options.enable_capture_replay:
+ if need_cache and not ie.current_env().options.enable_record_replay:
Review comment:
enable_recording_replay
##########
File path: sdks/python/apache_beam/runners/interactive/background_caching_job.py
##########
@@ -19,21 +19,21 @@
For internal use only; no backwards-compatibility guarantees.
-A background caching job is a job that captures events for all capturable
+A background caching job is a job that records events for all recordable
sources of a given pipeline. With Interactive Beam, one such job is started when
a pipeline run happens (which produces a main job in contrast to the background
caching job) and meets the following conditions:
- #. The pipeline contains capturable sources, configured through
- interactive_beam.options.capturable_sources.
+ #. The pipeline contains recordable sources, configured through
+ interactive_beam.options.recordable_sources.
#. No such background job is running.
#. No such background job has completed successfully and the cached events are
- still valid (invalidated when capturable sources change in the pipeline).
+ still valid (invalidated when recordable sources change in the pipeline).
Once started, the background caching job runs asynchronously until it hits some
-capture limit configured in interactive_beam.options. Meanwhile, the main job
+record limit configured in interactive_beam.options. Meanwhile, the main job
Review comment:
recording
##########
File path: sdks/python/apache_beam/runners/interactive/background_caching_job.py
##########
@@ -301,13 +301,13 @@ def sizeof_fmt(num, suffix='B'):
'In order to have a deterministic replay, a segment of data will '
'be recorded from all sources for %s seconds or until a total of '
'%s have been written to disk.',
- options.capture_duration.total_seconds(),
- sizeof_fmt(options.capture_size_limit))
+ options.record_duration.total_seconds(),
Review comment:
recording_duration
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -52,98 +53,116 @@
class Options(interactive_options.InteractiveOptions):
"""Options that guide how Interactive Beam works."""
@property
- def enable_capture_replay(self):
- """Whether replayable source data capture should be replayed for multiple
- PCollection evaluations and pipeline runs as long as the data captured is
+ def enable_record_replay(self):
Review comment:
enable_recording_replay. Basically, if "capture" is used as a noun, change it to "recording" instead of "record", since "record" might have a notion of an individual record. Please check all occurrences.
##########
File path: sdks/python/apache_beam/runners/interactive/background_caching_job.py
##########
@@ -301,13 +301,13 @@ def sizeof_fmt(num, suffix='B'):
'In order to have a deterministic replay, a segment of data will '
'be recorded from all sources for %s seconds or until a total of '
'%s have been written to disk.',
- options.capture_duration.total_seconds(),
- sizeof_fmt(options.capture_size_limit))
+ options.record_duration.total_seconds(),
+ sizeof_fmt(options.record_size_limit))
Review comment:
recording_size_limit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #13080:
URL: https://github.com/apache/beam/pull/13080#issuecomment-707399997
Run Python_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #13080:
URL: https://github.com/apache/beam/pull/13080#issuecomment-707397613
R: @pabloem
PTAL, thx!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on a change in pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #13080:
URL: https://github.com/apache/beam/pull/13080#discussion_r503617113
##########
File path: sdks/python/apache_beam/runners/interactive/background_caching_job.py
##########
@@ -19,21 +19,21 @@
For internal use only; no backwards-compatibility guarantees.
-A background caching job is a job that captures events for all capturable
+A background caching job is a job that records events for all recordable
Review comment:
Switched to background source recording jobs for all the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay merged pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
aaltay merged pull request #13080:
URL: https://github.com/apache/beam/pull/13080
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #13080:
URL: https://github.com/apache/beam/pull/13080#issuecomment-707462094
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #13080:
URL: https://github.com/apache/beam/pull/13080#issuecomment-707417478
R: @davidyan74
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #13080: [BEAM-11056] Fix warning message and rename old APIs
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #13080:
URL: https://github.com/apache/beam/pull/13080#issuecomment-707449603
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org