You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/10/09 16:06:43 UTC
[GitHub] [beam] chadrik opened a new pull request #13060: [BEAM-7746] Fix typing in runners
chadrik opened a new pull request #13060:
URL: https://github.com/apache/beam/pull/13060
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on a change in pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on a change in pull request #13060:
URL: https://github.com/apache/beam/pull/13060#discussion_r506778717
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py
##########
@@ -539,6 +548,7 @@ def merge_results(last_result):
else:
data_input = deferred_inputs
input_timers = fired_timers
+ # FIXME: this seems unused, and produces an attr-defined error
bundle_manager._registered = True
Review comment:
Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13060:
URL: https://github.com/apache/beam/pull/13060#issuecomment-714022908
windows failures are a known flake (https://issues.apache.org/jira/browse/BEAM-10921)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #13060:
URL: https://github.com/apache/beam/pull/13060#issuecomment-714022462
Run Python_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on a change in pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on a change in pull request #13060:
URL: https://github.com/apache/beam/pull/13060#discussion_r505725707
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py
##########
@@ -62,13 +64,15 @@
from apache_beam.runners.portability.fn_api_runner.translations import only_element
from apache_beam.runners.portability.fn_api_runner.worker_handlers import WorkerHandlerManager
from apache_beam.transforms import environments
-from apache_beam.utils import profiler
from apache_beam.utils import proto_utils
from apache_beam.utils import thread_pool_executor
+from apache_beam.utils.profiler import Profile
Review comment:
I changed this import because `profiler` was conflicting with some local variable names, and this was the simplest resolution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on a change in pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on a change in pull request #13060:
URL: https://github.com/apache/beam/pull/13060#discussion_r506778596
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py
##########
@@ -355,9 +361,9 @@ def _run_bundle_multiple_times_for_testing(
self,
runner_execution_context, # type: execution.FnApiRunnerExecutionContext
bundle_manager, # type: BundleManager
- data_input,
+ data_input, # type: Dict[str, PartitionableBuffer]
data_output, # type: DataOutput
- fired_timers,
+ fired_timers, # type: Mapping[Tuple[str, str], execution.PartitionableBuffer]
Review comment:
resolved
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #13060:
URL: https://github.com/apache/beam/pull/13060#discussion_r506748112
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py
##########
@@ -539,6 +548,7 @@ def merge_results(last_result):
else:
data_input = deferred_inputs
input_timers = fired_timers
+ # FIXME: this seems unused, and produces an attr-defined error
bundle_manager._registered = True
Review comment:
Yes, this should not be needed anymore.
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
##########
@@ -1062,34 +1104,41 @@ def close(self):
class ControlFuture(object):
- def __init__(self, instruction_id, response=None):
+ def __init__(self,
+ instruction_id, # type: str
+ response=None # type: Optional[beam_fn_api_pb2.InstructionResponse]
+ ):
+ # type: (...) -> None
self.instruction_id = instruction_id
- if response:
- self._response = response
- else:
- self._response = None
+ self._response = response
+ if response is None:
self._condition = threading.Condition()
- self._exception = None
+ self._exception = None # type: Optional[Exception]
def is_done(self):
+ # type: () -> bool
return self._response is not None
def set(self, response):
+ # type: (beam_fn_api_pb2.InstructionResponse) -> None
with self._condition:
self._response = response
self._condition.notify_all()
def get(self, timeout=None):
+ # type: (Optional[float]) -> beam_fn_api_pb2.InstructionResponse
if not self._response and not self._exception:
with self._condition:
if not self._response and not self._exception:
self._condition.wait(timeout)
if self._exception:
raise self._exception
else:
+ assert self._response is not None
Review comment:
Correct, either an exception will be thrown, a timeout will be thrown, or the response is non-None.
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/execution.py
##########
@@ -227,12 +251,24 @@ def __iter__(self):
"""
return itertools.chain(*self.partition(1))
+ # these should never be accessed, but they allow this class to meet the
Review comment:
How about making them raise errors rather than be no-ops?
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py
##########
@@ -307,6 +308,9 @@ def executable_stage_transform(
for side in side_inputs
},
main_input=main_input_id)
+ # at this point we should have resolved an environment, as the key of
+ # components.environments cannot be None.
+ assert self.environment is not None
Review comment:
Yes, that looks safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on a change in pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on a change in pull request #13060:
URL: https://github.com/apache/beam/pull/13060#discussion_r505933134
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
##########
@@ -1062,34 +1104,41 @@ def close(self):
class ControlFuture(object):
- def __init__(self, instruction_id, response=None):
+ def __init__(self,
+ instruction_id, # type: str
+ response=None # type: Optional[beam_fn_api_pb2.InstructionResponse]
+ ):
+ # type: (...) -> None
self.instruction_id = instruction_id
- if response:
- self._response = response
- else:
- self._response = None
+ self._response = response
+ if response is None:
self._condition = threading.Condition()
- self._exception = None
+ self._exception = None # type: Optional[Exception]
def is_done(self):
+ # type: () -> bool
return self._response is not None
def set(self, response):
+ # type: (beam_fn_api_pb2.InstructionResponse) -> None
with self._condition:
self._response = response
self._condition.notify_all()
def get(self, timeout=None):
+ # type: (Optional[float]) -> beam_fn_api_pb2.InstructionResponse
if not self._response and not self._exception:
with self._condition:
if not self._response and not self._exception:
self._condition.wait(timeout)
if self._exception:
raise self._exception
else:
+ assert self._response is not None
Review comment:
It would be good to get confirmation that this assert is correct. AFAICT this method should never return None.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on pull request #13060:
URL: https://github.com/apache/beam/pull/13060#issuecomment-714188423
Woohoo!
On Wed, Oct 21, 2020 at 7:41 PM Pablo <no...@github.com> wrote:
> Merged #13060 <https://github.com/apache/beam/pull/13060> into master.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/13060#event-3906778383>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAPOE3UCUGF64PD35DNASDSL6LUXANCNFSM4SKIJL5Q>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #13060:
URL: https://github.com/apache/beam/pull/13060#issuecomment-709511400
/cc @udim
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on a change in pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on a change in pull request #13060:
URL: https://github.com/apache/beam/pull/13060#discussion_r505932577
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py
##########
@@ -355,9 +361,9 @@ def _run_bundle_multiple_times_for_testing(
self,
runner_execution_context, # type: execution.FnApiRunnerExecutionContext
bundle_manager, # type: BundleManager
- data_input,
+ data_input, # type: Dict[str, PartitionableBuffer]
data_output, # type: DataOutput
- fired_timers,
+ fired_timers, # type: Mapping[Tuple[str, str], execution.PartitionableBuffer]
Review comment:
note to self: standardize on `PartitionableBuffer` vs `execution.PartitionableBuffer`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #13060:
URL: https://github.com/apache/beam/pull/13060
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on a change in pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on a change in pull request #13060:
URL: https://github.com/apache/beam/pull/13060#discussion_r506776533
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/execution.py
##########
@@ -227,12 +251,24 @@ def __iter__(self):
"""
return itertools.chain(*self.partition(1))
+ # these should never be accessed, but they allow this class to meet the
Review comment:
I prefer the current solution because it's reasonable and safe to call `clear()` and `reset()` on this class and have it do nothing: there's simply nothing to clear or reset. It seems wrong to meet the protocol definition, but then error if the methods are called: that's not really meeting the protocol.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on a change in pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on a change in pull request #13060:
URL: https://github.com/apache/beam/pull/13060#discussion_r505726270
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py
##########
@@ -307,6 +308,9 @@ def executable_stage_transform(
for side in side_inputs
},
main_input=main_input_id)
+ # at this point we should have resolved an environment, as the key of
+ # components.environments cannot be None.
+ assert self.environment is not None
Review comment:
We discussed this before and you said this was safe, but just wanted to double check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on a change in pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on a change in pull request #13060:
URL: https://github.com/apache/beam/pull/13060#discussion_r505725952
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py
##########
@@ -539,6 +548,7 @@ def merge_results(last_result):
else:
data_input = deferred_inputs
input_timers = fired_timers
+ # FIXME: this seems unused, and produces an attr-defined error
bundle_manager._registered = True
Review comment:
Can I safely get rid of this attribute?
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py
##########
@@ -307,6 +308,9 @@ def executable_stage_transform(
for side in side_inputs
},
main_input=main_input_id)
+ # at this point we should have resolved an environment, as the key of
+ # components.environments cannot be None.
+ assert self.environment is not None
Review comment:
We discussed this before and said this was safe, but just wanted to double check.
##########
File path: sdks/python/apache_beam/utils/profiler.py
##########
@@ -113,7 +116,10 @@ def __exit__(self, *args):
h = self.hpy.heap()
heap_dump_data = '%s\n%s' % (h, h.more)
self._upload_profile_data(
- 'memory_profile', heap_dump_data, write_binary=False)
+ self.profile_location,
+ 'memory_profile',
+ heap_dump_data,
+ write_binary=False)
Review comment:
I decided to change this to pass in `self.profile_location` because in this context, where it's called, we know that it's not `None`. Let's us avoid calling `assert` inside `self._upload_profile_data`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on pull request #13060:
URL: https://github.com/apache/beam/pull/13060#issuecomment-709484661
R: @robertwb
we're in the final stretch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chadrik commented on pull request #13060: [BEAM-7746] Fix typing in runners
Posted by GitBox <gi...@apache.org>.
chadrik commented on pull request #13060:
URL: https://github.com/apache/beam/pull/13060#issuecomment-712414839
Lint is passing now. I think the rest are flakes, but would be good for someone to double check
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org