You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/02/27 21:19:44 UTC
[GitHub] [beam] rohdesamuel opened a new pull request #10994:
Teststreamservice merge
rohdesamuel opened a new pull request #10994: Teststreamservice merge
URL: https://github.com/apache/beam/pull/10994
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rohdesamuel commented on a change in pull request #10994:
[BEAM-8335] TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#discussion_r387992717
##########
File path: sdks/python/apache_beam/testing/test_stream_service.py
##########
@@ -52,5 +56,9 @@ def stop(self):
def Events(self, request, context):
"""Streams back all of the events from the streaming cache."""
- for e in self._events:
+ # TODO(srohde): Once we get rid of the CacheManager, get rid of this 'full'
+ # label.
+ reader = self._reader.read_multiple([['full', key]
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-597206306
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595420007
there are some test failures
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595361581
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595065101
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-594948530
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-596833856
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595361710
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-596823224
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595035694
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rohdesamuel commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-592695120
This was rebased onto the cherry-pick of PR/10497 (ReverseTestStream implementation)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rohdesamuel commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-594768428
> Please see and address comments at #11005
Ack, addressed the comment for the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595377475
Run PythonLint PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595427003
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rohdesamuel commented on a change in pull request #10994:
[BEAM-8335] TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#discussion_r387992755
##########
File path: sdks/python/apache_beam/testing/test_stream.py
##########
@@ -236,14 +235,19 @@ class TestStream(PTransform):
"""
def __init__(
- self, coder=coders.FastPrimitivesCoder(), events=None, output_tags=None):
+ self,
+ coder=coders.FastPrimitivesCoder(),
+ events=None,
+ output_tags=None,
+ endpoint=None):
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #10994:
[BEAM-8335] TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#discussion_r387976936
##########
File path: sdks/python/apache_beam/runners/direct/test_stream_impl.py
##########
@@ -226,17 +237,53 @@ def expand(self, pcoll):
def _infer_output_coder(self, input_type=None, input_coder=None):
return self.coder
- def _events_from_script(self, index):
- yield self._events[index]
-
- def events(self, index):
- return self._events_from_script(index)
-
- def begin(self):
- return 0
-
- def end(self, index):
- return index >= len(self._events)
+ @staticmethod
+ def events_from_script(events):
+ """Yields the in-memory events.
+ """
+ return itertools.chain(events)
Review comment:
Let's add an assert to make sure that this is only called when we have in-memory events vs rpc events (also in the method below).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595458238
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rohdesamuel commented on a change in pull request #10994:
[BEAM-8335] TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#discussion_r387973912
##########
File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py
##########
@@ -527,9 +548,21 @@ def process_element(self, element):
for event in self.test_stream._set_up(self.test_stream.output_tags):
events.append(event)
- events += [e for e in self.test_stream.events(self.current_index)]
+ # Index into the global state of all the different TestStream event streams.
+ # Retrieve this TestStream's event stream and read from it.
+ try:
+ events = [next(self.test_stream_events[self.event_index])]
Review comment:
This should be `events +=` instead of `events =`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rohdesamuel commented on a change in pull request #10994:
[BEAM-8335] TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#discussion_r387992663
##########
File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py
##########
@@ -527,9 +548,21 @@ def process_element(self, element):
for event in self.test_stream._set_up(self.test_stream.output_tags):
events.append(event)
- events += [e for e in self.test_stream.events(self.current_index)]
+ # Index into the global state of all the different TestStream event streams.
+ # Retrieve this TestStream's event stream and read from it.
+ try:
+ events = [next(self.test_stream_events[self.event_index])]
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-596825310
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-596827343
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595436740
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem merged pull request #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595361504
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rohdesamuel commented on a change in pull request #10994:
[BEAM-8335] TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#discussion_r387993084
##########
File path: sdks/python/apache_beam/runners/direct/test_stream_impl.py
##########
@@ -226,17 +237,53 @@ def expand(self, pcoll):
def _infer_output_coder(self, input_type=None, input_coder=None):
return self.coder
- def _events_from_script(self, index):
- yield self._events[index]
-
- def events(self, index):
- return self._events_from_script(index)
-
- def begin(self):
- return 0
-
- def end(self, index):
- return index >= len(self._events)
+ @staticmethod
+ def events_from_script(events):
+ """Yields the in-memory events.
+ """
+ return itertools.chain(events)
Review comment:
I added an assert in the TestStream constructor so that this shouldn't happen at pipeline run-time but construction time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595377514
Run PythonFormatter PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #10994:
[BEAM-8335] TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#discussion_r387979245
##########
File path: sdks/python/apache_beam/testing/test_stream_service.py
##########
@@ -52,5 +56,9 @@ def stop(self):
def Events(self, request, context):
"""Streams back all of the events from the streaming cache."""
- for e in self._events:
+ # TODO(srohde): Once we get rid of the CacheManager, get rid of this 'full'
+ # label.
+ reader = self._reader.read_multiple([['full', key]
Review comment:
s/key/tag/ ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-596829649
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-594884067
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rohdesamuel commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-594912381
ugh: `ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-596825273
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595439800
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #10994:
[BEAM-8335] TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#discussion_r387966654
##########
File path: sdks/python/apache_beam/testing/test_stream_service.py
##########
@@ -52,5 +56,9 @@ def stop(self):
def Events(self, request, context):
"""Streams back all of the events from the streaming cache."""
- for e in self._events:
+ # TODO(srohde): Once we get rid of the CacheManager, get rid of this 'full'
+ # label.
+ reader = self._reader.read_multiple([['full', key]
Review comment:
```suggestion
reader = self._reader.read_multiple([('full', key)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-596823393
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #10994:
[BEAM-8335] TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#discussion_r387967821
##########
File path: sdks/python/apache_beam/testing/test_stream.py
##########
@@ -236,14 +235,19 @@ class TestStream(PTransform):
"""
def __init__(
- self, coder=coders.FastPrimitivesCoder(), events=None, output_tags=None):
+ self,
+ coder=coders.FastPrimitivesCoder(),
+ events=None,
+ output_tags=None,
+ endpoint=None):
Review comment:
please document inputs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595377292
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-596831495
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-595427106
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on issue #10994: [BEAM-8335]
TeststreamService integration with DirectRunner
Posted by GitBox <gi...@apache.org>.
robertwb commented on issue #10994: [BEAM-8335] TeststreamService integration with DirectRunner
URL: https://github.com/apache/beam/pull/10994#issuecomment-596026188
My comments are in the discussions on the other PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services