You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/29 13:03:40 UTC
[GitHub] [beam] Tesio opened a new pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Tesio opened a new pull request #11568:
URL: https://github.com/apache/beam/pull/11568
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Tesio edited a comment on pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
Tesio edited a comment on pull request #11568:
URL: https://github.com/apache/beam/pull/11568#issuecomment-621190636
R: @udim
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
udim commented on pull request #11568:
URL: https://github.com/apache/beam/pull/11568#issuecomment-621986469
And now the previous comment launched tests on ffefeba
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Tesio commented on pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
Tesio commented on pull request #11568:
URL: https://github.com/apache/beam/pull/11568#issuecomment-621190636
R: @udim
Kindly review the request
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
udim commented on pull request #11568:
URL: https://github.com/apache/beam/pull/11568#issuecomment-622006811
sigh, again ran the test on the previous commit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on a change in pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #11568:
URL: https://github.com/apache/beam/pull/11568#discussion_r417705115
##########
File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging_test.py
##########
@@ -0,0 +1,109 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit test for the streaming wordcount example with debug."""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import unittest
+
+import mock
+
+import apache_beam as beam
+from apache_beam.examples import streaming_wordcount_debugging
+from apache_beam.testing.test_stream import TestStream
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+# Protect against environments where the PubSub library is not available.
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from google.cloud import pubsub
+except ImportError:
+ pubsub = None
+# pylint: enable=wrong-import-order, wrong-import-position
+
+
+class StreamingWordcountDebugging(unittest.TestCase):
+ @unittest.skipIf(pubsub is None, 'GCP dependencies are not installed')
+ @mock.patch('apache_beam.io.ReadFromPubSub')
+ @mock.patch('apache_beam.io.WriteToPubSub')
+ def test_streaming_wordcount_debugging(self, *unused_mocks):
+ def FakeReadFromPubSub(topic=None, subscription=None, values=None):
+ expected_topic = topic
+ expected_subscription = subscription
+
+ def _inner(topic=None, subscription=None):
+ assert topic == expected_topic
+ assert subscription == expected_subscription
+ return TestStream().add_elements(values)
+
+ return _inner
+
+ class AssertTransform(beam.PTransform):
+ def __init__(self, matcher):
+ self.matcher = matcher
+
+ def expand(self, pcoll):
+ assert_that(pcoll, self.matcher)
+
+ def FakeWriteToPubSub(topic=None, values=None):
+ expected_topic = topic
+
+ def _inner(topic=None, subscription=None):
+ assert topic == expected_topic
+ return AssertTransform(equal_to(values))
+
+ return _inner
+
+ input_topic = 'projects/fake-beam-test-project/topic/intopic'
+ input_values = [
+ '150', '151', '152', '153', '154', '210', '211', '212', '213', '214'
+ ]
+ output_topic = 'projects/fake-beam-test-project/topic/outtopic'
+ output_values = [
+ '150: 1',
+ '151: 1',
+ '152: 1',
+ '153: 1',
+ '154: 1',
+ '210: 1',
+ '211: 1',
+ '212: 1',
+ '213: 1',
+ '214: 1'
+ ]
+ beam.io.ReadFromPubSub = (
+ FakeReadFromPubSub(
+ topic=input_topic,
+ values=list(x.encode('utf-8') for x in input_values)))
+ beam.io.WriteToPubSub = (
+ FakeWriteToPubSub(
+ topic=output_topic,
+ values=list(x.encode('utf-8') for x in output_values)))
+ streaming_wordcount_debugging.run([
Review comment:
I believe you're missing a save_main_session=False here, otherwise it won't run successfully with the pytest-xdist plugin (pickle error). You can test it on your machine with `tox -e py37-cloud`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
udim commented on pull request #11568:
URL: https://github.com/apache/beam/pull/11568#issuecomment-622006543
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
udim commented on pull request #11568:
URL: https://github.com/apache/beam/pull/11568#issuecomment-621506456
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
udim commented on pull request #11568:
URL: https://github.com/apache/beam/pull/11568#issuecomment-621985196
For some reason my comment launched tests on dbf0396 instead of ffefeba. Trying again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
udim commented on pull request #11568:
URL: https://github.com/apache/beam/pull/11568#issuecomment-621981150
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
udim commented on pull request #11568:
URL: https://github.com/apache/beam/pull/11568#issuecomment-621506645
Waiting to check that streaming_wordcount_debugging_test runs in python precommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Tesio commented on a change in pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
Tesio commented on a change in pull request #11568:
URL: https://github.com/apache/beam/pull/11568#discussion_r417888379
##########
File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging_test.py
##########
@@ -0,0 +1,109 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit test for the streaming wordcount example with debug."""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import unittest
+
+import mock
+
+import apache_beam as beam
+from apache_beam.examples import streaming_wordcount_debugging
+from apache_beam.testing.test_stream import TestStream
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+# Protect against environments where the PubSub library is not available.
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from google.cloud import pubsub
+except ImportError:
+ pubsub = None
+# pylint: enable=wrong-import-order, wrong-import-position
+
+
+class StreamingWordcountDebugging(unittest.TestCase):
+ @unittest.skipIf(pubsub is None, 'GCP dependencies are not installed')
+ @mock.patch('apache_beam.io.ReadFromPubSub')
+ @mock.patch('apache_beam.io.WriteToPubSub')
+ def test_streaming_wordcount_debugging(self, *unused_mocks):
+ def FakeReadFromPubSub(topic=None, subscription=None, values=None):
+ expected_topic = topic
+ expected_subscription = subscription
+
+ def _inner(topic=None, subscription=None):
+ assert topic == expected_topic
+ assert subscription == expected_subscription
+ return TestStream().add_elements(values)
+
+ return _inner
+
+ class AssertTransform(beam.PTransform):
+ def __init__(self, matcher):
+ self.matcher = matcher
+
+ def expand(self, pcoll):
+ assert_that(pcoll, self.matcher)
+
+ def FakeWriteToPubSub(topic=None, values=None):
+ expected_topic = topic
+
+ def _inner(topic=None, subscription=None):
+ assert topic == expected_topic
+ return AssertTransform(equal_to(values))
+
+ return _inner
+
+ input_topic = 'projects/fake-beam-test-project/topic/intopic'
+ input_values = [
+ '150', '151', '152', '153', '154', '210', '211', '212', '213', '214'
+ ]
+ output_topic = 'projects/fake-beam-test-project/topic/outtopic'
+ output_values = [
+ '150: 1',
+ '151: 1',
+ '152: 1',
+ '153: 1',
+ '154: 1',
+ '210: 1',
+ '211: 1',
+ '212: 1',
+ '213: 1',
+ '214: 1'
+ ]
+ beam.io.ReadFromPubSub = (
+ FakeReadFromPubSub(
+ topic=input_topic,
+ values=list(x.encode('utf-8') for x in input_values)))
+ beam.io.WriteToPubSub = (
+ FakeWriteToPubSub(
+ topic=output_topic,
+ values=list(x.encode('utf-8') for x in output_values)))
+ streaming_wordcount_debugging.run([
Review comment:
Yes, I missed that param. Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on pull request #11568: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Posted by GitBox <gi...@apache.org>.
udim commented on pull request #11568:
URL: https://github.com/apache/beam/pull/11568#issuecomment-622008669
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org