You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/02/28 11:40:45 UTC
[GitHub] [beam] kamilwu opened a new pull request #11000: Add integration
test for AnnotateImage transform
kamilwu opened a new pull request #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] EDjur commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
EDjur commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-592529995
And seems we need an infra admin to enable the Vision API in the build project.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-592487521
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-592489417
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-593351271
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-593937852
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] EDjur commented on a change in pull request #11000: Add
integration test for AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
EDjur commented on a change in pull request #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#discussion_r385715132
##########
File path: sdks/python/apache_beam/ml/gcp/visionml_test_it.py
##########
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import unittest
+
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.ml.gcp.visionml import AnnotateImage
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+try:
+ from google.cloud.vision import types
+except ImportError:
+ types = None
+
+
+def extract(response):
+ for r in response.responses:
+ for text_annotation in r.text_annotations:
+ yield text_annotation.description
+
+
+@attr('IT')
+class VisionMlTestIT(unittest.TestCase):
Review comment:
LGTM.
One small question though: Should we guard against environments where vision isn't installed by having the `unittest.skipIf...`?
```suggestion
@unittest.skipIf(types is None, 'GCP dependencies are not installed')
@attr('IT')
class VisionMlTestIT(unittest.TestCase):
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] EDjur commented on a change in pull request #11000: Add
integration test for AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
EDjur commented on a change in pull request #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#discussion_r385715132
##########
File path: sdks/python/apache_beam/ml/gcp/visionml_test_it.py
##########
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import unittest
+
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.ml.gcp.visionml import AnnotateImage
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+try:
+ from google.cloud.vision import types
+except ImportError:
+ types = None
+
+
+def extract(response):
+ for r in response.responses:
+ for text_annotation in r.text_annotations:
+ yield text_annotation.description
+
+
+@attr('IT')
+class VisionMlTestIT(unittest.TestCase):
Review comment:
LGTM.
One small question though: Should we guard against environments where vision isn't installed by having the `unittest.skipIf...`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] EDjur commented on a change in pull request #11000: Add
integration test for AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
EDjur commented on a change in pull request #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#discussion_r385715132
##########
File path: sdks/python/apache_beam/ml/gcp/visionml_test_it.py
##########
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import unittest
+
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.ml.gcp.visionml import AnnotateImage
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+try:
+ from google.cloud.vision import types
+except ImportError:
+ types = None
+
+
+def extract(response):
+ for r in response.responses:
+ for text_annotation in r.text_annotations:
+ yield text_annotation.description
+
+
+@attr('IT')
+class VisionMlTestIT(unittest.TestCase):
Review comment:
LGTM.
One small question though: Should we guard against environments where vision isn't installed by having the `unittest.skipIf...`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu merged pull request #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu merged pull request #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-592560759
Thanks @EDjur. I've already asked for enabling the Vision API, let's just wait until this happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-592524425
R: @EDjur
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-593287584
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on a change in pull request #11000: Add
integration test for AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#discussion_r385742358
##########
File path: sdks/python/apache_beam/ml/gcp/visionml_test_it.py
##########
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import unittest
+
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.ml.gcp.visionml import AnnotateImage
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+try:
+ from google.cloud.vision import types
+except ImportError:
+ types = None
+
+
+def extract(response):
+ for r in response.responses:
+ for text_annotation in r.text_annotations:
+ yield text_annotation.description
+
+
+@attr('IT')
+class VisionMlTestIT(unittest.TestCase):
Review comment:
Fair enough.
The test will always be skipped if `TestPipeline(is_integration_test=True)` is present and `--test-pipeline-options` is not provided (it's only provided in gcp environments). But maybe more explicit guard could be useful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-593940615
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] EDjur commented on a change in pull request #11000: Add
integration test for AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
EDjur commented on a change in pull request #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#discussion_r385743721
##########
File path: sdks/python/apache_beam/ml/gcp/visionml_test_it.py
##########
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import unittest
+
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.ml.gcp.visionml import AnnotateImage
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+try:
+ from google.cloud.vision import types
+except ImportError:
+ types = None
+
+
+def extract(response):
+ for r in response.responses:
+ for text_annotation in r.text_annotations:
+ yield text_annotation.description
+
+
+@attr('IT')
+class VisionMlTestIT(unittest.TestCase):
Review comment:
I see. Yeah fair either way then. LGTM :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-593980382
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11000: Add integration test for
AnnotateImage transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11000: Add integration test for AnnotateImage transform
URL: https://github.com/apache/beam/pull/11000#issuecomment-594027219
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services