You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/02/27 16:36:02 UTC
[GitHub] [beam] kamilwu opened a new pull request #10986: Add integration
test for AnnotateVideoWithContext transform
kamilwu opened a new pull request #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-594444394
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-595104987
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] EDjur commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
EDjur commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-594618961
LGTM, cheers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-594443357
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-594559807
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-592416998
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-594505339
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-595142300
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-593289095
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] EDjur commented on a change in pull request #10986: Add
integration test for AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
EDjur commented on a change in pull request #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#discussion_r387757358
##########
File path: sdks/python/apache_beam/ml/gcp/videointelligenceml_test_it.py
##########
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+"""An integration test that labels entities appearing in a video and checks
+if some expected entities were properly recognized."""
+
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import unittest
+
+import hamcrest as hc
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import matches_all
+
+# Protect against environments where Google Cloud VideoIntelligence client is
+# not available.
+try:
+ from apache_beam.ml.gcp.videointelligenceml import AnnotateVideoWithContext
+ from google.cloud.videointelligence import enums
+ from google.cloud.videointelligence import types
+except ImportError:
+ AnnotateVideoWithContext = None
+
+
+def extract_entities_descriptions(response):
+ for result in response.annotation_results:
+ for segment in result.segment_label_annotations:
+ yield segment.entity.description
+
+
+@attr('IT')
+@unittest.skipIf(
+ AnnotateVideoWithContext is None, 'GCP dependencies are not installed')
+class VideoIntelligenceMlTestIT(unittest.TestCase):
+ VIDEO_PATH = 'gs://cloudmleap/video/next/gbikes_dinosaur.mp4'
Review comment:
Short pondering: Is it over-protective to have a separate bucket in e.g. Beam infra project in case Google would pull / update this file?
PR looks great btw. Just wanted to put that thought out there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-594607324
R: @EDjur Could you take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-595114482
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on a change in pull request #10986: Add
integration test for AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#discussion_r387774576
##########
File path: sdks/python/apache_beam/ml/gcp/videointelligenceml_test_it.py
##########
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+"""An integration test that labels entities appearing in a video and checks
+if some expected entities were properly recognized."""
+
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import unittest
+
+import hamcrest as hc
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import matches_all
+
+# Protect against environments where Google Cloud VideoIntelligence client is
+# not available.
+try:
+ from apache_beam.ml.gcp.videointelligenceml import AnnotateVideoWithContext
+ from google.cloud.videointelligence import enums
+ from google.cloud.videointelligence import types
+except ImportError:
+ AnnotateVideoWithContext = None
+
+
+def extract_entities_descriptions(response):
+ for result in response.annotation_results:
+ for segment in result.segment_label_annotations:
+ yield segment.entity.description
+
+
+@attr('IT')
+@unittest.skipIf(
+ AnnotateVideoWithContext is None, 'GCP dependencies are not installed')
+class VideoIntelligenceMlTestIT(unittest.TestCase):
+ VIDEO_PATH = 'gs://cloudmleap/video/next/gbikes_dinosaur.mp4'
Review comment:
Good point. I don't have enough permissions to create new files in Beam infra project, but I'll ask someone.
Until then I'll refrain from merging the PR.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu merged pull request #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu merged pull request #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-595106128
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-592487598
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #10986: Add integration test for
AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#issuecomment-592415309
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] EDjur commented on a change in pull request #10986: Add
integration test for AnnotateVideoWithContext transform
Posted by GitBox <gi...@apache.org>.
EDjur commented on a change in pull request #10986: Add integration test for AnnotateVideoWithContext transform
URL: https://github.com/apache/beam/pull/10986#discussion_r387790847
##########
File path: sdks/python/apache_beam/ml/gcp/videointelligenceml_test_it.py
##########
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+"""An integration test that labels entities appearing in a video and checks
+if some expected entities were properly recognized."""
+
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import unittest
+
+import hamcrest as hc
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import matches_all
+
+# Protect against environments where Google Cloud VideoIntelligence client is
+# not available.
+try:
+ from apache_beam.ml.gcp.videointelligenceml import AnnotateVideoWithContext
+ from google.cloud.videointelligence import enums
+ from google.cloud.videointelligence import types
+except ImportError:
+ AnnotateVideoWithContext = None
+
+
+def extract_entities_descriptions(response):
+ for result in response.annotation_results:
+ for segment in result.segment_label_annotations:
+ yield segment.entity.description
+
+
+@attr('IT')
+@unittest.skipIf(
+ AnnotateVideoWithContext is None, 'GCP dependencies are not installed')
+class VideoIntelligenceMlTestIT(unittest.TestCase):
+ VIDEO_PATH = 'gs://cloudmleap/video/next/gbikes_dinosaur.mp4'
Review comment:
Thank you!
The same line of thinking would apply for the vision API sample too then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services