You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2017/02/09 21:50:18 UTC
Build failed in Jenkins: beam_PostCommit_Python_Verify #1223
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/1223/changes>
Changes:
[tgroh] Explicitly pass Pipeline in AppliedPTransform
------------------------------------------
[...truncated 2626 lines...]
--project=$PROJECT \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--sdk_location=$SDK_LOCATION \
--job_name=$JOBNAME_VR_TEST \
--requirements_file=postcommit_requirements.txt \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/setuptools/dist.py>:333: UserWarning: Normalizing '0.6.0.dev' to '0.6.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam_sdk.egg-info/requires.txt
writing apache_beam_sdk.egg-info/PKG-INFO
writing top-level names to apache_beam_sdk.egg-info/top_level.txt
writing dependency_links to apache_beam_sdk.egg-info/dependency_links.txt
writing entry points to apache_beam_sdk.egg-info/entry_points.txt
reading manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
Searching for pyhamcrest<2.0,>=1.9
Best match: PyHamcrest 1.9.0
Processing PyHamcrest-1.9.0-py2.7.egg
Using <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/.eggs/PyHamcrest-1.9.0-py2.7.egg>
test_empty_side_outputs (apache_beam.transforms.ptransform_test.PTransformTest) ... <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136: UserWarning: Using fallback coder for typehint: Union[SideOutputValue, int].
warnings.warn('Using fallback coder for typehint: %r.' % typehint)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136: UserWarning: Using fallback coder for typehint: Any.
warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
SNIMissingWarning
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
File was already downloaded /tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging appdirs pyparsing
ok
test_par_do_with_multiple_outputs_and_using_return (apache_beam.transforms.ptransform_test.PTransformTest) ... DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging appdirs pyparsing
ok
test_par_do_with_multiple_outputs_and_using_yield (apache_beam.transforms.ptransform_test.PTransformTest) ... DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
SNIMissingWarning
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging appdirs pyparsing
ok
test_undeclared_side_outputs (apache_beam.transforms.ptransform_test.PTransformTest) ... DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
SNIMissingWarning
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
File was already downloaded /tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging appdirs pyparsing
ok
test_as_dict_with_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_and_as_dict_side_inputs (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_with_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_without_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_singleton_with_different_defaults_with_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_singleton_with_different_defaults_without_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/sideinputs_test.py>:241: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
e.exception.message.startswith(
ok
test_as_singleton_without_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_default_value_singleton_side_input (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_empty_singleton_side_input (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_iterable_side_input (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
----------------------------------------------------------------------
Ran 14 tests in 1120.590s
OK
# Run wordcount on the Google Cloud Dataflow service
# and validate job that finishes successfully.
echo ">>> RUNNING TEST DATAFLOW RUNNER py-wordcount"
>>> RUNNING TEST DATAFLOW RUNNER py-wordcount
python setup.py nosetests \
-a IT --test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--staging_location=$GCS_LOCATION/staging-wordcount \
--temp_location=$GCS_LOCATION/temp-wordcount \
--output=$GCS_LOCATION/py-wordcount-cloud/output \
--sdk_location=$SDK_LOCATION \
--job_name=$JOBNAME_E2E_WC \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/setuptools/dist.py>:333: UserWarning: Normalizing '0.6.0.dev' to '0.6.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam_sdk.egg-info/requires.txt
writing apache_beam_sdk.egg-info/PKG-INFO
writing top-level names to apache_beam_sdk.egg-info/top_level.txt
writing dependency_links to apache_beam_sdk.egg-info/dependency_links.txt
writing entry points to apache_beam_sdk.egg-info/entry_points.txt
reading manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
Searching for pyhamcrest<2.0,>=1.9
Best match: PyHamcrest 1.9.0
Processing PyHamcrest-1.9.0-py2.7.egg
Using <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/.eggs/PyHamcrest-1.9.0-py2.7.egg>
test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/io/gcsio.py>:109: DeprecationWarning: object() takes no parameters
super(GcsIO, cls).__new__(cls, storage_client))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136: UserWarning: Using fallback coder for typehint: Any.
warnings.warn('Using fallback coder for typehint: %r.' % typehint)
ERROR
======================================================================
ERROR: test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT)
----------------------------------------------------------------------
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount_it_test.py",> line 54, in test_wordcount_it
wordcount.run(test_pipeline.get_full_options_as_args(**extra_opts))
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount.py",> line 101, in run
result = p.run()
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 163, in run
return self.runner.run(self)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/test/test_dataflow_runner.py",> line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py",> line 167, in run
super(DataflowRunner, self).run(pipeline)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/runner.py",> line 113, in run
pipeline.visit(RunVisitor(self))
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 186, in visit
self._root_transform().visit(visitor, self, visited)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
part.visit(visitor, pipeline, visited)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
part.visit(visitor, pipeline, visited)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
part.visit(visitor, pipeline, visited)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
part.visit(visitor, pipeline, visited)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 426, in visit
visitor.visit_transform(self)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/runner.py",> line 108, in visit_transform
self.runner.run_transform(transform_node)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/runner.py",> line 183, in run_transform
return m(transform_node)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py",> line 395, in run_ParDo
transform_node.transform.side_output_tags)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py",> line 267, in _add_step
DisplayData.create_from(transform_node.transform).items])
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 152, in create_from
return cls(has_display_data._namespace(), has_display_data.display_data())
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 83, in __init__
self._populate_items(display_data_dict)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 91, in _populate_items
element.display_data())
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 83, in __init__
self._populate_items(display_data_dict)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 91, in _populate_items
element.display_data())
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/io/textio.py",> line 321, in display_data
self.append_trailing_newlines,
AttributeError: '_TextSink' object has no attribute 'append_trailing_newlines'
-------------------- >> begin captured logging << --------------------
oauth2client.contrib.multistore_file: WARNING: The oauth2client.contrib.multistore_file module has been deprecated and will be removed in the next release of oauth2client. Please migrate to multiprocess_file_storage.
root: WARNING: snappy is not installed; some tests will be skipped.
root: WARNING: Tensorflow is not installed, so skipping some tests.
root: DEBUG: PValue computed by read/Read (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by split (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by pair_with_one (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by group (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by count (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by write/Write/WriteImpl/DoOnce (tag 1): refcount: 2 => 1
root: DEBUG: PValue computed by write/Write/WriteImpl/InitializeWrite (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by format (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by write/Write/WriteImpl/ViewAsSingleton(write|Write|WriteImpl|InitializeWrite.None)/CreatePCollectionView (tag 1): refcount: 2 => 1
root: ERROR: Error while visiting write/Write/WriteImpl/WriteBundles
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
Ran 1 test in 2.375s
FAILED (errors=1)
Build step 'Execute shell' marked build as failure
Jenkins build is back to normal : beam_PostCommit_Python_Verify
#1225
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/1225/changes>
Re: Build failed in Jenkins: beam_PostCommit_Python_Verify #1224
Posted by Ahmet Altay <al...@google.com.INVALID>.
Opened the JIRA issue: https://issues.apache.org/jira/browse/BEAM-1454
This is caused by: https://github.com/apache/beam/pull/1918. At first look
the change failed to rename one instance of
`self.append_trailing_newlines`. We can fix that, if that fails we should
consider rolling back the change.
Ahmet
On Thu, Feb 9, 2017 at 4:12 PM, Apache Jenkins Server <
jenkins@builds.apache.org> wrote:
> See <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/1224/changes>
>
> Changes:
>
> [klk] Add proto definition for Runner API
>
> [robertwb] [BEAM-1450] Fix NewDoFn handling of window explosion.
>
> [altay] Get nosetests working after removing the setup.py change
>
> ------------------------------------------
> [...truncated 2631 lines...]
> --project=$PROJECT \
> --staging_location=$GCS_LOCATION/staging-validatesrunner-test \
> --temp_location=$GCS_LOCATION/temp-validatesrunner-test \
> --sdk_location=$SDK_LOCATION \
> --job_name=$JOBNAME_VR_TEST \
> --requirements_file=postcommit_requirements.txt \
> --num_workers=1"
> <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/local/lib/python2.7/site-packages/setuptools/dist.py>:333:
> UserWarning: Normalizing '0.6.0.dev' to '0.6.0.dev0'
> normalized_version,
> running nosetests
> running egg_info
> writing requirements to apache_beam_sdk.egg-info/requires.txt
> writing apache_beam_sdk.egg-info/PKG-INFO
> writing top-level names to apache_beam_sdk.egg-info/top_level.txt
> writing dependency_links to apache_beam_sdk.egg-info/dependency_links.txt
> writing entry points to apache_beam_sdk.egg-info/entry_points.txt
> reading manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
> reading manifest template 'MANIFEST.in'
> writing manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
> Searching for pyhamcrest<2.0,>=1.9
> Best match: PyHamcrest 1.9.0
> Processing PyHamcrest-1.9.0-py2.7.egg
>
> Using <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/.eggs/PyHamcrest-1.9.0-py2.7.egg>
> test_empty_side_outputs (apache_beam.transforms.ptransform_test.PTransformTest)
> ... <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136: UserWarning:
> Using fallback coder for typehint: Union[SideOutputValue, int].
> warnings.warn('Using fallback coder for typehint: %r.' % typehint)
> <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136: UserWarning:
> Using fallback coder for typehint: Any.
> warnings.warn('Using fallback coder for typehint: %r.' % typehint)
> DEPRECATION: pip install --download has been deprecated and will be
> removed in the future. Pip now has a download command that should be used
> instead.
> Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
> <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/local/lib/python2.7/site-packages/
> pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318:
> SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject
> Name Indication) extension to TLS is not available on this platform. This
> may cause the server to present an incorrect TLS certificate, which can
> cause validation failures. You can upgrade to a newer version of Python to
> solve this. For more information, see https://urllib3.readthedocs.
> io/en/latest/security.html#snimissingwarning.
> SNIMissingWarning
> <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/local/lib/python2.7/site-packages/
> pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122:
> InsecurePlatformWarning: A true SSLContext object is not available. This
> prevents urllib3 from configuring SSL appropriately and may cause certain
> SSL connections to fail. You can upgrade to a newer version of Python to
> solve this. For more information, see https://urllib3.readthedocs.
> io/en/latest/security.html#insecureplatformwarning.
> InsecurePlatformWarning
> File was already downloaded /tmp/dataflow-requirements-
> cache/PyHamcrest-1.9.0.tar.gz
> Collecting mock (from -r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/mock-2.0.0.tar.gz
> Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt
> (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/setuptools-34.1.1.zip
> Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/six-1.10.0.tar.gz
> Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/funcsigs-1.0.2.tar.gz
> Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/pbr-1.10.0.tar.gz
> Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/packaging-16.8.tar.gz
> Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/appdirs-1.4.0.tar.gz
> Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/pyparsing-2.1.10.zip
> Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
> packaging appdirs pyparsing
> ok
> test_par_do_with_multiple_outputs_and_using_return
> (apache_beam.transforms.ptransform_test.PTransformTest) ... DEPRECATION:
> pip install --download has been deprecated and will be removed in the
> future. Pip now has a download command that should be used instead.
> Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/PyHamcrest-1.9.0.tar.gz
> Collecting mock (from -r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/mock-2.0.0.tar.gz
> Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt
> (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/setuptools-34.1.1.zip
> Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/six-1.10.0.tar.gz
> Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/funcsigs-1.0.2.tar.gz
> Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/pbr-1.10.0.tar.gz
> Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/packaging-16.8.tar.gz
> Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/appdirs-1.4.0.tar.gz
> Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/pyparsing-2.1.10.zip
> Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
> packaging appdirs pyparsing
> ok
> test_par_do_with_multiple_outputs_and_using_yield (apache_beam.transforms.
> ptransform_test.PTransformTest) ... DEPRECATION: pip install --download
> has been deprecated and will be removed in the future. Pip now has a
> download command that should be used instead.
> Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/PyHamcrest-1.9.0.tar.gz
> Collecting mock (from -r postcommit_requirements.txt (line 2))
> <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/local/lib/python2.7/site-packages/
> pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318:
> SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject
> Name Indication) extension to TLS is not available on this platform. This
> may cause the server to present an incorrect TLS certificate, which can
> cause validation failures. You can upgrade to a newer version of Python to
> solve this. For more information, see https://urllib3.readthedocs.
> io/en/latest/security.html#snimissingwarning.
> SNIMissingWarning
> <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/local/lib/python2.7/site-packages/
> pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122:
> InsecurePlatformWarning: A true SSLContext object is not available. This
> prevents urllib3 from configuring SSL appropriately and may cause certain
> SSL connections to fail. You can upgrade to a newer version of Python to
> solve this. For more information, see https://urllib3.readthedocs.
> io/en/latest/security.html#insecureplatformwarning.
> InsecurePlatformWarning
> File was already downloaded /tmp/dataflow-requirements-
> cache/mock-2.0.0.tar.gz
> Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt
> (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/setuptools-34.1.1.zip
> Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/six-1.10.0.tar.gz
> Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/funcsigs-1.0.2.tar.gz
> Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/pbr-1.10.0.tar.gz
> Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/packaging-16.8.tar.gz
> Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/appdirs-1.4.0.tar.gz
> Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/pyparsing-2.1.10.zip
> Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
> packaging appdirs pyparsing
> ok
> test_undeclared_side_outputs (apache_beam.transforms.ptransform_test.PTransformTest)
> ... DEPRECATION: pip install --download has been deprecated and will be
> removed in the future. Pip now has a download command that should be used
> instead.
> Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
> <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/local/lib/python2.7/site-packages/
> pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318:
> SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject
> Name Indication) extension to TLS is not available on this platform. This
> may cause the server to present an incorrect TLS certificate, which can
> cause validation failures. You can upgrade to a newer version of Python to
> solve this. For more information, see https://urllib3.readthedocs.
> io/en/latest/security.html#snimissingwarning.
> SNIMissingWarning
> <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/local/lib/python2.7/site-packages/
> pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122:
> InsecurePlatformWarning: A true SSLContext object is not available. This
> prevents urllib3 from configuring SSL appropriately and may cause certain
> SSL connections to fail. You can upgrade to a newer version of Python to
> solve this. For more information, see https://urllib3.readthedocs.
> io/en/latest/security.html#insecureplatformwarning.
> InsecurePlatformWarning
> File was already downloaded /tmp/dataflow-requirements-
> cache/PyHamcrest-1.9.0.tar.gz
> Collecting mock (from -r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/mock-2.0.0.tar.gz
> Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt
> (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/setuptools-34.1.1.zip
> Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/six-1.10.0.tar.gz
> Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/funcsigs-1.0.2.tar.gz
> Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
> File was already downloaded /tmp/dataflow-requirements-
> cache/pbr-1.10.0.tar.gz
> Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/packaging-16.8.tar.gz
> Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/appdirs-1.4.0.tar.gz
> Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r
> postcommit_requirements.txt (line 1))
> File was already downloaded /tmp/dataflow-requirements-
> cache/pyparsing-2.1.10.zip
> Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
> packaging appdirs pyparsing
> ok
> test_as_dict_with_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest)
> ... ok
> test_as_list_and_as_dict_side_inputs (apache_beam.transforms.
> sideinputs_test.SideInputsTest) ... ok
> test_as_list_with_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest)
> ... ok
> test_as_list_without_unique_labels (apache_beam.transforms.
> sideinputs_test.SideInputsTest) ... ok
> test_as_singleton_with_different_defaults_with_unique_labels
> (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
> test_as_singleton_with_different_defaults_without_unique_labels
> (apache_beam.transforms.sideinputs_test.SideInputsTest) ... <
> https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/transforms/sideinputs_test.py>:241:
> DeprecationWarning: BaseException.message has been deprecated as of Python
> 2.6
> e.exception.message.startswith(
> ok
> test_as_singleton_without_unique_labels (apache_beam.transforms.
> sideinputs_test.SideInputsTest) ... ok
> test_default_value_singleton_side_input (apache_beam.transforms.
> sideinputs_test.SideInputsTest) ... ok
> test_empty_singleton_side_input (apache_beam.transforms.sideinputs_test.SideInputsTest)
> ... ok
> test_iterable_side_input (apache_beam.transforms.sideinputs_test.SideInputsTest)
> ... ok
>
> ----------------------------------------------------------------------
> Ran 14 tests in 1149.537s
>
> OK
>
> # Run wordcount on the Google Cloud Dataflow service
> # and validate job that finishes successfully.
> echo ">>> RUNNING TEST DATAFLOW RUNNER py-wordcount"
> >>> RUNNING TEST DATAFLOW RUNNER py-wordcount
> python setup.py nosetests \
> -a IT --test-pipeline-options=" \
> --runner=TestDataflowRunner \
> --project=$PROJECT \
> --staging_location=$GCS_LOCATION/staging-wordcount \
> --temp_location=$GCS_LOCATION/temp-wordcount \
> --output=$GCS_LOCATION/py-wordcount-cloud/output \
> --sdk_location=$SDK_LOCATION \
> --job_name=$JOBNAME_E2E_WC \
> --num_workers=1"
> <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/local/lib/python2.7/site-packages/setuptools/dist.py>:333:
> UserWarning: Normalizing '0.6.0.dev' to '0.6.0.dev0'
> normalized_version,
> running nosetests
> running egg_info
> writing requirements to apache_beam_sdk.egg-info/requires.txt
> writing apache_beam_sdk.egg-info/PKG-INFO
> writing top-level names to apache_beam_sdk.egg-info/top_level.txt
> writing dependency_links to apache_beam_sdk.egg-info/dependency_links.txt
> writing entry points to apache_beam_sdk.egg-info/entry_points.txt
> reading manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
> reading manifest template 'MANIFEST.in'
> writing manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
> Searching for pyhamcrest<2.0,>=1.9
> Best match: PyHamcrest 1.9.0
> Processing PyHamcrest-1.9.0-py2.7.egg
>
> Using <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/.eggs/PyHamcrest-1.9.0-py2.7.egg>
> test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT)
> ... <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/io/gcsio.py>:109: DeprecationWarning:
> object() takes no parameters
> super(GcsIO, cls).__new__(cls, storage_client))
> <https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136: UserWarning:
> Using fallback coder for typehint: Any.
> warnings.warn('Using fallback coder for typehint: %r.' % typehint)
> ERROR
>
> ======================================================================
> ERROR: test_wordcount_it (apache_beam.examples.
> wordcount_it_test.WordCountIT)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/examples/wordcount_it_test.py",> line
> 54, in test_wordcount_it
> wordcount.run(test_pipeline.get_full_options_as_args(**extra_opts))
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/examples/wordcount.py",> line 101, in
> run
> result = p.run()
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/pipeline.py",> line 163, in run
> return self.runner.run(self)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/runners/test/test_dataflow_runner.py",>
> line 32, in run
> self.result = super(TestDataflowRunner, self).run(pipeline)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py",> line 167,
> in run
> super(DataflowRunner, self).run(pipeline)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/runners/runner.py",> line 113, in run
> pipeline.visit(RunVisitor(self))
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/pipeline.py",> line 186, in visit
> self._root_transform().visit(visitor, self, visited)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
> part.visit(visitor, pipeline, visited)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
> part.visit(visitor, pipeline, visited)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
> part.visit(visitor, pipeline, visited)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
> part.visit(visitor, pipeline, visited)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/pipeline.py",> line 426, in visit
> visitor.visit_transform(self)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/runners/runner.py",> line 108, in
> visit_transform
> self.runner.run_transform(transform_node)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/runners/runner.py",> line 183, in
> run_transform
> return m(transform_node)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py",> line 395,
> in run_ParDo
> transform_node.transform.side_output_tags)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py",> line 267,
> in _add_step
> DisplayData.create_from(transform_node.transform).items])
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 152, in
> create_from
> return cls(has_display_data._namespace(),
> has_display_data.display_data())
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 83, in
> __init__
> self._populate_items(display_data_dict)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 91, in
> _populate_items
> element.display_data())
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 83, in
> __init__
> self._populate_items(display_data_dict)
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 91, in
> _populate_items
> element.display_data())
> File "<https://builds.apache.org/job/beam_PostCommit_Python_
> Verify/ws/sdks/python/apache_beam/io/textio.py",> line 321, in
> display_data
> self.append_trailing_newlines,
> AttributeError: '_TextSink' object has no attribute
> 'append_trailing_newlines'
> -------------------- >> begin captured logging << --------------------
> oauth2client.contrib.multistore_file: WARNING: The oauth2client.contrib.multistore_file
> module has been deprecated and will be removed in the next release of
> oauth2client. Please migrate to multiprocess_file_storage.
> root: WARNING: snappy is not installed; some tests will be skipped.
> root: WARNING: Tensorflow is not installed, so skipping some tests.
> root: DEBUG: PValue computed by read/Read (tag 1): refcount: 1 => 0
> root: DEBUG: PValue computed by split (tag 1): refcount: 1 => 0
> root: DEBUG: PValue computed by pair_with_one (tag 1): refcount: 1 => 0
> root: DEBUG: PValue computed by group (tag 1): refcount: 1 => 0
> root: DEBUG: PValue computed by count (tag 1): refcount: 1 => 0
> root: DEBUG: PValue computed by write/Write/WriteImpl/DoOnce (tag 1):
> refcount: 2 => 1
> root: DEBUG: PValue computed by write/Write/WriteImpl/InitializeWrite
> (tag 1): refcount: 1 => 0
> root: DEBUG: PValue computed by format (tag 1): refcount: 1 => 0
> root: DEBUG: PValue computed by write/Write/WriteImpl/
> ViewAsSingleton(write|Write|WriteImpl|InitializeWrite.None)/CreatePCollectionView
> (tag 1): refcount: 2 => 1
> root: ERROR: Error while visiting write/Write/WriteImpl/WriteBundles
> --------------------- >> end captured logging << ---------------------
>
> ----------------------------------------------------------------------
> Ran 1 test in 7.356s
>
> FAILED (errors=1)
> Build step 'Execute shell' marked build as failure
>
Build failed in Jenkins: beam_PostCommit_Python_Verify #1224
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/1224/changes>
Changes:
[klk] Add proto definition for Runner API
[robertwb] [BEAM-1450] Fix NewDoFn handling of window explosion.
[altay] Get nosetests working after removing the setup.py change
------------------------------------------
[...truncated 2631 lines...]
--project=$PROJECT \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--sdk_location=$SDK_LOCATION \
--job_name=$JOBNAME_VR_TEST \
--requirements_file=postcommit_requirements.txt \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/setuptools/dist.py>:333: UserWarning: Normalizing '0.6.0.dev' to '0.6.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam_sdk.egg-info/requires.txt
writing apache_beam_sdk.egg-info/PKG-INFO
writing top-level names to apache_beam_sdk.egg-info/top_level.txt
writing dependency_links to apache_beam_sdk.egg-info/dependency_links.txt
writing entry points to apache_beam_sdk.egg-info/entry_points.txt
reading manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
Searching for pyhamcrest<2.0,>=1.9
Best match: PyHamcrest 1.9.0
Processing PyHamcrest-1.9.0-py2.7.egg
Using <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/.eggs/PyHamcrest-1.9.0-py2.7.egg>
test_empty_side_outputs (apache_beam.transforms.ptransform_test.PTransformTest) ... <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136: UserWarning: Using fallback coder for typehint: Union[SideOutputValue, int].
warnings.warn('Using fallback coder for typehint: %r.' % typehint)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136: UserWarning: Using fallback coder for typehint: Any.
warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
SNIMissingWarning
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
File was already downloaded /tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging appdirs pyparsing
ok
test_par_do_with_multiple_outputs_and_using_return (apache_beam.transforms.ptransform_test.PTransformTest) ... DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging appdirs pyparsing
ok
test_par_do_with_multiple_outputs_and_using_yield (apache_beam.transforms.ptransform_test.PTransformTest) ... DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
SNIMissingWarning
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging appdirs pyparsing
ok
test_undeclared_side_outputs (apache_beam.transforms.ptransform_test.PTransformTest) ... DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
SNIMissingWarning
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
File was already downloaded /tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging appdirs pyparsing
ok
test_as_dict_with_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_and_as_dict_side_inputs (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_with_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_without_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_singleton_with_different_defaults_with_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_singleton_with_different_defaults_without_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/sideinputs_test.py>:241: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
e.exception.message.startswith(
ok
test_as_singleton_without_unique_labels (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_default_value_singleton_side_input (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_empty_singleton_side_input (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_iterable_side_input (apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
----------------------------------------------------------------------
Ran 14 tests in 1149.537s
OK
# Run wordcount on the Google Cloud Dataflow service
# and validate job that finishes successfully.
echo ">>> RUNNING TEST DATAFLOW RUNNER py-wordcount"
>>> RUNNING TEST DATAFLOW RUNNER py-wordcount
python setup.py nosetests \
-a IT --test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--staging_location=$GCS_LOCATION/staging-wordcount \
--temp_location=$GCS_LOCATION/temp-wordcount \
--output=$GCS_LOCATION/py-wordcount-cloud/output \
--sdk_location=$SDK_LOCATION \
--job_name=$JOBNAME_E2E_WC \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/setuptools/dist.py>:333: UserWarning: Normalizing '0.6.0.dev' to '0.6.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam_sdk.egg-info/requires.txt
writing apache_beam_sdk.egg-info/PKG-INFO
writing top-level names to apache_beam_sdk.egg-info/top_level.txt
writing dependency_links to apache_beam_sdk.egg-info/dependency_links.txt
writing entry points to apache_beam_sdk.egg-info/entry_points.txt
reading manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
Searching for pyhamcrest<2.0,>=1.9
Best match: PyHamcrest 1.9.0
Processing PyHamcrest-1.9.0-py2.7.egg
Using <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/.eggs/PyHamcrest-1.9.0-py2.7.egg>
test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/io/gcsio.py>:109: DeprecationWarning: object() takes no parameters
super(GcsIO, cls).__new__(cls, storage_client))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136: UserWarning: Using fallback coder for typehint: Any.
warnings.warn('Using fallback coder for typehint: %r.' % typehint)
ERROR
======================================================================
ERROR: test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT)
----------------------------------------------------------------------
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount_it_test.py",> line 54, in test_wordcount_it
wordcount.run(test_pipeline.get_full_options_as_args(**extra_opts))
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount.py",> line 101, in run
result = p.run()
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 163, in run
return self.runner.run(self)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/test/test_dataflow_runner.py",> line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py",> line 167, in run
super(DataflowRunner, self).run(pipeline)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/runner.py",> line 113, in run
pipeline.visit(RunVisitor(self))
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 186, in visit
self._root_transform().visit(visitor, self, visited)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
part.visit(visitor, pipeline, visited)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
part.visit(visitor, pipeline, visited)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
part.visit(visitor, pipeline, visited)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 423, in visit
part.visit(visitor, pipeline, visited)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py",> line 426, in visit
visitor.visit_transform(self)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/runner.py",> line 108, in visit_transform
self.runner.run_transform(transform_node)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/runner.py",> line 183, in run_transform
return m(transform_node)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py",> line 395, in run_ParDo
transform_node.transform.side_output_tags)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py",> line 267, in _add_step
DisplayData.create_from(transform_node.transform).items])
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 152, in create_from
return cls(has_display_data._namespace(), has_display_data.display_data())
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 83, in __init__
self._populate_items(display_data_dict)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 91, in _populate_items
element.display_data())
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 83, in __init__
self._populate_items(display_data_dict)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py",> line 91, in _populate_items
element.display_data())
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/io/textio.py",> line 321, in display_data
self.append_trailing_newlines,
AttributeError: '_TextSink' object has no attribute 'append_trailing_newlines'
-------------------- >> begin captured logging << --------------------
oauth2client.contrib.multistore_file: WARNING: The oauth2client.contrib.multistore_file module has been deprecated and will be removed in the next release of oauth2client. Please migrate to multiprocess_file_storage.
root: WARNING: snappy is not installed; some tests will be skipped.
root: WARNING: Tensorflow is not installed, so skipping some tests.
root: DEBUG: PValue computed by read/Read (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by split (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by pair_with_one (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by group (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by count (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by write/Write/WriteImpl/DoOnce (tag 1): refcount: 2 => 1
root: DEBUG: PValue computed by write/Write/WriteImpl/InitializeWrite (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by format (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by write/Write/WriteImpl/ViewAsSingleton(write|Write|WriteImpl|InitializeWrite.None)/CreatePCollectionView (tag 1): refcount: 2 => 1
root: ERROR: Error while visiting write/Write/WriteImpl/WriteBundles
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
Ran 1 test in 7.356s
FAILED (errors=1)
Build step 'Execute shell' marked build as failure