You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2017/03/15 20:52:12 UTC
Build failed in Jenkins: beam_PostCommit_Python_Verify #1516
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/1516/display/redirect?page=changes>
Changes:
[altay] Add ValueProvider class for FileBasedSource I/O Transforms
[davor] Increment shade-plugin version back to 3.0.0
------------------------------------------
[...truncated 338.75 KB...]
test_static_value_provider_type_cast (apache_beam.utils.value_provider_test.ValueProviderTests) ... ok
test_equality (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_hash (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_pickle (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_timestamps (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_with_value (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_context_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_element (apache_beam.pipeline_test.DoFnTest) ... ok
test_side_input_no_tag (apache_beam.pipeline_test.DoFnTest) ... ok
test_side_input_tagged (apache_beam.pipeline_test.DoFnTest) ... ok
test_timestamp_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_window_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_attribute_setting (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_defaults (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_dir (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_flag_parsing (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_keyword_parsing (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_view_as (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_aggregator_empty_input (apache_beam.pipeline_test.PipelineTest) ... ok
test_apply_custom_transform (apache_beam.pipeline_test.PipelineTest) ... ok
test_create (apache_beam.pipeline_test.PipelineTest) ... ok
test_create_singleton_pcollection (apache_beam.pipeline_test.PipelineTest) ... ok
test_eager_pipeline (apache_beam.pipeline_test.PipelineTest) ... ok
test_flatmap_builtin (apache_beam.pipeline_test.PipelineTest) ... ok
test_memory_usage (apache_beam.pipeline_test.PipelineTest) ... ok
test_metrics_in_source (apache_beam.pipeline_test.PipelineTest) ... ok
test_pipeline_as_context (apache_beam.pipeline_test.PipelineTest) ... ok
test_read (apache_beam.pipeline_test.PipelineTest) ... ok
test_reuse_cloned_custom_transform_instance (apache_beam.pipeline_test.PipelineTest) ... ok
test_reuse_custom_transform_instance (apache_beam.pipeline_test.PipelineTest) ... <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline_test.py>:195: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
cm.exception.message,
ok
test_transform_no_super_init (apache_beam.pipeline_test.PipelineTest) ... ok
test_visit_entire_graph (apache_beam.pipeline_test.PipelineTest) ... ok
test_pcollectionview_not_recreated (apache_beam.pvalue_test.PValueTest) ... ok
test_pvalue_expected_arguments (apache_beam.pvalue_test.PValueTest) ... ok
test_append_extra_options (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_append_verifier_in_extra_opt (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_create_test_pipeline_options (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_empty_option_args_parsing (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_get_option (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_option_args_parsing (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_skip_IT (apache_beam.test_pipeline_test.TestPipelineTest) ... SKIP: IT is skipped because --test-pipeline-options is not specified
test_file_checksum_matcher_read_failed (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_file_checksum_matcher_service_error (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_file_checksum_matcher_success (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_pipeline_state_matcher_fails (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher fails when using default expected state ... ok
test_pipeline_state_matcher_given_state (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher successes when matches given state ... ok
test_pipeline_state_matcher_success (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher successes when using default expected state ... ok
test_version (apache_beam.version_test.Version) ... ok
----------------------------------------------------------------------
Ran 1025 tests in 44.469s
OK (skipped=13)
___________________________________ summary ____________________________________
docs: commands succeeded
lint: commands succeeded
py27: commands succeeded
py27cython: commands succeeded
py27gcp: commands succeeded
congratulations :)
# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
New python executable in <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/bin/python>
Installing setuptools, pip, wheel...done.
. sdks/python/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly
deactivate () {
unset -f pydoc >/dev/null 2>&1
# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi
unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}
# unset irrelevant variables
deactivate nondestructive
VIRTUAL_ENV="<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python">
export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi
if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
basename "$VIRTUAL_ENV"
# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc
pydoc () {
python -m pydoc "$@"
}
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
cd sdks/python
pip install -e .[gcp,test]
Obtaining file://<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python>
Collecting avro<2.0.0,>=1.7.7 (from apache-beam==0.7.0.dev0)
Collecting crcmod<2.0,>=1.7 (from apache-beam==0.7.0.dev0)
Collecting dill==0.2.6 (from apache-beam==0.7.0.dev0)
Collecting httplib2<0.10,>=0.8 (from apache-beam==0.7.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==0.7.0.dev0)
Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==0.7.0.dev0)
Collecting protobuf==3.2.0 (from apache-beam==0.7.0.dev0)
Using cached protobuf-3.2.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.10 (from apache-beam==0.7.0.dev0)
Collecting google-apitools<1.0.0,>=0.5.6 (from apache-beam==0.7.0.dev0)
Using cached google_apitools-0.5.8-py2-none-any.whl
Collecting proto-google-cloud-datastore-v1==0.90.0 (from apache-beam==0.7.0.dev0)
Collecting googledatastore==7.0.0 (from apache-beam==0.7.0.dev0)
Collecting google-cloud-bigquery<0.23,>=0.22.1 (from apache-beam==0.7.0.dev0)
Using cached google_cloud_bigquery-0.22.1-py2.py3-none-any.whl
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==0.7.0.dev0)
Using cached PyHamcrest-1.9.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.9 in ./lib/python2.7/site-packages (from mock<3.0.0,>=1.0.1->apache-beam==0.7.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from mock<3.0.0,>=1.0.1->apache-beam==0.7.0.dev0)
Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==0.7.0.dev0)
Using cached pbr-2.0.0-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==0.7.0.dev0)
Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from oauth2client<4.0.0,>=2.0.1->apache-beam==0.7.0.dev0)
Using cached pyasn1_modules-0.0.8-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from oauth2client<4.0.0,>=2.0.1->apache-beam==0.7.0.dev0)
Using cached pyasn1-0.2.3-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./lib/python2.7/site-packages (from protobuf==3.2.0->apache-beam==0.7.0.dev0)
Collecting googleapis-common-protos<2.0dev,>=1.5.0 (from proto-google-cloud-datastore-v1==0.90.0->apache-beam==0.7.0.dev0)
Collecting google-cloud-core<0.23dev,>=0.22.1 (from google-cloud-bigquery<0.23,>=0.22.1->apache-beam==0.7.0.dev0)
Using cached google_cloud_core-0.22.1-py2.py3-none-any.whl
Requirement already satisfied: appdirs>=1.4.0 in ./lib/python2.7/site-packages (from setuptools->protobuf==3.2.0->apache-beam==0.7.0.dev0)
Requirement already satisfied: packaging>=16.8 in ./lib/python2.7/site-packages (from setuptools->protobuf==3.2.0->apache-beam==0.7.0.dev0)
Collecting google-auth-httplib2 (from google-cloud-core<0.23dev,>=0.22.1->google-cloud-bigquery<0.23,>=0.22.1->apache-beam==0.7.0.dev0)
Using cached google_auth_httplib2-0.0.2-py2.py3-none-any.whl
Collecting google-auth<2.0.0dev,>=0.4.0 (from google-cloud-core<0.23dev,>=0.22.1->google-cloud-bigquery<0.23,>=0.22.1->apache-beam==0.7.0.dev0)
Using cached google_auth-0.8.0-py2.py3-none-any.whl
Requirement already satisfied: pyparsing in ./lib/python2.7/site-packages (from packaging>=16.8->setuptools->protobuf==3.2.0->apache-beam==0.7.0.dev0)
Installing collected packages: avro, crcmod, dill, httplib2, funcsigs, pbr, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pyyaml, google-apitools, googleapis-common-protos, proto-google-cloud-datastore-v1, googledatastore, google-auth, google-auth-httplib2, google-cloud-core, google-cloud-bigquery, pyhamcrest, apache-beam
Running setup.py develop for apache-beam
Successfully installed apache-beam avro-1.8.1 crcmod-1.7 dill-0.2.6 funcsigs-1.0.2 google-apitools-0.5.8 google-auth-0.8.0 google-auth-httplib2-0.0.2 google-cloud-bigquery-0.22.1 google-cloud-core-0.22.1 googleapis-common-protos-1.5.2 googledatastore-7.0.0 httplib2-0.9.2 mock-2.0.0 oauth2client-3.0.0 pbr-2.0.0 proto-google-cloud-datastore-v1-0.90.0 protobuf-3.2.0 pyasn1-0.2.3 pyasn1-modules-0.0.8 pyhamcrest-1.9.0 pyyaml-3.12 rsa-3.4.2
# Run wordcount in the Direct Runner and validate output.
echo ">>> RUNNING DIRECT RUNNER py-wordcount"
>>> RUNNING DIRECT RUNNER py-wordcount
python -m apache_beam.examples.wordcount --output /tmp/py-wordcount-direct
No handlers could be found for logger "oauth2client.contrib.multistore_file"
INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO:root:Running pipeline with DirectRunner.
WARNING:root:Task failed: Traceback (most recent call last):
File "apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
File "apache_beam/runners/direct/transform_evaluator.py", line 204, in finish_bundle
range_tracker = self._source.get_range_tracker(None, None)
File "apache_beam/io/filebasedsource.py", line 224, in get_range_tracker
return self._get_concat_source().get_range_tracker(start_position,
File "apache_beam/utils/value_provider.py", line 105, in _f
raise RuntimeError('%s not accessible' % obj)
RuntimeError: RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
Traceback (most recent call last):
File "apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
File "apache_beam/runners/direct/transform_evaluator.py", line 204, in finish_bundle
range_tracker = self._source.get_range_tracker(None, None)
File "apache_beam/io/filebasedsource.py", line 224, in get_range_tracker
return self._get_concat_source().get_range_tracker(start_position,
File "apache_beam/utils/value_provider.py", line 105, in _f
raise RuntimeError('%s not accessible' % obj)
RuntimeError: RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
WARNING:root:A task failed with exception.
RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount.py",> line 119, in <module>
run()
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount.py",> line 109, in run
result.wait_until_finish()
File "apache_beam/runners/direct/direct_runner.py", line 157, in wait_until_finish
self._executor.await_completion()
File "apache_beam/runners/direct/executor.py", line 335, in await_completion
self._executor.await_completion()
File "apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
File "apache_beam/runners/direct/transform_evaluator.py", line 204, in finish_bundle
range_tracker = self._source.get_range_tracker(None, None)
File "apache_beam/io/filebasedsource.py", line 224, in get_range_tracker
return self._get_concat_source().get_range_tracker(start_position,
File "apache_beam/utils/value_provider.py", line 105, in _f
raise RuntimeError('%s not accessible' % obj)
RuntimeError: RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
Build step 'Execute shell' marked build as failure
Jenkins build is back to normal : beam_PostCommit_Python_Verify
#1520
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/1520/display/redirect?page=changes>
Build failed in Jenkins: beam_PostCommit_Python_Verify #1519
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/1519/display/redirect?page=changes>
Changes:
[altay] Revert "[BEAM-547] Version should be accessed from pom file"
------------------------------------------
[...truncated 342.34 KB...]
test_static_value_provider_positional_argument (apache_beam.utils.value_provider_test.ValueProviderTests) ... ok
test_static_value_provider_type_cast (apache_beam.utils.value_provider_test.ValueProviderTests) ... ok
test_equality (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_hash (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_pickle (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_timestamps (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_with_value (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_context_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_element (apache_beam.pipeline_test.DoFnTest) ... ok
test_side_input_no_tag (apache_beam.pipeline_test.DoFnTest) ... ok
test_side_input_tagged (apache_beam.pipeline_test.DoFnTest) ... ok
test_timestamp_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_window_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_attribute_setting (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_defaults (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_dir (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_flag_parsing (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_keyword_parsing (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_view_as (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_aggregator_empty_input (apache_beam.pipeline_test.PipelineTest) ... ok
test_apply_custom_transform (apache_beam.pipeline_test.PipelineTest) ... ok
test_create (apache_beam.pipeline_test.PipelineTest) ... ok
test_create_singleton_pcollection (apache_beam.pipeline_test.PipelineTest) ... ok
test_eager_pipeline (apache_beam.pipeline_test.PipelineTest) ... ok
test_flatmap_builtin (apache_beam.pipeline_test.PipelineTest) ... ok
test_memory_usage (apache_beam.pipeline_test.PipelineTest) ... ok
test_metrics_in_source (apache_beam.pipeline_test.PipelineTest) ... ok
test_pipeline_as_context (apache_beam.pipeline_test.PipelineTest) ... ok
test_read (apache_beam.pipeline_test.PipelineTest) ... ok
test_reuse_cloned_custom_transform_instance (apache_beam.pipeline_test.PipelineTest) ... ok
test_reuse_custom_transform_instance (apache_beam.pipeline_test.PipelineTest) ... <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline_test.py>:195: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
cm.exception.message,
ok
test_transform_no_super_init (apache_beam.pipeline_test.PipelineTest) ... ok
test_visit_entire_graph (apache_beam.pipeline_test.PipelineTest) ... ok
test_pcollectionview_not_recreated (apache_beam.pvalue_test.PValueTest) ... ok
test_pvalue_expected_arguments (apache_beam.pvalue_test.PValueTest) ... ok
test_append_extra_options (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_append_verifier_in_extra_opt (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_create_test_pipeline_options (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_empty_option_args_parsing (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_get_option (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_option_args_parsing (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_skip_IT (apache_beam.test_pipeline_test.TestPipelineTest) ... SKIP: IT is skipped because --test-pipeline-options is not specified
test_file_checksum_matcher_read_failed (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_file_checksum_matcher_service_error (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_file_checksum_matcher_success (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_pipeline_state_matcher_fails (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher fails when using default expected state ... ok
test_pipeline_state_matcher_given_state (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher successes when matches given state ... ok
test_pipeline_state_matcher_success (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher successes when using default expected state ... ok
----------------------------------------------------------------------
Ran 1024 tests in 52.289s
OK (skipped=13)
___________________________________ summary ____________________________________
docs: commands succeeded
lint: commands succeeded
py27: commands succeeded
py27cython: commands succeeded
py27gcp: commands succeeded
congratulations :)
# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
New python executable in <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/bin/python>
Installing setuptools, pip, wheel...done.
. sdks/python/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly
deactivate () {
unset -f pydoc >/dev/null 2>&1
# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi
unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}
# unset irrelevant variables
deactivate nondestructive
VIRTUAL_ENV="<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python">
export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi
if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
basename "$VIRTUAL_ENV"
# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc
pydoc () {
python -m pydoc "$@"
}
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
cd sdks/python
pip install -e .[gcp,test]
Obtaining file://<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python>
Collecting avro<2.0.0,>=1.7.7 (from apache-beam==0.7.0.dev0)
Collecting crcmod<2.0,>=1.7 (from apache-beam==0.7.0.dev0)
Collecting dill==0.2.6 (from apache-beam==0.7.0.dev0)
Collecting httplib2<0.10,>=0.8 (from apache-beam==0.7.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==0.7.0.dev0)
Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==0.7.0.dev0)
Collecting protobuf==3.2.0 (from apache-beam==0.7.0.dev0)
Using cached protobuf-3.2.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.10 (from apache-beam==0.7.0.dev0)
Collecting google-apitools<1.0.0,>=0.5.6 (from apache-beam==0.7.0.dev0)
Using cached google_apitools-0.5.8-py2-none-any.whl
Collecting proto-google-cloud-datastore-v1==0.90.0 (from apache-beam==0.7.0.dev0)
Collecting googledatastore==7.0.0 (from apache-beam==0.7.0.dev0)
Collecting google-cloud-bigquery<0.23,>=0.22.1 (from apache-beam==0.7.0.dev0)
Using cached google_cloud_bigquery-0.22.1-py2.py3-none-any.whl
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==0.7.0.dev0)
Using cached PyHamcrest-1.9.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.9 in ./lib/python2.7/site-packages (from mock<3.0.0,>=1.0.1->apache-beam==0.7.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from mock<3.0.0,>=1.0.1->apache-beam==0.7.0.dev0)
Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==0.7.0.dev0)
Using cached pbr-2.0.0-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==0.7.0.dev0)
Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from oauth2client<4.0.0,>=2.0.1->apache-beam==0.7.0.dev0)
Using cached pyasn1_modules-0.0.8-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from oauth2client<4.0.0,>=2.0.1->apache-beam==0.7.0.dev0)
Using cached pyasn1-0.2.3-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./lib/python2.7/site-packages (from protobuf==3.2.0->apache-beam==0.7.0.dev0)
Collecting googleapis-common-protos<2.0dev,>=1.5.0 (from proto-google-cloud-datastore-v1==0.90.0->apache-beam==0.7.0.dev0)
Collecting google-cloud-core<0.23dev,>=0.22.1 (from google-cloud-bigquery<0.23,>=0.22.1->apache-beam==0.7.0.dev0)
Using cached google_cloud_core-0.22.1-py2.py3-none-any.whl
Requirement already satisfied: appdirs>=1.4.0 in ./lib/python2.7/site-packages (from setuptools->protobuf==3.2.0->apache-beam==0.7.0.dev0)
Requirement already satisfied: packaging>=16.8 in ./lib/python2.7/site-packages (from setuptools->protobuf==3.2.0->apache-beam==0.7.0.dev0)
Collecting google-auth-httplib2 (from google-cloud-core<0.23dev,>=0.22.1->google-cloud-bigquery<0.23,>=0.22.1->apache-beam==0.7.0.dev0)
Using cached google_auth_httplib2-0.0.2-py2.py3-none-any.whl
Collecting google-auth<2.0.0dev,>=0.4.0 (from google-cloud-core<0.23dev,>=0.22.1->google-cloud-bigquery<0.23,>=0.22.1->apache-beam==0.7.0.dev0)
Using cached google_auth-0.8.0-py2.py3-none-any.whl
Requirement already satisfied: pyparsing in ./lib/python2.7/site-packages (from packaging>=16.8->setuptools->protobuf==3.2.0->apache-beam==0.7.0.dev0)
Installing collected packages: avro, crcmod, dill, httplib2, funcsigs, pbr, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pyyaml, google-apitools, googleapis-common-protos, proto-google-cloud-datastore-v1, googledatastore, google-auth, google-auth-httplib2, google-cloud-core, google-cloud-bigquery, pyhamcrest, apache-beam
Running setup.py develop for apache-beam
Successfully installed apache-beam avro-1.8.1 crcmod-1.7 dill-0.2.6 funcsigs-1.0.2 google-apitools-0.5.8 google-auth-0.8.0 google-auth-httplib2-0.0.2 google-cloud-bigquery-0.22.1 google-cloud-core-0.22.1 googleapis-common-protos-1.5.2 googledatastore-7.0.0 httplib2-0.9.2 mock-2.0.0 oauth2client-3.0.0 pbr-2.0.0 proto-google-cloud-datastore-v1-0.90.0 protobuf-3.2.0 pyasn1-0.2.3 pyasn1-modules-0.0.8 pyhamcrest-1.9.0 pyyaml-3.12 rsa-3.4.2
# Run wordcount in the Direct Runner and validate output.
echo ">>> RUNNING DIRECT RUNNER py-wordcount"
>>> RUNNING DIRECT RUNNER py-wordcount
python -m apache_beam.examples.wordcount --output /tmp/py-wordcount-direct
No handlers could be found for logger "oauth2client.contrib.multistore_file"
INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO:root:Running pipeline with DirectRunner.
WARNING:root:Task failed: Traceback (most recent call last):
File "apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
File "apache_beam/runners/direct/transform_evaluator.py", line 204, in finish_bundle
range_tracker = self._source.get_range_tracker(None, None)
File "apache_beam/io/filebasedsource.py", line 224, in get_range_tracker
return self._get_concat_source().get_range_tracker(start_position,
File "apache_beam/utils/value_provider.py", line 105, in _f
raise RuntimeError('%s not accessible' % obj)
RuntimeError: RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
Traceback (most recent call last):
File "apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
File "apache_beam/runners/direct/transform_evaluator.py", line 204, in finish_bundle
range_tracker = self._source.get_range_tracker(None, None)
File "apache_beam/io/filebasedsource.py", line 224, in get_range_tracker
return self._get_concat_source().get_range_tracker(start_position,
File "apache_beam/utils/value_provider.py", line 105, in _f
raise RuntimeError('%s not accessible' % obj)
RuntimeError: RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
WARNING:root:A task failed with exception.
RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount.py",> line 119, in <module>
run()
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount.py",> line 109, in run
result.wait_until_finish()
File "apache_beam/runners/direct/direct_runner.py", line 157, in wait_until_finish
self._executor.await_completion()
File "apache_beam/runners/direct/executor.py", line 335, in await_completion
self._executor.await_completion()
File "apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
File "apache_beam/runners/direct/transform_evaluator.py", line 204, in finish_bundle
range_tracker = self._source.get_range_tracker(None, None)
File "apache_beam/io/filebasedsource.py", line 224, in get_range_tracker
return self._get_concat_source().get_range_tracker(start_position,
File "apache_beam/utils/value_provider.py", line 105, in _f
raise RuntimeError('%s not accessible' % obj)
RuntimeError: RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
Build step 'Execute shell' marked build as failure
Build failed in Jenkins: beam_PostCommit_Python_Verify #1518
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/1518/display/redirect>
------------------------------------------
[...truncated 342.57 KB...]
test_static_value_provider_type_cast (apache_beam.utils.value_provider_test.ValueProviderTests) ... ok
test_equality (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_hash (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_pickle (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_timestamps (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_with_value (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_context_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_element (apache_beam.pipeline_test.DoFnTest) ... ok
test_side_input_no_tag (apache_beam.pipeline_test.DoFnTest) ... ok
test_side_input_tagged (apache_beam.pipeline_test.DoFnTest) ... ok
test_timestamp_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_window_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_attribute_setting (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_defaults (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_dir (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_flag_parsing (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_keyword_parsing (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_view_as (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_aggregator_empty_input (apache_beam.pipeline_test.PipelineTest) ... ok
test_apply_custom_transform (apache_beam.pipeline_test.PipelineTest) ... ok
test_create (apache_beam.pipeline_test.PipelineTest) ... ok
test_create_singleton_pcollection (apache_beam.pipeline_test.PipelineTest) ... ok
test_eager_pipeline (apache_beam.pipeline_test.PipelineTest) ... ok
test_flatmap_builtin (apache_beam.pipeline_test.PipelineTest) ... ok
test_memory_usage (apache_beam.pipeline_test.PipelineTest) ... ok
test_metrics_in_source (apache_beam.pipeline_test.PipelineTest) ... ok
test_pipeline_as_context (apache_beam.pipeline_test.PipelineTest) ... ok
test_read (apache_beam.pipeline_test.PipelineTest) ... ok
test_reuse_cloned_custom_transform_instance (apache_beam.pipeline_test.PipelineTest) ... ok
test_reuse_custom_transform_instance (apache_beam.pipeline_test.PipelineTest) ... <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline_test.py>:195: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
cm.exception.message,
ok
test_transform_no_super_init (apache_beam.pipeline_test.PipelineTest) ... ok
test_visit_entire_graph (apache_beam.pipeline_test.PipelineTest) ... ok
test_pcollectionview_not_recreated (apache_beam.pvalue_test.PValueTest) ... ok
test_pvalue_expected_arguments (apache_beam.pvalue_test.PValueTest) ... ok
test_append_extra_options (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_append_verifier_in_extra_opt (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_create_test_pipeline_options (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_empty_option_args_parsing (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_get_option (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_option_args_parsing (apache_beam.test_pipeline_test.TestPipelineTest) ... ok
test_skip_IT (apache_beam.test_pipeline_test.TestPipelineTest) ... SKIP: IT is skipped because --test-pipeline-options is not specified
test_file_checksum_matcher_read_failed (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_file_checksum_matcher_service_error (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_file_checksum_matcher_success (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_pipeline_state_matcher_fails (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher fails when using default expected state ... ok
test_pipeline_state_matcher_given_state (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher successes when matches given state ... ok
test_pipeline_state_matcher_success (apache_beam.tests.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher successes when using default expected state ... ok
test_version (apache_beam.version_test.Version) ... ok
----------------------------------------------------------------------
Ran 1025 tests in 44.502s
OK (skipped=13)
___________________________________ summary ____________________________________
docs: commands succeeded
lint: commands succeeded
py27: commands succeeded
py27cython: commands succeeded
py27gcp: commands succeeded
congratulations :)
# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
New python executable in <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/bin/python>
Installing setuptools, pip, wheel...done.
. sdks/python/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly
deactivate () {
unset -f pydoc >/dev/null 2>&1
# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi
unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}
# unset irrelevant variables
deactivate nondestructive
VIRTUAL_ENV="<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python">
export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi
if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
basename "$VIRTUAL_ENV"
# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc
pydoc () {
python -m pydoc "$@"
}
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
cd sdks/python
pip install -e .[gcp,test]
Obtaining file://<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python>
Collecting avro<2.0.0,>=1.7.7 (from apache-beam==0.7.0.dev0)
Collecting crcmod<2.0,>=1.7 (from apache-beam==0.7.0.dev0)
Collecting dill==0.2.6 (from apache-beam==0.7.0.dev0)
Collecting httplib2<0.10,>=0.8 (from apache-beam==0.7.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==0.7.0.dev0)
Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==0.7.0.dev0)
Collecting protobuf==3.2.0 (from apache-beam==0.7.0.dev0)
Using cached protobuf-3.2.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.10 (from apache-beam==0.7.0.dev0)
Collecting google-apitools<1.0.0,>=0.5.6 (from apache-beam==0.7.0.dev0)
Using cached google_apitools-0.5.8-py2-none-any.whl
Collecting proto-google-cloud-datastore-v1==0.90.0 (from apache-beam==0.7.0.dev0)
Collecting googledatastore==7.0.0 (from apache-beam==0.7.0.dev0)
Collecting google-cloud-bigquery<0.23,>=0.22.1 (from apache-beam==0.7.0.dev0)
Using cached google_cloud_bigquery-0.22.1-py2.py3-none-any.whl
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==0.7.0.dev0)
Using cached PyHamcrest-1.9.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.9 in ./lib/python2.7/site-packages (from mock<3.0.0,>=1.0.1->apache-beam==0.7.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from mock<3.0.0,>=1.0.1->apache-beam==0.7.0.dev0)
Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==0.7.0.dev0)
Using cached pbr-2.0.0-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==0.7.0.dev0)
Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from oauth2client<4.0.0,>=2.0.1->apache-beam==0.7.0.dev0)
Using cached pyasn1_modules-0.0.8-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from oauth2client<4.0.0,>=2.0.1->apache-beam==0.7.0.dev0)
Using cached pyasn1-0.2.3-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./lib/python2.7/site-packages (from protobuf==3.2.0->apache-beam==0.7.0.dev0)
Collecting googleapis-common-protos<2.0dev,>=1.5.0 (from proto-google-cloud-datastore-v1==0.90.0->apache-beam==0.7.0.dev0)
Collecting google-cloud-core<0.23dev,>=0.22.1 (from google-cloud-bigquery<0.23,>=0.22.1->apache-beam==0.7.0.dev0)
Using cached google_cloud_core-0.22.1-py2.py3-none-any.whl
Requirement already satisfied: appdirs>=1.4.0 in ./lib/python2.7/site-packages (from setuptools->protobuf==3.2.0->apache-beam==0.7.0.dev0)
Requirement already satisfied: packaging>=16.8 in ./lib/python2.7/site-packages (from setuptools->protobuf==3.2.0->apache-beam==0.7.0.dev0)
Collecting google-auth-httplib2 (from google-cloud-core<0.23dev,>=0.22.1->google-cloud-bigquery<0.23,>=0.22.1->apache-beam==0.7.0.dev0)
Using cached google_auth_httplib2-0.0.2-py2.py3-none-any.whl
Collecting google-auth<2.0.0dev,>=0.4.0 (from google-cloud-core<0.23dev,>=0.22.1->google-cloud-bigquery<0.23,>=0.22.1->apache-beam==0.7.0.dev0)
Using cached google_auth-0.8.0-py2.py3-none-any.whl
Requirement already satisfied: pyparsing in ./lib/python2.7/site-packages (from packaging>=16.8->setuptools->protobuf==3.2.0->apache-beam==0.7.0.dev0)
Installing collected packages: avro, crcmod, dill, httplib2, funcsigs, pbr, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pyyaml, google-apitools, googleapis-common-protos, proto-google-cloud-datastore-v1, googledatastore, google-auth, google-auth-httplib2, google-cloud-core, google-cloud-bigquery, pyhamcrest, apache-beam
Running setup.py develop for apache-beam
Successfully installed apache-beam avro-1.8.1 crcmod-1.7 dill-0.2.6 funcsigs-1.0.2 google-apitools-0.5.8 google-auth-0.8.0 google-auth-httplib2-0.0.2 google-cloud-bigquery-0.22.1 google-cloud-core-0.22.1 googleapis-common-protos-1.5.2 googledatastore-7.0.0 httplib2-0.9.2 mock-2.0.0 oauth2client-3.0.0 pbr-2.0.0 proto-google-cloud-datastore-v1-0.90.0 protobuf-3.2.0 pyasn1-0.2.3 pyasn1-modules-0.0.8 pyhamcrest-1.9.0 pyyaml-3.12 rsa-3.4.2
# Run wordcount in the Direct Runner and validate output.
echo ">>> RUNNING DIRECT RUNNER py-wordcount"
>>> RUNNING DIRECT RUNNER py-wordcount
python -m apache_beam.examples.wordcount --output /tmp/py-wordcount-direct
No handlers could be found for logger "oauth2client.contrib.multistore_file"
INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO:root:Running pipeline with DirectRunner.
WARNING:root:Task failed: Traceback (most recent call last):
File "apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
File "apache_beam/runners/direct/transform_evaluator.py", line 204, in finish_bundle
range_tracker = self._source.get_range_tracker(None, None)
File "apache_beam/io/filebasedsource.py", line 224, in get_range_tracker
return self._get_concat_source().get_range_tracker(start_position,
File "apache_beam/utils/value_provider.py", line 105, in _f
raise RuntimeError('%s not accessible' % obj)
RuntimeError: RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
Traceback (most recent call last):
File "apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
File "apache_beam/runners/direct/transform_evaluator.py", line 204, in finish_bundle
range_tracker = self._source.get_range_tracker(None, None)
File "apache_beam/io/filebasedsource.py", line 224, in get_range_tracker
return self._get_concat_source().get_range_tracker(start_position,
File "apache_beam/utils/value_provider.py", line 105, in _f
raise RuntimeError('%s not accessible' % obj)
RuntimeError: RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
WARNING:root:A task failed with exception.
RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount.py",> line 119, in <module>
run()
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount.py",> line 109, in run
result.wait_until_finish()
File "apache_beam/runners/direct/direct_runner.py", line 157, in wait_until_finish
self._executor.await_completion()
File "apache_beam/runners/direct/executor.py", line 335, in await_completion
self._executor.await_completion()
File "apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
File "apache_beam/runners/direct/transform_evaluator.py", line 204, in finish_bundle
range_tracker = self._source.get_range_tracker(None, None)
File "apache_beam/io/filebasedsource.py", line 224, in get_range_tracker
return self._get_concat_source().get_range_tracker(start_position,
File "apache_beam/utils/value_provider.py", line 105, in _f
raise RuntimeError('%s not accessible' % obj)
RuntimeError: RuntimeValueProvider(option: input, type: str, default_value: 'gs://dataflow-samples/shakespeare/kinglear.txt') not accessible
Build step 'Execute shell' marked build as failure
Build failed in Jenkins: beam_PostCommit_Python_Verify #1517
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/1517/display/redirect>
------------------------------------------
GitHub pull request #2244 of commit 08037d289319374401b9189c3a5df27b37b13e33, no merge conflicts.
Setting status of 08037d289319374401b9189c3a5df27b37b13e33 to PENDING with url https://builds.apache.org/job/beam_PostCommit_Python_Verify/1517/ and message: 'Build started sha1 is merged.'
Using context: Jenkins: Python SDK PostCommit Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/beam.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:806)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
at hudson.scm.SCM.checkout(SCM.java:495)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1278)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1728)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*" returned status code 128:
stdout:
stderr: error: RPC failed; result=7, HTTP code = 0
fatal: The remote end hung up unexpectedly
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1793)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1519)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:315)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to beam3(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1537)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:822)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor756.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy81.execute(Unknown Source)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:804)
... 11 more
ERROR: null
Retrying after 10 seconds
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
> git rev-parse refs/remotes/origin/pr/2244/merge^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/pr/2244/merge^{commit} # timeout=10
Checking out Revision 8219fd67aadd4da90c250f96ca7e4416ca87a5e2 (refs/remotes/origin/pr/2244/merge)
> git config core.sparsecheckout # timeout=10
> git checkout -f 8219fd67aadd4da90c250f96ca7e4416ca87a5e2
First time build. Skipping changelog.
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
SPARK_LOCAL_IP=127.0.0.1
[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe /tmp/hudson759854892060690643.sh
+ bash sdks/python/run_postcommit.sh
# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/
# Remove any tox cache from previous workspace
rm -rf sdks/python/target/.tox
# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied (use --upgrade to upgrade): virtualenv in /home/jenkins/.local/lib/python2.7/site-packages
Cleaning up...
# INFRA does not install tox
pip install tox --user
Requirement already satisfied (use --upgrade to upgrade): tox in /home/jenkins/.local/lib/python2.7/site-packages
Cleaning up...
# Tox runs unit tests in a virtual environment
${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
GLOB sdist-make: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/setup.py>
ERROR: invocation failed (exit code 1), logfile: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/target/.tox/log/tox-0.log>
ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/bin/python', local('<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/setup.py'),> 'sdist', '--formats=zip', '--dist-dir', local('<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/target/.tox/dist')]>
env: None
setup.py:58: UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is recommended.
_PIP_VERSION, REQUIRED_PIP_VERSION
Traceback (most recent call last):
File "setup.py", line 156, in <module>
'beam_test_plugin = test_config:BeamTestPlugin'
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239, in __init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264, in fetch_build_eggs
replace_conflicting=True
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in best_match
return self.obtain(req, installer) # try and download/install
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in obtain
return installer(requirement)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314, in fetch_build_egg
return cmd.easy_install(req)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 604, in easy_install
self.local_index
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 602, in fetch_distribution
return dist.clone(location=self.download(dist.location, tmpdir))
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 518, in download
found = self._download_url(scheme.group(1), spec, tmpdir)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 766, in _download_url
return self._attempt_download(url, filename)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 772, in _attempt_download
headers = self._download_to(url, filename)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 680, in _download_to
tfp = open(filename,'wb')
IOError: [Errno 2] No such file or directory: '/tmp/easy_install-azqCRF/nose-1.3.7.tar.gz'
ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/setup.py> sdist --formats=zip --dist-dir <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/target/.tox/dist> (see <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/target/.tox/log/tox-0.log)',> 1)
Build step 'Execute shell' marked build as failure