You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2018/12/03 16:30:01 UTC
Build failed in Jenkins: beam_PostCommit_Python_Verify #6726
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/6726/display/redirect?page=changes>
Changes:
[thw] [BEAM-6146] Run pre-commit wordcount in batch and streaming mode.
------------------------------------------
[...truncated 167.82 KB...]
+ true
+ mkdir -p ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
+ cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml ./apache_beam/io/hdfs_integration_test/Dockerfile ./apache_beam/io/hdfs_integration_test/hdfscli.cfg ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
+ cp -r ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
+ cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6726
+ PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6726
+ '[' -z jenkins-beam_PostCommit_Python_Verify-6726 ']'
+ COLOR_OPT=--no-ansi
+ COMPOSE_OPT='-p hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6726 --no-ansi'
+ cd ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
+ docker network prune --force
+ trap finally EXIT
+ docker-compose -p hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6726 --no-ansi build
namenode uses an image, skipping
datanode uses an image, skipping
Building test
Step 1/9 : FROM python:2
---> f67e752245d6
Step 2/9 : WORKDIR /app
---> Using cache
---> 1c741002a3ed
Step 3/9 : ENV HDFSCLI_CONFIG /app/sdks/python/apache_beam/io/hdfs_integration_test/hdfscli.cfg
---> Using cache
---> 7962da95064b
Step 4/9 : RUN pip install --no-cache-dir holdup gsutil
---> Using cache
---> 7871e416a64c
Step 5/9 : RUN gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt .
---> Using cache
---> b78f9af4ffec
Step 6/9 : ADD sdks/python /app/sdks/python
---> ce94b5313483
Removing intermediate container c0cebd278d34
Step 7/9 : ADD model /app/model
---> d428f8362938
Removing intermediate container 6ba76c0e57fd
Step 8/9 : RUN cd sdks/python && python setup.py sdist && pip install --no-cache-dir $(ls dist/apache-beam-*.tar.gz | tail -n1)[gcp]
---> Running in 37e89ea2b088
Service 'test' failed to build: grpc: the connection is unavailable
+ finally
+ docker-compose -p hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6726 --no-ansi down
Removing network hdfs_it-jenkins-beam_postcommit_python_verify-6726_test_net
Network hdfs_it-jenkins-beam_postcommit_python_verify-6726_test_net not found.
real 0m0.315s
user 0m0.267s
sys 0m0.042s
> Task :beam-sdks-python:hdfsIntegrationTest FAILED
:beam-sdks-python:hdfsIntegrationTest (Thread[Task worker for ':',5,main]) completed. Took 15.879 secs.
:beam-sdks-python:postCommitIT (Thread[Task worker for ':',5,main]) started.
> Task :beam-sdks-python:postCommitIT
Caching disabled for task ':beam-sdks-python:postCommitIT': Caching has not been enabled for the task
Task ':beam-sdks-python:postCommitIT' is not up-to-date because:
Task has not declared any outputs despite executing actions.
Starting process 'command 'sh''. Working directory: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python> Command: sh -c . <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/bin/activate> && ./scripts/run_integration_test.sh --test_opts "--nocapture --processes=8 --process-timeout=4500 --attr=IT"
Successfully started process 'command 'sh''
###########################################################################
# Build pipeline options if not provided in --pipeline_opts from commandline
if [[ -z $PIPELINE_OPTS ]]; then
# Check that the script is running in a known directory.
if [[ $PWD != *sdks/python* ]]; then
echo 'Unable to locate Apache Beam Python SDK root directory'
exit 1
fi
# Go to the Apache Beam Python SDK root
if [[ "*sdks/python" != $PWD ]]; then
cd $(pwd | sed 's/sdks\/python.*/sdks\/python/')
fi
# Create a tarball if not exists
if [[ $(find ${SDK_LOCATION}) ]]; then
SDK_LOCATION=$(find ${SDK_LOCATION})
else
python setup.py -q sdist
SDK_LOCATION=$(find dist/apache-beam-*.tar.gz)
fi
# Install test dependencies for ValidatesRunner tests.
echo "pyhamcrest" > postcommit_requirements.txt
echo "mock" >> postcommit_requirements.txt
# Options used to run testing pipeline on Cloud Dataflow Service. Also used for
# running on DirectRunner (some options ignored).
opts=(
"--runner=$RUNNER"
"--project=$PROJECT"
"--staging_location=$GCS_LOCATION/staging-it"
"--temp_location=$GCS_LOCATION/temp-it"
"--output=$GCS_LOCATION/py-it-cloud/output"
"--sdk_location=$SDK_LOCATION"
"--requirements_file=postcommit_requirements.txt"
"--num_workers=$NUM_WORKERS"
"--sleep_secs=$SLEEP_SECS"
)
# Add --streaming if provided
if [[ "$STREAMING" = true ]]; then
opts+=("--streaming")
fi
# Add --dataflow_worker_jar if provided
if [[ ! -z "$WORKER_JAR" ]]; then
opts+=("--dataflow_worker_jar=$WORKER_JAR")
fi
PIPELINE_OPTS=$(IFS=" " ; echo "${opts[*]}")
fi
pwd | sed 's/sdks\/python.*/sdks\/python/'
find ${SDK_LOCATION}
find ${SDK_LOCATION}
IFS=" " ; echo "${opts[*]}"
###########################################################################
# Run tests and validate that jobs finish successfully.
echo ">>> RUNNING integration tests with pipeline options: $PIPELINE_OPTS"
python setup.py nosetests \
--test-pipeline-options="$PIPELINE_OPTS" \
$TEST_OPTS
>>> RUNNING integration tests with pipeline options: --runner=TestDataflowRunner --project=apache-beam-testing --staging_location=gs://temp-storage-for-end-to-end-tests/staging-it --temp_location=gs://temp-storage-for-end-to-end-tests/temp-it --output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output --sdk_location=build/apache-beam.tar.gz --requirements_file=postcommit_requirements.txt --num_workers=1 --sleep_secs=20
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/setuptools/dist.py>:470: UserWarning: Normalizing '2.10.0.dev' to '2.10.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
test_datastore_wordcount_it (apache_beam.examples.cookbook.datastore_wordcount_it_test.DatastoreWordCountIT) ... ok
test_bigquery_tornadoes_it (apache_beam.examples.cookbook.bigquery_tornadoes_it_test.BigqueryTornadoesIT) ... ok
test_autocomplete_it (apache_beam.examples.complete.autocomplete_test.AutocompleteTest) ... ok
test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... ok
test_streaming_wordcount_it (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) ... ok
test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... ok
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... ok
test_game_stats_it (apache_beam.examples.complete.game.game_stats_it_test.GameStatsIT) ... ok
test_user_score_it (apache_beam.examples.complete.game.user_score_it_test.UserScoreIT) ... ok
test_hourly_team_score_it (apache_beam.examples.complete.game.hourly_team_score_it_test.HourlyTeamScoreIT) ... ok
test_avro_it (apache_beam.examples.fastavro_it_test.FastavroIT) ... ok
test_bigquery_read_1M_python (apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT) ... ok
test_streaming_data_only (apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest) ... ok
test_streaming_with_attributes (apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest) ... ok
test_big_query_legacy_sql (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_big_query_new_types (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_big_query_standard_sql (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_datastore_write_limit (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT) ... ok
----------------------------------------------------------------------
XML: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 18 tests in 2790.119s
OK
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_43_40-13772820329352788735?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_50_56-15846424801118007801?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_57_42-4134577748112546691?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_08_03_28-17789273204033529859?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_08_10_23-15226497552090051855?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_08_17_13-8781988541621296993?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_08_23_33-14755229050793755118?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_43_42-8875025924756818422?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_43_41-12839423821742840430?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_43_43-17692856586917636288?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_56_28-8724762812108164136?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_08_02_41-158950842840120112?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_08_09_42-4714506481215450868?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_43_40-10525035945036875209?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_43_40-4312283440227002454?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_50_37-557863314050990283?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_58_02-8016252760449550359?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_08_04_22-13030124664961462232?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_43_40-6872011982588425340?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_50_35-4320271260632896921?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_57_28-6201917683585584228?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_43_41-9677227907378020690?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_07_51_56-11664232783393639442?project=apache-beam-testing.
:beam-sdks-python:postCommitIT (Thread[Task worker for ':',5,main]) completed. Took 46 mins 30.928 secs.
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build.gradle'> line: 222
* What went wrong:
Execution failed for task ':beam-sdks-python:directRunnerIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build.gradle'> line: 376
* What went wrong:
Execution failed for task ':beam-sdks-python:hdfsIntegrationTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
BUILD FAILED in 1h 3m 36s
6 actionable tasks: 6 executed
Publishing build scan...
https://gradle.com/s/tee2ozbldy2kw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_Python_Verify
#6730
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/6730/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Verify #6729
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/6729/display/redirect?page=changes>
Changes:
[relax] Add more AVRO utilities to convert between Beam and Avro. Add
------------------------------------------
[...truncated 403.04 KB...]
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": []
},
{
"@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": []
}
],
"is_pair_like": true
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "out",
"user_name": "write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "<string of 2364 bytes>",
"user_name": "write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: <Job
createTime: u'2018-12-03T20:59:00.377721Z'
currentStateTime: u'1970-01-01T00:00:00Z'
id: u'2018-12-03_12_58_59-12370474966678467864'
location: u'us-central1'
name: u'beamapp-jenkins-1203205849-967147'
projectId: u'apache-beam-testing'
stageStates: []
startTime: u'2018-12-03T20:59:00.377721Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
root: INFO: Created job with id: [2018-12-03_12_58_59-12370474966678467864]
root: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_58_59-12370474966678467864?project=apache-beam-testing
root: INFO: Job 2018-12-03_12_58_59-12370474966678467864 is in state JOB_STATE_RUNNING
root: INFO: 2018-12-03T20:58:59.613Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2018-12-03_12_58_59-12370474966678467864. The number of workers will be between 1 and 1000.
root: INFO: 2018-12-03T20:58:59.653Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2018-12-03_12_58_59-12370474966678467864.
root: INFO: 2018-12-03T20:59:02.353Z: JOB_MESSAGE_DETAILED: Checking permissions granted to controller Service Account.
root: INFO: 2018-12-03T20:59:03.652Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-b.
root: INFO: 2018-12-03T20:59:04.261Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
root: INFO: 2018-12-03T20:59:04.295Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
root: INFO: 2018-12-03T20:59:04.335Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step group: GroupByKey not followed by a combiner.
root: INFO: 2018-12-03T20:59:04.392Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
root: INFO: 2018-12-03T20:59:04.425Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-12-03T20:59:04.546Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
root: INFO: 2018-12-03T20:59:04.600Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
root: INFO: 2018-12-03T20:59:04.638Z: JOB_MESSAGE_DETAILED: Fusing consumer group/Write into group/Reify
root: INFO: 2018-12-03T20:59:04.673Z: JOB_MESSAGE_DETAILED: Fusing consumer group/GroupByWindow into group/Read
root: INFO: 2018-12-03T20:59:04.710Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/GroupByKey/GroupByWindow into write/Write/WriteImpl/GroupByKey/Read
root: INFO: 2018-12-03T20:59:04.745Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/GroupByKey/Write into write/Write/WriteImpl/GroupByKey/Reify
root: INFO: 2018-12-03T20:59:04.801Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/Extract into write/Write/WriteImpl/GroupByKey/GroupByWindow
root: INFO: 2018-12-03T20:59:04.844Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/WindowInto(WindowIntoFn) into write/Write/WriteImpl/Pair
root: INFO: 2018-12-03T20:59:04.888Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/GroupByKey/Reify into write/Write/WriteImpl/WindowInto(WindowIntoFn)
root: INFO: 2018-12-03T20:59:04.924Z: JOB_MESSAGE_DETAILED: Fusing consumer split into read/Read
root: INFO: 2018-12-03T20:59:04.957Z: JOB_MESSAGE_DETAILED: Fusing consumer group/Reify into pair_with_one
root: INFO: 2018-12-03T20:59:04.985Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/Pair into write/Write/WriteImpl/WriteBundles/WriteBundles
root: INFO: 2018-12-03T20:59:05.023Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/WriteBundles/WriteBundles into format
root: INFO: 2018-12-03T20:59:05.060Z: JOB_MESSAGE_DETAILED: Fusing consumer pair_with_one into split
root: INFO: 2018-12-03T20:59:05.098Z: JOB_MESSAGE_DETAILED: Fusing consumer count into group/GroupByWindow
root: INFO: 2018-12-03T20:59:05.129Z: JOB_MESSAGE_DETAILED: Fusing consumer format into count
root: INFO: 2018-12-03T20:59:05.167Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/InitializeWrite into write/Write/WriteImpl/DoOnce/Read
root: INFO: 2018-12-03T20:59:05.206Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
root: INFO: 2018-12-03T20:59:05.250Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
root: INFO: 2018-12-03T20:59:05.300Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
root: INFO: 2018-12-03T20:59:05.343Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
root: INFO: 2018-12-03T20:59:05.509Z: JOB_MESSAGE_DEBUG: Executing wait step start26
root: INFO: 2018-12-03T20:59:05.597Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/DoOnce/Read+write/Write/WriteImpl/InitializeWrite
root: INFO: 2018-12-03T20:59:05.633Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/GroupByKey/Create
root: INFO: 2018-12-03T20:59:05.646Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
root: INFO: 2018-12-03T20:59:05.665Z: JOB_MESSAGE_BASIC: Executing operation group/Create
root: INFO: 2018-12-03T20:59:05.682Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-b...
root: INFO: 2018-12-03T20:59:05.797Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/GroupByKey/Session" materialized.
root: INFO: 2018-12-03T20:59:05.836Z: JOB_MESSAGE_DEBUG: Value "group/Session" materialized.
root: INFO: 2018-12-03T20:59:05.922Z: JOB_MESSAGE_BASIC: Executing operation read/Read+split+pair_with_one+group/Reify+group/Write
root: INFO: 2018-12-03T20:59:24.908Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
root: INFO: 2018-12-03T21:00:44.447Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
root: INFO: 2018-12-03T21:01:26.630Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
root: INFO: 2018-12-03T21:01:26.669Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
root: INFO: 2018-12-03T21:03:35.935Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/DoOnce/Read.out" materialized.
root: INFO: 2018-12-03T21:03:35.996Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/InitializeWrite.out" materialized.
root: INFO: 2018-12-03T21:03:36.119Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
root: INFO: 2018-12-03T21:03:36.169Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
root: INFO: 2018-12-03T21:03:36.241Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
root: INFO: 2018-12-03T21:03:36.297Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
root: INFO: 2018-12-03T21:03:36.354Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
root: INFO: 2018-12-03T21:03:36.416Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
root: INFO: 2018-12-03T21:03:38.782Z: JOB_MESSAGE_BASIC: Executing operation group/Close
root: INFO: 2018-12-03T21:03:38.902Z: JOB_MESSAGE_BASIC: Executing operation group/Read+group/GroupByWindow+count+format+write/Write/WriteImpl/WriteBundles/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/GroupByKey/Reify+write/Write/WriteImpl/GroupByKey/Write
root: INFO: 2018-12-03T21:03:50.146Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/GroupByKey/Close
root: INFO: 2018-12-03T21:03:50.256Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/GroupByKey/Read+write/Write/WriteImpl/GroupByKey/GroupByWindow+write/Write/WriteImpl/Extract
root: INFO: 2018-12-03T21:04:01.220Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/Extract.out" materialized.
root: INFO: 2018-12-03T21:04:01.305Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
root: INFO: 2018-12-03T21:04:01.343Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
root: INFO: 2018-12-03T21:04:01.435Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
root: INFO: 2018-12-03T21:04:01.478Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
root: INFO: 2018-12-03T21:04:01.561Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/PreFinalize/PreFinalize
root: INFO: 2018-12-03T21:04:07.593Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/PreFinalize.out" materialized.
root: INFO: 2018-12-03T21:04:07.686Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
root: INFO: 2018-12-03T21:04:07.810Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
root: INFO: 2018-12-03T21:04:07.908Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
root: INFO: 2018-12-03T21:04:12.075Z: JOB_MESSAGE_DEBUG: Executing success step success24
root: INFO: 2018-12-03T21:04:12.234Z: JOB_MESSAGE_DETAILED: Cleaning up.
root: INFO: 2018-12-03T21:04:12.353Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
root: INFO: 2018-12-03T21:04:12.400Z: JOB_MESSAGE_BASIC: Stopping worker pool...
root: INFO: 2018-12-03T21:05:59.080Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
root: INFO: 2018-12-03T21:05:59.139Z: JOB_MESSAGE_DETAILED: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job.
root: INFO: 2018-12-03T21:05:59.234Z: JOB_MESSAGE_BASIC: Worker pool stopped.
root: INFO: 2018-12-03T21:05:59.298Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
root: INFO: Job 2018-12-03_12_58_59-12370474966678467864 is in state JOB_STATE_DONE
root: INFO: Wait 20 seconds...
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results*-of-*' -> 'gs\\:\\/\\/temp\\-storage\\-for\\-end\\-to\\-end\\-tests\\/py\\-it\\-cloud\\/output\\/1543870729\\/results[^/\\\\]*\\-of\\-[^/\\\\]*'
root: DEBUG: Connecting using Google Application Default Credentials.
root: INFO: Starting the size estimation of the input
oauth2client.transport: INFO: Attempting refresh to obtain initial access_token
root: INFO: Finished listing 0 files in 0.0510249137878 seconds.
root: WARNING: Retry with exponential backoff: waiting for 4.31455217034 seconds before retrying _read_with_retry because we caught exception: IOError: No such file or directory: gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results*-of-*
Traceback for above exception (most recent call last):
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/utils/retry.py",> line 184, in wrapper
return fun(*args, **kwargs)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/testing/pipeline_verifiers.py",> line 122, in _read_with_retry
raise IOError('No such file or directory: %s' % self.file_path)
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results*-of-*' -> 'gs\\:\\/\\/temp\\-storage\\-for\\-end\\-to\\-end\\-tests\\/py\\-it\\-cloud\\/output\\/1543870729\\/results[^/\\\\]*\\-of\\-[^/\\\\]*'
root: DEBUG: Connecting using Google Application Default Credentials.
root: INFO: Starting the size estimation of the input
oauth2client.transport: INFO: Attempting refresh to obtain initial access_token
root: INFO: Finished listing 0 files in 0.0611021518707 seconds.
root: WARNING: Retry with exponential backoff: waiting for 7.49365743076 seconds before retrying _read_with_retry because we caught exception: IOError: No such file or directory: gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results*-of-*
Traceback for above exception (most recent call last):
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/utils/retry.py",> line 184, in wrapper
return fun(*args, **kwargs)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/testing/pipeline_verifiers.py",> line 122, in _read_with_retry
raise IOError('No such file or directory: %s' % self.file_path)
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results*-of-*' -> 'gs\\:\\/\\/temp\\-storage\\-for\\-end\\-to\\-end\\-tests\\/py\\-it\\-cloud\\/output\\/1543870729\\/results[^/\\\\]*\\-of\\-[^/\\\\]*'
root: DEBUG: Connecting using Google Application Default Credentials.
root: INFO: Starting the size estimation of the input
oauth2client.transport: INFO: Attempting refresh to obtain initial access_token
root: INFO: Finished listing 0 files in 0.05863904953 seconds.
root: WARNING: Retry with exponential backoff: waiting for 14.1921020421 seconds before retrying _read_with_retry because we caught exception: IOError: No such file or directory: gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results*-of-*
Traceback for above exception (most recent call last):
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/utils/retry.py",> line 184, in wrapper
return fun(*args, **kwargs)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/testing/pipeline_verifiers.py",> line 122, in _read_with_retry
raise IOError('No such file or directory: %s' % self.file_path)
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results*-of-*' -> 'gs\\:\\/\\/temp\\-storage\\-for\\-end\\-to\\-end\\-tests\\/py\\-it\\-cloud\\/output\\/1543870729\\/results[^/\\\\]*\\-of\\-[^/\\\\]*'
root: DEBUG: Connecting using Google Application Default Credentials.
root: INFO: Starting the size estimation of the input
oauth2client.transport: INFO: Attempting refresh to obtain initial access_token
root: INFO: Finished listing 0 files in 0.109956026077 seconds.
root: WARNING: Retry with exponential backoff: waiting for 21.0772065078 seconds before retrying _read_with_retry because we caught exception: IOError: No such file or directory: gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results*-of-*
Traceback for above exception (most recent call last):
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/utils/retry.py",> line 184, in wrapper
return fun(*args, **kwargs)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/testing/pipeline_verifiers.py",> line 122, in _read_with_retry
raise IOError('No such file or directory: %s' % self.file_path)
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1543870729/results*-of-*' -> 'gs\\:\\/\\/temp\\-storage\\-for\\-end\\-to\\-end\\-tests\\/py\\-it\\-cloud\\/output\\/1543870729\\/results[^/\\\\]*\\-of\\-[^/\\\\]*'
root: DEBUG: Connecting using Google Application Default Credentials.
root: INFO: Starting the size estimation of the input
oauth2client.transport: INFO: Attempting refresh to obtain initial access_token
root: INFO: Finished listing 0 files in 0.0576350688934 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 18 tests in 2927.334s
FAILED (errors=1)
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_51_47-17404234383436144790?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_58_59-1504022593666987318?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_06_41-6077525281555364371?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_11_47-18328906429259571458?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_18_49-8337411273491287513?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_26_41-64817165657044287?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_33_52-8446896109882998958?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_51_51-12546051392931984583?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_06_11-5707054306572349961?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_51_48-15324541914704386930?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_51_49-3091225294289201464?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_04_52-7972281351601959269?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_10_45-10294669503302078667?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_16_43-2049163303092519491?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_51_47-9605525696972181336?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_51_47-13209287438985498185?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_58_33-12781178033881288843?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_06_52-4367552905654817106?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_13_03-1788626803067059044?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_51_47-14418650846430674988?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_58_59-12370474966678467864?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_12_51_47-10220867255695258818?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_13_00_24-18097657083890911691?project=apache-beam-testing.
> Task :beam-sdks-python:postCommitIT FAILED
:beam-sdks-python:postCommitIT (Thread[Task worker for ':',5,main]) completed. Took 48 mins 48.131 secs.
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build.gradle'> line: 337
* What went wrong:
Execution failed for task ':beam-sdks-python:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 53m 15s
6 actionable tasks: 6 executed
Publishing build scan...
https://gradle.com/s/m7vqaijval6l2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Verify #6728
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/6728/display/redirect?page=changes>
Changes:
[mxm] [BEAM-5859] Improve operator names for portable pipelines
------------------------------------------
[...truncated 315.20 KB...]
datanode_1 | 18/12/03 19:14:21 INFO datanode.DataNode: PacketResponder: BP-913278859-172.18.0.2-1543864407037:blk_1073741826_1002, type=LAST_IN_PIPELINE terminating
namenode_1 | 18/12/03 19:14:21 INFO hdfs.StateChange: DIR* completeFile: /beam-temp-py-wordcount-integration-a310aa0ef72f11e8bc330242ac120004/7a019ed1-4de1-4d82-a072-52dac7bbd488.py-wordcount-integration is closed by DFSClient_NONMAPREDUCE_-1010147539_69
test_1 | INFO:root:Running (write/Write/WriteImpl/GroupByKey/Read)+((ref_AppliedPTransform_write/Write/WriteImpl/Extract_25)+(ref_PCollection_PCollection_17/Write))
test_1 | INFO:root:Running ((ref_PCollection_PCollection_9/Read)+(ref_AppliedPTransform_write/Write/WriteImpl/PreFinalize_26))+(ref_PCollection_PCollection_18/Write)
test_1 | INFO:root:Running (ref_PCollection_PCollection_9/Read)+(ref_AppliedPTransform_write/Write/WriteImpl/FinalizeWrite_27)
test_1 | INFO:root:Starting finalize_write threads with num_shards: 1 (skipped: 0), batches: 1, num_threads: 1
test_1 | INFO:root:Renamed 1 shards in 0.14 seconds.
test_1 | INFO:root:number of empty lines: 1663
test_1 | INFO:root:average word length: 4
hdfs_it-jenkins-beam_postcommit_python_verify-6728_test_1 exited with code 0
Stopping hdfs_it-jenkins-beam_postcommit_python_verify-6728_datanode_1 ...
Stopping hdfs_it-jenkins-beam_postcommit_python_verify-6728_namenode_1 ...
Stopping hdfs_it-jenkins-beam_postcommit_python_verify-6728_datanode_1 ... done
Stopping hdfs_it-jenkins-beam_postcommit_python_verify-6728_namenode_1 ... done
Aborting on container exit...
real 1m19.921s
user 0m1.261s
sys 0m0.218s
+ finally
+ docker-compose -p hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6728 --no-ansi down
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6728_test_1 ...
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6728_datanode_1 ...
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6728_namenode_1 ...
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6728_datanode_1 ... done
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6728_test_1 ... done
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6728_namenode_1 ... done
Removing network hdfs_it-jenkins-beam_postcommit_python_verify-6728_test_net
real 0m0.700s
user 0m0.238s
sys 0m0.097s
:beam-sdks-python:hdfsIntegrationTest (Thread[Task worker for ':',5,main]) completed. Took 2 mins 47.672 secs.
:beam-sdks-python:postCommitIT (Thread[Task worker for ':',5,main]) started.
> Task :beam-sdks-python:postCommitIT
Caching disabled for task ':beam-sdks-python:postCommitIT': Caching has not been enabled for the task
Task ':beam-sdks-python:postCommitIT' is not up-to-date because:
Task has not declared any outputs despite executing actions.
Starting process 'command 'sh''. Working directory: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python> Command: sh -c . <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/bin/activate> && ./scripts/run_integration_test.sh --test_opts "--nocapture --processes=8 --process-timeout=4500 --attr=IT"
Successfully started process 'command 'sh''
###########################################################################
# Build pipeline options if not provided in --pipeline_opts from commandline
if [[ -z $PIPELINE_OPTS ]]; then
# Check that the script is running in a known directory.
if [[ $PWD != *sdks/python* ]]; then
echo 'Unable to locate Apache Beam Python SDK root directory'
exit 1
fi
# Go to the Apache Beam Python SDK root
if [[ "*sdks/python" != $PWD ]]; then
cd $(pwd | sed 's/sdks\/python.*/sdks\/python/')
fi
# Create a tarball if not exists
if [[ $(find ${SDK_LOCATION}) ]]; then
SDK_LOCATION=$(find ${SDK_LOCATION})
else
python setup.py -q sdist
SDK_LOCATION=$(find dist/apache-beam-*.tar.gz)
fi
# Install test dependencies for ValidatesRunner tests.
echo "pyhamcrest" > postcommit_requirements.txt
echo "mock" >> postcommit_requirements.txt
# Options used to run testing pipeline on Cloud Dataflow Service. Also used for
# running on DirectRunner (some options ignored).
opts=(
"--runner=$RUNNER"
"--project=$PROJECT"
"--staging_location=$GCS_LOCATION/staging-it"
"--temp_location=$GCS_LOCATION/temp-it"
"--output=$GCS_LOCATION/py-it-cloud/output"
"--sdk_location=$SDK_LOCATION"
"--requirements_file=postcommit_requirements.txt"
"--num_workers=$NUM_WORKERS"
"--sleep_secs=$SLEEP_SECS"
)
# Add --streaming if provided
if [[ "$STREAMING" = true ]]; then
opts+=("--streaming")
fi
# Add --dataflow_worker_jar if provided
if [[ ! -z "$WORKER_JAR" ]]; then
opts+=("--dataflow_worker_jar=$WORKER_JAR")
fi
PIPELINE_OPTS=$(IFS=" " ; echo "${opts[*]}")
fi
pwd | sed 's/sdks\/python.*/sdks\/python/'
find ${SDK_LOCATION}
find ${SDK_LOCATION}
IFS=" " ; echo "${opts[*]}"
###########################################################################
# Run tests and validate that jobs finish successfully.
echo ">>> RUNNING integration tests with pipeline options: $PIPELINE_OPTS"
python setup.py nosetests \
--test-pipeline-options="$PIPELINE_OPTS" \
$TEST_OPTS
>>> RUNNING integration tests with pipeline options: --runner=TestDataflowRunner --project=apache-beam-testing --staging_location=gs://temp-storage-for-end-to-end-tests/staging-it --temp_location=gs://temp-storage-for-end-to-end-tests/temp-it --output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output --sdk_location=build/apache-beam.tar.gz --requirements_file=postcommit_requirements.txt --num_workers=1 --sleep_secs=20
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/setuptools/dist.py>:470: UserWarning: Normalizing '2.10.0.dev' to '2.10.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
test_bigquery_tornadoes_it (apache_beam.examples.cookbook.bigquery_tornadoes_it_test.BigqueryTornadoesIT) ... ok
test_autocomplete_it (apache_beam.examples.complete.autocomplete_test.AutocompleteTest) ... ok
test_datastore_wordcount_it (apache_beam.examples.cookbook.datastore_wordcount_it_test.DatastoreWordCountIT) ... ok
test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... ok
test_streaming_wordcount_it (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) ... ok
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... ok
test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... ok
test_game_stats_it (apache_beam.examples.complete.game.game_stats_it_test.GameStatsIT) ... ok
test_user_score_it (apache_beam.examples.complete.game.user_score_it_test.UserScoreIT) ... ok
test_avro_it (apache_beam.examples.fastavro_it_test.FastavroIT) ... ok
test_hourly_team_score_it (apache_beam.examples.complete.game.hourly_team_score_it_test.HourlyTeamScoreIT) ... ok
test_bigquery_read_1M_python (apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT) ... ok
test_streaming_data_only (apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest) ... ok
test_streaming_with_attributes (apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest) ... ok
test_big_query_legacy_sql (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_big_query_new_types (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_big_query_standard_sql (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_14_55-17099401326547483385?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_21_17-2624717569133755854?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_28_19-200084910725104306?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_34_00-6258127070976838088?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_41_02-389429062816355263?project=apache-beam-testing.
test_datastore_write_limit (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT) ... ERROR
======================================================================
ERROR: test suite for <class 'apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",> line 812, in run
test(orig)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/suite.py",> line 177, in __call__
return self.run(*arg, **kw)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",> line 822, in run
test.config.plugins.addError(test,err)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/manager.py",> line 99, in __call__
return self.call(*arg, **kw)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/manager.py",> line 167, in simple
result = meth(*arg, **kw)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/xunit.py",> line 287, in addError
tb = format_exception(err, self.encoding)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/pyversion.py",> line 214, in format_exception
''.join(traceback.format_exception(*exc_info)),
File "/usr/lib/python2.7/traceback.py", line 141, in format_exception
list = list + format_tb(tb, limit)
File "/usr/lib/python2.7/traceback.py", line 76, in format_tb
return format_list(extract_tb(tb, limit))
File "/usr/lib/python2.7/traceback.py", line 101, in extract_tb
line = linecache.getline(filename, lineno, f.f_globals)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/lib/python2.7/linecache.py",> line 14, in getline
lines = getlines(filename, module_globals)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/lib/python2.7/linecache.py",> line 41, in getlines
return updatecache(filename, module_globals)
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/lib/python2.7/linecache.py",> line 132, in updatecache
lines = fp.readlines()
File "<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",> line 276, in signalhandler
raise TimedOutException()
TimedOutException: '<nose.plugins.multiprocess.NoSharedFixtureContextSuite context=DatastoreWriteIT>'
----------------------------------------------------------------------
XML: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 18 tests in 5306.414s
FAILED (errors=1)
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_14_59-6242933708465181875?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_14_55-15225362793031026859?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_14_58-2221778447835242095?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_27_56-3152183166525773717?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_34_08-13513050039165857630?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_40_41-7731226018863020415?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_14_55-11121215214008527124?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_14_54-14627900940745613184?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_21_15-16197398806541755765?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_28_08-2440635523214490724?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_14_55-11789661623949843891?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_22_17-18433499798521513329?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_29_02-646555723431735953?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_34_48-9640753078834476755?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_14_55-4398940798511063758?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_11_23_22-4517474965600120803?project=apache-beam-testing.
> Task :beam-sdks-python:postCommitIT FAILED
:beam-sdks-python:postCommitIT (Thread[Task worker for ':',5,main]) completed. Took 1 hrs 28 mins 27.21 secs.
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build.gradle'> line: 337
* What went wrong:
Execution failed for task ':beam-sdks-python:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 1h 32m 40s
6 actionable tasks: 6 executed
Publishing build scan...
https://gradle.com/s/7jfszfkvgndea
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Verify #6727
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/6727/display/redirect>
------------------------------------------
[...truncated 149.03 KB...]
OK
:beam-sdks-python:directRunnerIT (Thread[Task worker for ':',5,main]) completed. Took 51.277 secs.
:beam-sdks-python:hdfsIntegrationTest (Thread[Task worker for ':',5,main]) started.
> Task :beam-sdks-python:hdfsIntegrationTest
Caching disabled for task ':beam-sdks-python:hdfsIntegrationTest': Caching has not been enabled for the task
Task ':beam-sdks-python:hdfsIntegrationTest' is not up-to-date because:
Task has not declared any outputs despite executing actions.
Starting process 'command 'sh''. Working directory: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python> Command: sh -c . <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/bin/activate> && ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
Successfully started process 'command 'sh''
++ dirname ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
+ TEST_DIR=./apache_beam/io/hdfs_integration_test
+ ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
+ CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
+ rm -r ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
rm: cannot remove './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration': No such file or directory
+ true
+ mkdir -p ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
+ cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml ./apache_beam/io/hdfs_integration_test/Dockerfile ./apache_beam/io/hdfs_integration_test/hdfscli.cfg ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
+ cp -r ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
+ cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6727
+ PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6727
+ '[' -z jenkins-beam_PostCommit_Python_Verify-6727 ']'
+ COLOR_OPT=--no-ansi
+ COMPOSE_OPT='-p hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6727 --no-ansi'
+ cd ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
+ docker network prune --force
+ trap finally EXIT
+ docker-compose -p hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6727 --no-ansi build
namenode uses an image, skipping
datanode uses an image, skipping
Building test
Step 1/9 : FROM python:2
---> f67e752245d6
Step 2/9 : WORKDIR /app
---> Using cache
---> 1c741002a3ed
Step 3/9 : ENV HDFSCLI_CONFIG /app/sdks/python/apache_beam/io/hdfs_integration_test/hdfscli.cfg
---> Using cache
---> 7962da95064b
Step 4/9 : RUN pip install --no-cache-dir holdup gsutil
---> Using cache
---> 7871e416a64c
Step 5/9 : RUN gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt .
---> Using cache
---> b78f9af4ffec
Step 6/9 : ADD sdks/python /app/sdks/python
---> 483f083dbce2
Removing intermediate container 25ae4a0276b8
Step 7/9 : ADD model /app/model
---> cc3819cb3edf
Removing intermediate container de5ea20f0843
Step 8/9 : RUN cd sdks/python && python setup.py sdist && pip install --no-cache-dir $(ls dist/apache-beam-*.tar.gz | tail -n1)[gcp]
---> Running in 5d65e6a4ef17
Service 'test' failed to build: grpc: the connection is unavailable
+ finally
+ docker-compose -p hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6727 --no-ansi down
Removing network hdfs_it-jenkins-beam_postcommit_python_verify-6727_test_net
Network hdfs_it-jenkins-beam_postcommit_python_verify-6727_test_net not found.
real 0m0.306s
user 0m0.239s
sys 0m0.062s
> Task :beam-sdks-python:hdfsIntegrationTest FAILED
:beam-sdks-python:hdfsIntegrationTest (Thread[Task worker for ':',5,main]) completed. Took 15.889 secs.
:beam-sdks-python:postCommitIT (Thread[Task worker for ':',5,main]) started.
> Task :beam-sdks-python:postCommitIT
Caching disabled for task ':beam-sdks-python:postCommitIT': Caching has not been enabled for the task
Task ':beam-sdks-python:postCommitIT' is not up-to-date because:
Task has not declared any outputs despite executing actions.
Starting process 'command 'sh''. Working directory: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python> Command: sh -c . <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/bin/activate> && ./scripts/run_integration_test.sh --test_opts "--nocapture --processes=8 --process-timeout=4500 --attr=IT"
Successfully started process 'command 'sh''
###########################################################################
# Build pipeline options if not provided in --pipeline_opts from commandline
if [[ -z $PIPELINE_OPTS ]]; then
# Check that the script is running in a known directory.
if [[ $PWD != *sdks/python* ]]; then
echo 'Unable to locate Apache Beam Python SDK root directory'
exit 1
fi
# Go to the Apache Beam Python SDK root
if [[ "*sdks/python" != $PWD ]]; then
cd $(pwd | sed 's/sdks\/python.*/sdks\/python/')
fi
# Create a tarball if not exists
if [[ $(find ${SDK_LOCATION}) ]]; then
SDK_LOCATION=$(find ${SDK_LOCATION})
else
python setup.py -q sdist
SDK_LOCATION=$(find dist/apache-beam-*.tar.gz)
fi
# Install test dependencies for ValidatesRunner tests.
echo "pyhamcrest" > postcommit_requirements.txt
echo "mock" >> postcommit_requirements.txt
# Options used to run testing pipeline on Cloud Dataflow Service. Also used for
# running on DirectRunner (some options ignored).
opts=(
"--runner=$RUNNER"
"--project=$PROJECT"
"--staging_location=$GCS_LOCATION/staging-it"
"--temp_location=$GCS_LOCATION/temp-it"
"--output=$GCS_LOCATION/py-it-cloud/output"
"--sdk_location=$SDK_LOCATION"
"--requirements_file=postcommit_requirements.txt"
"--num_workers=$NUM_WORKERS"
"--sleep_secs=$SLEEP_SECS"
)
# Add --streaming if provided
if [[ "$STREAMING" = true ]]; then
opts+=("--streaming")
fi
# Add --dataflow_worker_jar if provided
if [[ ! -z "$WORKER_JAR" ]]; then
opts+=("--dataflow_worker_jar=$WORKER_JAR")
fi
PIPELINE_OPTS=$(IFS=" " ; echo "${opts[*]}")
fi
pwd | sed 's/sdks\/python.*/sdks\/python/'
find ${SDK_LOCATION}
find ${SDK_LOCATION}
IFS=" " ; echo "${opts[*]}"
###########################################################################
# Run tests and validate that jobs finish successfully.
echo ">>> RUNNING integration tests with pipeline options: $PIPELINE_OPTS"
>>> RUNNING integration tests with pipeline options: --runner=TestDataflowRunner --project=apache-beam-testing --staging_location=gs://temp-storage-for-end-to-end-tests/staging-it --temp_location=gs://temp-storage-for-end-to-end-tests/temp-it --output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output --sdk_location=build/apache-beam.tar.gz --requirements_file=postcommit_requirements.txt --num_workers=1 --sleep_secs=20
python setup.py nosetests \
--test-pipeline-options="$PIPELINE_OPTS" \
$TEST_OPTS
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/setuptools/dist.py>:470: UserWarning: Normalizing '2.10.0.dev' to '2.10.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
test_bigquery_tornadoes_it (apache_beam.examples.cookbook.bigquery_tornadoes_it_test.BigqueryTornadoesIT) ... ok
test_autocomplete_it (apache_beam.examples.complete.autocomplete_test.AutocompleteTest) ... ok
test_datastore_wordcount_it (apache_beam.examples.cookbook.datastore_wordcount_it_test.DatastoreWordCountIT) ... ok
test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... ok
test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... ok
test_game_stats_it (apache_beam.examples.complete.game.game_stats_it_test.GameStatsIT) ... ok
test_streaming_wordcount_it (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) ... ok
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... ok
test_avro_it (apache_beam.examples.fastavro_it_test.FastavroIT) ... ok
test_hourly_team_score_it (apache_beam.examples.complete.game.hourly_team_score_it_test.HourlyTeamScoreIT) ... ok
test_user_score_it (apache_beam.examples.complete.game.user_score_it_test.UserScoreIT) ... ok
test_bigquery_read_1M_python (apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT) ... ok
test_streaming_data_only (apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest) ... ok
test_streaming_with_attributes (apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest) ... ok
test_big_query_legacy_sql (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_big_query_new_types (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_big_query_standard_sql (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_datastore_write_limit (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT) ... ok
----------------------------------------------------------------------
XML: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 18 tests in 2844.189s
OK
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_02_04-14436217429548882716?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_16_23-3347293405940171042?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_23_28-13734449926404958010?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_30_15-7862378516466938726?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_37_07-5969583619680901140?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_43_03-15425445257087043641?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_02_03-12505608979725248482?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_02_05-3274822205684798430?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_14_20-2617944508543719827?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_20_52-9999216565517854652?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_27_23-3826748060615970813?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_02_02-280926078897244759?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_02_01-648247753088031196?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_09_12-11314073523952314272?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_02_01-5019073753394997385?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_08_34-4468746041022537799?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_16_49-4637753410082188685?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_24_38-18241335732970215570?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_02_01-14906972513966307435?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_09_22-14799805343042218050?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_16_13-16798601431321285870?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_02_01-9673962534465971082?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-03_10_09_47-12291465752535212438?project=apache-beam-testing.
:beam-sdks-python:postCommitIT (Thread[Task worker for ':',5,main]) completed. Took 47 mins 24.978 secs.
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build.gradle'> line: 376
* What went wrong:
Execution failed for task ':beam-sdks-python:hdfsIntegrationTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 49m 7s
6 actionable tasks: 6 executed
Publishing build scan...
https://gradle.com/s/nnuzwoxnfvh72
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org