You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2018/12/02 07:31:28 UTC
Build failed in Jenkins: beam_PostCommit_Python_Verify #6714
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/6714/display/redirect>
------------------------------------------
[...truncated 517.92 KB...]
"label": "Transform Function",
"namespace": "apache_beam.transforms.core.ParDo",
"shortValue": "CallableWrapperDoFn",
"type": "STRING",
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
}
],
"non_parallel_inputs": {},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": []
},
{
"@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": []
}
],
"is_pair_like": true
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "out",
"user_name": "assert_that/Unkey.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s21"
},
"serialized_fn": "<string of 968 bytes>",
"user_name": "assert_that/Unkey"
}
},
{
"kind": "ParallelDo",
"name": "s23",
"properties": {
"display_data": [
{
"key": "fn",
"label": "Transform Function",
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn",
"type": "STRING",
"value": "_equal"
},
{
"key": "fn",
"label": "Transform Function",
"namespace": "apache_beam.transforms.core.ParDo",
"shortValue": "CallableWrapperDoFn",
"type": "STRING",
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
}
],
"non_parallel_inputs": {},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": []
},
{
"@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": []
}
],
"is_pair_like": true
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "out",
"user_name": "assert_that/Match.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s22"
},
"serialized_fn": "<string of 1332 bytes>",
"user_name": "assert_that/Match"
}
}
],
"type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: <Job
createTime: u'2018-12-02T06:29:27.947368Z'
currentStateTime: u'1970-01-01T00:00:00Z'
id: u'2018-12-01_22_29_27-868325378819143934'
location: u'us-central1'
name: u'beamapp-jenkins-1202061723-241911'
projectId: u'apache-beam-testing'
stageStates: []
startTime: u'2018-12-02T06:29:27.947368Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
root: INFO: Created job with id: [2018-12-01_22_29_27-868325378819143934]
root: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_29_27-868325378819143934?project=apache-beam-testing
root: INFO: Job 2018-12-01_22_29_27-868325378819143934 is in state JOB_STATE_RUNNING
root: INFO: 2018-12-02T06:29:27.151Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2018-12-01_22_29_27-868325378819143934. The number of workers will be between 1 and 1000.
root: INFO: 2018-12-02T06:29:27.179Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2018-12-01_22_29_27-868325378819143934.
root: INFO: 2018-12-02T06:29:29.954Z: JOB_MESSAGE_DETAILED: Checking permissions granted to controller Service Account.
root: INFO: 2018-12-02T06:29:30.721Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-b.
root: INFO: 2018-12-02T06:29:31.406Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
root: INFO: 2018-12-02T06:29:31.436Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a combiner.
root: INFO: 2018-12-02T06:29:31.464Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step read from datastore/GroupByKey: GroupByKey not followed by a combiner.
root: INFO: 2018-12-02T06:29:31.498Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
root: INFO: 2018-12-02T06:29:31.536Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-12-02T06:29:31.665Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
root: INFO: 2018-12-02T06:29:31.717Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
root: INFO: 2018-12-02T06:29:31.747Z: JOB_MESSAGE_DETAILED: Fusing consumer read from datastore/SplitQuery into read from datastore/UserQuery/Read
root: INFO: 2018-12-02T06:29:31.784Z: JOB_MESSAGE_DETAILED: Fusing consumer read from datastore/GroupByKey/Write into read from datastore/GroupByKey/Reify
root: INFO: 2018-12-02T06:29:31.810Z: JOB_MESSAGE_DETAILED: Fusing consumer read from datastore/Read into read from datastore/Flatten
root: INFO: 2018-12-02T06:29:31.846Z: JOB_MESSAGE_DETAILED: Fusing consumer Globally/CombineGlobally(CountCombineFn)/UnKey into Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract
root: INFO: 2018-12-02T06:29:31.880Z: JOB_MESSAGE_DETAILED: Fusing consumer read from datastore/Flatten into read from datastore/Values
root: INFO: 2018-12-02T06:29:31.917Z: JOB_MESSAGE_DETAILED: Fusing consumer Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract into Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine
root: INFO: 2018-12-02T06:29:31.952Z: JOB_MESSAGE_DETAILED: Fusing consumer Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine into Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read
root: INFO: 2018-12-02T06:29:31.987Z: JOB_MESSAGE_DETAILED: Fusing consumer read from datastore/GroupByKey/Reify into read from datastore/SplitQuery
root: INFO: 2018-12-02T06:29:32.021Z: JOB_MESSAGE_DETAILED: Fusing consumer read from datastore/Values into read from datastore/GroupByKey/GroupByWindow
root: INFO: 2018-12-02T06:29:32.047Z: JOB_MESSAGE_DETAILED: Fusing consumer Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write into Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Reify
root: INFO: 2018-12-02T06:29:32.079Z: JOB_MESSAGE_DETAILED: Fusing consumer Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Reify into Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial
root: INFO: 2018-12-02T06:29:32.115Z: JOB_MESSAGE_DETAILED: Fusing consumer read from datastore/GroupByKey/GroupByWindow into read from datastore/GroupByKey/Read
root: INFO: 2018-12-02T06:29:32.141Z: JOB_MESSAGE_DETAILED: Fusing consumer Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial into Globally/CombineGlobally(CountCombineFn)/KeyWithVoid
root: INFO: 2018-12-02T06:29:32.177Z: JOB_MESSAGE_DETAILED: Fusing consumer Globally/CombineGlobally(CountCombineFn)/KeyWithVoid into read from datastore/Read
root: INFO: 2018-12-02T06:29:32.212Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Match into assert_that/Unkey
root: INFO: 2018-12-02T06:29:32.247Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2018-12-02T06:29:32.279Z: JOB_MESSAGE_DETAILED: Unzipping flatten s19 for input s17.out
root: INFO: 2018-12-02T06:29:32.305Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/GroupByKey/Reify, through flatten assert_that/Group/Flatten, into producer assert_that/Group/pair_with_0
root: INFO: 2018-12-02T06:29:32.338Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/Map(_merge_tagged_vals_under_key) into assert_that/Group/GroupByKey/GroupByWindow
root: INFO: 2018-12-02T06:29:32.375Z: JOB_MESSAGE_DETAILED: Unzipping flatten s19-u40 for input s20-reify-value9-c38
root: INFO: 2018-12-02T06:29:32.409Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/GroupByKey/Write, through flatten s19-u40, into producer assert_that/Group/GroupByKey/Reify
root: INFO: 2018-12-02T06:29:32.432Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/GroupByKey/GroupByWindow into assert_that/Group/GroupByKey/Read
root: INFO: 2018-12-02T06:29:32.466Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/pair_with_0 into assert_that/Create/Read
root: INFO: 2018-12-02T06:29:32.499Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/GroupByKey/Write into assert_that/Group/GroupByKey/Reify
root: INFO: 2018-12-02T06:29:32.525Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/GroupByKey/Reify into assert_that/Group/pair_with_1
root: INFO: 2018-12-02T06:29:32.555Z: JOB_MESSAGE_DETAILED: Fusing consumer Globally/CombineGlobally(CountCombineFn)/InjectDefault/InjectDefault into Globally/CombineGlobally(CountCombineFn)/DoOnce/Read
root: INFO: 2018-12-02T06:29:32.584Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/WindowInto(WindowIntoFn) into Globally/CombineGlobally(CountCombineFn)/InjectDefault/InjectDefault
root: INFO: 2018-12-02T06:29:32.619Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/pair_with_1 into assert_that/ToVoidKey
root: INFO: 2018-12-02T06:29:32.653Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/ToVoidKey into assert_that/WindowInto(WindowIntoFn)
root: INFO: 2018-12-02T06:29:32.687Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
root: INFO: 2018-12-02T06:29:32.719Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
root: INFO: 2018-12-02T06:29:32.745Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
root: INFO: 2018-12-02T06:29:32.768Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
root: INFO: 2018-12-02T06:29:32.907Z: JOB_MESSAGE_DEBUG: Executing wait step start51
root: INFO: 2018-12-02T06:29:32.965Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Create
root: INFO: 2018-12-02T06:29:32.990Z: JOB_MESSAGE_BASIC: Executing operation read from datastore/GroupByKey/Create
root: INFO: 2018-12-02T06:29:33.013Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
root: INFO: 2018-12-02T06:29:33.016Z: JOB_MESSAGE_BASIC: Executing operation Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create
root: INFO: 2018-12-02T06:29:33.048Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-b...
root: INFO: 2018-12-02T06:29:33.111Z: JOB_MESSAGE_DEBUG: Value "assert_that/Group/GroupByKey/Session" materialized.
root: INFO: 2018-12-02T06:29:33.133Z: JOB_MESSAGE_DEBUG: Value "read from datastore/GroupByKey/Session" materialized.
root: INFO: 2018-12-02T06:29:33.155Z: JOB_MESSAGE_DEBUG: Value "Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Session" materialized.
root: INFO: 2018-12-02T06:29:33.179Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Create/Read+assert_that/Group/pair_with_0+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
root: INFO: 2018-12-02T06:29:33.203Z: JOB_MESSAGE_BASIC: Executing operation read from datastore/UserQuery/Read+read from datastore/SplitQuery+read from datastore/GroupByKey/Reify+read from datastore/GroupByKey/Write
root: INFO: 2018-12-02T06:29:45.373Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
root: INFO: 2018-12-02T06:31:22.208Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
root: INFO: 2018-12-02T06:31:22.245Z: JOB_MESSAGE_DETAILED: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job.
root: INFO: 2018-12-02T06:33:13.505Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
root: INFO: 2018-12-02T06:33:13.536Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
oauth2client.transport: INFO: Refreshing due to a 401 (attempt 1/2)
root: INFO: 2018-12-02T07:29:33.193Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. You can get help with Cloud Dataflow at https://cloud.google.com/dataflow/support.
root: INFO: 2018-12-02T07:29:33.326Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2018-12-01_22_29_27-868325378819143934.
root: INFO: 2018-12-02T07:29:33.408Z: JOB_MESSAGE_DETAILED: Cleaning up.
root: INFO: 2018-12-02T07:29:33.468Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
root: INFO: 2018-12-02T07:29:33.489Z: JOB_MESSAGE_BASIC: Stopping worker pool...
root: INFO: 2018-12-02T07:31:19.919Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of workers to 0 based on the rate of progress in the currently running step(s).
root: INFO: 2018-12-02T07:31:19.964Z: JOB_MESSAGE_BASIC: Worker pool stopped.
root: INFO: 2018-12-02T07:31:19.988Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
root: INFO: Job 2018-12-01_22_29_27-868325378819143934 is in state JOB_STATE_FAILED
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 18 tests in 5235.854s
FAILED (errors=1)
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_04_19-5595966427716484827?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_04_18-9634657184535844677?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_04_20-16884776109961813286?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_16_43-10452707213595096261?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_22_06-7613789009157284764?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_27_39-3459536486386733455?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_04_18-13571678968442446455?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_04_18-1980446780551430351?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_10_50-12694933266070196027?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_17_28-3835627698892079191?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_22_50-10266317565774430123?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_29_27-868325378819143934?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_04_17-18377250108890791670?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_10_28-7329307758668748582?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_18_11-5529856063371398851?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_23_55-3395583769245792811?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_04_18-7887776766304353098?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_10_55-9676014128485638824?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_17_23-9226788979904284066?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_04_18-15790591395323581510?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-01_22_12_00-537327980471455726?project=apache-beam-testing.
> Task :beam-sdks-python:postCommitIT FAILED
:beam-sdks-python:postCommitIT (Thread[Task worker for ':',5,main]) completed. Took 1 hrs 27 mins 16.686 secs.
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build.gradle'> line: 331
* What went wrong:
Execution failed for task ':beam-sdks-python:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 1h 31m 19s
6 actionable tasks: 6 executed
Publishing build scan...
https://gradle.com/s/yfsjyggziramw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_Python_Verify
#6716
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/6716/display/redirect>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Verify #6715
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/6715/display/redirect>
------------------------------------------
[...truncated 313.95 KB...]
test_1 | DEBUG Uploading 1 files using 1 thread(s).
test_1 | DEBUG Uploading 'kinglear.txt' to '/kinglear.txt'.
test_1 | INFO Writing to '/kinglear.txt'.
test_1 | DEBUG Resolved path '/kinglear.txt' to '/kinglear.txt'.
test_1 | DEBUG http://namenode:50070 "PUT /webhdfs/v1/kinglear.txt?user.name=root&overwrite=True&op=CREATE HTTP/1.1" 307 0
test_1 | DEBUG Starting new HTTP connection (1): datanode:50075
datanode_1 | 18/12/02 12:03:19 INFO datanode.webhdfs: 172.18.0.4 PUT /webhdfs/v1/kinglear.txt?op=CREATE&user.name=root&namenoderpcaddress=namenode:8020&createflag=&createparent=true&overwrite=true&user.name=root 201
namenode_1 | 18/12/02 12:03:19 INFO hdfs.StateChange: BLOCK* allocate blk_1073741825_1001, replicas=172.18.0.3:50010 for /kinglear.txt
datanode_1 | 18/12/02 12:03:19 INFO datanode.DataNode: Receiving BP-227273154-172.18.0.2-1543752150311:blk_1073741825_1001 src: /172.18.0.3:39470 dest: /172.18.0.3:50010
datanode_1 | 18/12/02 12:03:19 INFO DataNode.clienttrace: src: /172.18.0.3:39470, dest: /172.18.0.3:50010, bytes: 157283, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1668177278_67, offset: 0, srvID: 371bc449-e1e9-4269-b6d2-152360cd3f13, blockid: BP-227273154-172.18.0.2-1543752150311:blk_1073741825_1001, duration: 13604626
datanode_1 | 18/12/02 12:03:19 INFO datanode.DataNode: PacketResponder: BP-227273154-172.18.0.2-1543752150311:blk_1073741825_1001, type=LAST_IN_PIPELINE terminating
namenode_1 | 18/12/02 12:03:19 INFO namenode.FSNamesystem: BLOCK* blk_1073741825_1001 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /kinglear.txt
namenode_1 | 18/12/02 12:03:19 INFO namenode.EditLogFileOutputStream: Nothing to flush
namenode_1 | 18/12/02 12:03:19 INFO hdfs.StateChange: DIR* completeFile: /kinglear.txt is closed by DFSClient_NONMAPREDUCE_-1668177278_67
test_1 | DEBUG Upload of 'kinglear.txt' to '/kinglear.txt' complete.
test_1 | INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
test_1 | INFO:root:==================== <function annotate_downstream_side_inputs at 0x7f1f52170b90> ====================
test_1 | INFO:root:==================== <function fix_side_input_pcoll_coders at 0x7f1f521707d0> ====================
test_1 | INFO:root:==================== <function lift_combiners at 0x7f1f52170c08> ====================
test_1 | INFO:root:==================== <function expand_gbk at 0x7f1f52170d70> ====================
test_1 | INFO:root:==================== <function sink_flattens at 0x7f1f521706e0> ====================
test_1 | INFO:root:==================== <function greedily_fuse at 0x7f1f52170aa0> ====================
test_1 | INFO:root:==================== <function impulse_to_input at 0x7f1f52170758> ====================
test_1 | INFO:root:==================== <function inject_timer_pcollections at 0x7f1f52170140> ====================
test_1 | INFO:root:==================== <function sort_stages at 0x7f1f52170cf8> ====================
test_1 | INFO:root:Running (((ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read_16)+(ref_AppliedPTransform_write/Write/WriteImpl/InitializeWrite_17))+(ref_PCollection_PCollection_9/Write))+(ref_PCollection_PCollection_10/Write)
test_1 | INFO:root:Running (ref_AppliedPTransform_read/Read_3)+((ref_AppliedPTransform_split_4)+((ref_AppliedPTransform_pair_with_one_5)+(group/Write)))
datanode_1 | 18/12/02 12:03:21 INFO datanode.webhdfs: 172.18.0.4 GET /webhdfs/v1/kinglear.txt?op=OPEN&user.name=root&namenoderpcaddress=namenode:8020&length=157284&offset=0 200
test_1 | INFO:root:Running (((group/Read)+((ref_AppliedPTransform_count_10)+(ref_AppliedPTransform_format_11)))+(ref_AppliedPTransform_write/Write/WriteImpl/WriteBundles_18))+((ref_AppliedPTransform_write/Write/WriteImpl/Pair_19)+((ref_AppliedPTransform_write/Write/WriteImpl/WindowInto(WindowIntoFn)_20)+(write/Write/WriteImpl/GroupByKey/Write)))
test_1 | WARNING:root:Mime types are not supported. Got non-default mime_type: text/plain
datanode_1 | 18/12/02 12:03:23 INFO datanode.webhdfs: 172.18.0.4 PUT /webhdfs/v1/beam-temp-py-wordcount-integration-4333d51cf62a11e897060242ac120004/6556938b-eb21-48e8-baa4-b4c8d7c7b992.py-wordcount-integration?op=CREATE&user.name=root&namenoderpcaddress=namenode:8020&createflag=&createparent=true&overwrite=false&user.name=root 201
namenode_1 | 18/12/02 12:03:23 INFO hdfs.StateChange: BLOCK* allocate blk_1073741826_1002, replicas=172.18.0.3:50010 for /beam-temp-py-wordcount-integration-4333d51cf62a11e897060242ac120004/6556938b-eb21-48e8-baa4-b4c8d7c7b992.py-wordcount-integration
datanode_1 | 18/12/02 12:03:23 INFO datanode.DataNode: Receiving BP-227273154-172.18.0.2-1543752150311:blk_1073741826_1002 src: /172.18.0.3:39488 dest: /172.18.0.3:50010
datanode_1 | 18/12/02 12:03:23 INFO DataNode.clienttrace: src: /172.18.0.3:39488, dest: /172.18.0.3:50010, bytes: 48944, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-2068209323_69, offset: 0, srvID: 371bc449-e1e9-4269-b6d2-152360cd3f13, blockid: BP-227273154-172.18.0.2-1543752150311:blk_1073741826_1002, duration: 4229654
datanode_1 | 18/12/02 12:03:23 INFO datanode.DataNode: PacketResponder: BP-227273154-172.18.0.2-1543752150311:blk_1073741826_1002, type=LAST_IN_PIPELINE terminating
namenode_1 | 18/12/02 12:03:23 INFO hdfs.StateChange: DIR* completeFile: /beam-temp-py-wordcount-integration-4333d51cf62a11e897060242ac120004/6556938b-eb21-48e8-baa4-b4c8d7c7b992.py-wordcount-integration is closed by DFSClient_NONMAPREDUCE_-2068209323_69
test_1 | INFO:root:Running (write/Write/WriteImpl/GroupByKey/Read)+((ref_AppliedPTransform_write/Write/WriteImpl/Extract_25)+(ref_PCollection_PCollection_17/Write))
test_1 | INFO:root:Running ((ref_PCollection_PCollection_9/Read)+(ref_AppliedPTransform_write/Write/WriteImpl/PreFinalize_26))+(ref_PCollection_PCollection_18/Write)
test_1 | INFO:root:Running (ref_PCollection_PCollection_9/Read)+(ref_AppliedPTransform_write/Write/WriteImpl/FinalizeWrite_27)
test_1 | INFO:root:Starting finalize_write threads with num_shards: 1 (skipped: 0), batches: 1, num_threads: 1
test_1 | INFO:root:Renamed 1 shards in 0.14 seconds.
test_1 | INFO:root:number of empty lines: 1663
test_1 | INFO:root:average word length: 4
hdfs_it-jenkins-beam_postcommit_python_verify-6715_test_1 exited with code 0
Stopping hdfs_it-jenkins-beam_postcommit_python_verify-6715_datanode_1 ...
Stopping hdfs_it-jenkins-beam_postcommit_python_verify-6715_namenode_1 ...
Stopping hdfs_it-jenkins-beam_postcommit_python_verify-6715_datanode_1 ... done
Stopping hdfs_it-jenkins-beam_postcommit_python_verify-6715_namenode_1 ... done
Aborting on container exit...
real 1m16.817s
user 0m0.789s
sys 0m0.127s
+ finally
+ docker-compose -p hdfs_IT-jenkins-beam_PostCommit_Python_Verify-6715 --no-ansi down
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6715_test_1 ...
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6715_datanode_1 ...
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6715_namenode_1 ...
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6715_datanode_1 ... done
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6715_namenode_1 ... done
Removing hdfs_it-jenkins-beam_postcommit_python_verify-6715_test_1 ... done
Removing network hdfs_it-jenkins-beam_postcommit_python_verify-6715_test_net
real 0m0.490s
user 0m0.238s
sys 0m0.054s
:beam-sdks-python:hdfsIntegrationTest (Thread[Task worker for ':',5,main]) completed. Took 2 mins 37.84 secs.
:beam-sdks-python:postCommitIT (Thread[Task worker for ':',5,main]) started.
> Task :beam-sdks-python:postCommitIT
Caching disabled for task ':beam-sdks-python:postCommitIT': Caching has not been enabled for the task
Task ':beam-sdks-python:postCommitIT' is not up-to-date because:
Task has not declared any outputs despite executing actions.
Starting process 'command 'sh''. Working directory: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python> Command: sh -c . <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/bin/activate> && ./scripts/run_integration_test.sh --test_opts "--nocapture --processes=8 --process-timeout=4500 --attr=IT"
Successfully started process 'command 'sh''
###########################################################################
# Build pipeline options if not provided in --pipeline_opts from commandline
if [[ -z $PIPELINE_OPTS ]]; then
# Check that the script is running in a known directory.
if [[ $PWD != *sdks/python* ]]; then
echo 'Unable to locate Apache Beam Python SDK root directory'
exit 1
fi
# Go to the Apache Beam Python SDK root
if [[ "*sdks/python" != $PWD ]]; then
cd $(pwd | sed 's/sdks\/python.*/sdks\/python/')
fi
# Create a tarball if not exists
if [[ $(find ${SDK_LOCATION}) ]]; then
SDK_LOCATION=$(find ${SDK_LOCATION})
else
python setup.py -q sdist
SDK_LOCATION=$(find dist/apache-beam-*.tar.gz)
fi
# Install test dependencies for ValidatesRunner tests.
echo "pyhamcrest" > postcommit_requirements.txt
echo "mock" >> postcommit_requirements.txt
# Options used to run testing pipeline on Cloud Dataflow Service. Also used for
# running on DirectRunner (some options ignored).
opts=(
"--runner=$RUNNER"
"--project=$PROJECT"
"--staging_location=$GCS_LOCATION/staging-it"
"--temp_location=$GCS_LOCATION/temp-it"
"--output=$GCS_LOCATION/py-it-cloud/output"
"--sdk_location=$SDK_LOCATION"
"--requirements_file=postcommit_requirements.txt"
"--num_workers=$NUM_WORKERS"
"--sleep_secs=$SLEEP_SECS"
)
# Add --streaming if provided
if [[ "$STREAMING" = true ]]; then
opts+=("--streaming")
fi
# Add --dataflow_worker_jar if provided
if [[ ! -z "$WORKER_JAR" ]]; then
opts+=("--dataflow_worker_jar=$WORKER_JAR")
fi
PIPELINE_OPTS=$(IFS=" " ; echo "${opts[*]}")
fi
pwd | sed 's/sdks\/python.*/sdks\/python/'
find ${SDK_LOCATION}
find ${SDK_LOCATION}
IFS=" " ; echo "${opts[*]}"
>>> RUNNING integration tests with pipeline options: --runner=TestDataflowRunner --project=apache-beam-testing --staging_location=gs://temp-storage-for-end-to-end-tests/staging-it --temp_location=gs://temp-storage-for-end-to-end-tests/temp-it --output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output --sdk_location=build/apache-beam.tar.gz --requirements_file=postcommit_requirements.txt --num_workers=1 --sleep_secs=20
###########################################################################
# Run tests and validate that jobs finish successfully.
echo ">>> RUNNING integration tests with pipeline options: $PIPELINE_OPTS"
python setup.py nosetests \
--test-pipeline-options="$PIPELINE_OPTS" \
$TEST_OPTS
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/setuptools/dist.py>:470: UserWarning: Normalizing '2.10.0.dev' to '2.10.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
options = pbegin.pipeline.options.view_as(DebugOptions)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:815: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
standard_options = transform_node.inputs[0].pipeline.options.view_as(
test_bigquery_tornadoes_it (apache_beam.examples.cookbook.bigquery_tornadoes_it_test.BigqueryTornadoesIT) ... ok
test_datastore_wordcount_it (apache_beam.examples.cookbook.datastore_wordcount_it_test.DatastoreWordCountIT) ... ok
test_autocomplete_it (apache_beam.examples.complete.autocomplete_test.AutocompleteTest) ... ok
test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... ok
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... ok
test_streaming_wordcount_it (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) ... ok
test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... ok
test_game_stats_it (apache_beam.examples.complete.game.game_stats_it_test.GameStatsIT) ... ok
test_avro_it (apache_beam.examples.fastavro_it_test.FastavroIT) ... ok
test_hourly_team_score_it (apache_beam.examples.complete.game.hourly_team_score_it_test.HourlyTeamScoreIT) ... ok
test_user_score_it (apache_beam.examples.complete.game.user_score_it_test.UserScoreIT) ... ok
test_bigquery_read_1M_python (apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT) ... ok
test_streaming_data_only (apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest) ... ok
test_streaming_with_attributes (apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest) ... ok
test_big_query_legacy_sql (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_big_query_new_types (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_big_query_standard_sql (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) ... ok
test_datastore_write_limit (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT) ... ok
----------------------------------------------------------------------
XML: <https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 18 tests in 2597.663s
OK
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_03_52-15459518292800222051?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_09_51-14990497598809566411?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_17_05-12047948165489312959?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_22_42-9967785846124903870?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_28_33-11647350476159128186?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_34_25-12998227964286790741?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_40_21-4579573791788647854?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_03_55-6711203642834634968?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_03_53-4179344075887073925?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_03_54-3827488932797819996?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_16_10-2305130526678023377?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_21_43-9182805128770681180?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_27_41-710097622863450377?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_03_52-2145455455238185033?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_03_52-9683616704291240838?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_10_54-11560749523346217660?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_17_14-5141803343322940522?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_22_54-12400644511634404238?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_03_52-1016413887868528520?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_10_23-7192863111444307744?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_16_51-15373182830671608353?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_03_53-2448789964155217708?project=apache-beam-testing.
Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-02_04_11_19-1342769490851622244?project=apache-beam-testing.
:beam-sdks-python:postCommitIT (Thread[Task worker for ':',5,main]) completed. Took 43 mins 18.391 secs.
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build.gradle'> line: 221
* What went wrong:
Execution failed for task ':beam-sdks-python:directRunnerIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 46m 55s
6 actionable tasks: 6 executed
Publishing build scan...
https://gradle.com/s/luv63tzypicik
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org