You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2019/08/02 15:46:14 UTC
Build failed in Jenkins: beam_LoadTests_Python_Combine_Flink_Batch
#1
See <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/1/display/redirect>
------------------------------------------
[...truncated 133.39 KB...]
OK
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 40s
3 actionable tasks: 2 executed, 1 up-to-date
Publishing build scan...
https://gradle.com/s/eiwfsau76xhnw
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins4423295309452691493.sh
+ echo Changing number of workers to 6
Changing number of workers to 6
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins6293321116090028737.sh
+ gcloud dataproc clusters update beam-loadtests-python-combine-flink-batch-1 --num-workers=6 --quiet
Waiting on operation [projects/apache-beam-testing/regions/global/operations/49d1e105-0eb4-3d4e-8b51-ac09d2487b8e].
Waiting for cluster update operation...
.....WARNING: Cluster has active YARN containers. If any container is actively writing to HDFS then the downsize operation may block until all writers are stopped.
........................................................................................................................................................................................................................done.
Updated [https://dataproc.googleapis.com/v1/projects/apache-beam-testing/regions/global/clusters/beam-loadtests-python-combine-flink-batch-1].
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins2736429533717097451.sh
+ echo src Combine Python Load test: 2GB 10 byte records src
src Combine Python Load test: 2GB 10 byte records src
[Gradle] - Launching build.
[src] $ <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/gradlew> --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g -PloadTest.mainClass=apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally -Prunner=PortableRunner '-PloadTest.args=--job_name=load-tests-python-flink-batch-combine-1-0802150153 --project=apache-beam-testing --publish_to_big_query=true --metrics_dataset=load_test --metrics_table=python_flink_batch_combine_1 --input_options='{"num_records": 200000000,"key_size": 1,"value_size": 9}' --parallelism=5 --job_endpoint=localhost:8099 --environment_config=gcr.io/apache-beam-testing/beam_portability/python:latest --environment_type=DOCKER --top_count=20 --runner=PortableRunner' :sdks:python:apache_beam:testing:load_tests:run
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy UP-TO-DATE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources UP-TO-DATE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:spotlessGroovy UP-TO-DATE
> Task :buildSrc:spotlessGroovyCheck UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradle UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE
> Task :buildSrc:spotlessCheck UP-TO-DATE
> Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:validateTaskProperties UP-TO-DATE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE
Configuration on demand is an incubating feature.
> Task :sdks:python:apache_beam:testing:load_tests:setupVirtualenv UP-TO-DATE
> Task :sdks:python:apache_beam:testing:load_tests:installGcpTest
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Obtaining file://<https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/sdks/python>
Requirement already satisfied: crcmod<2.0,>=1.7 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (1.7)
Requirement already satisfied: dill<0.2.10,>=0.2.9 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.2.9)
Requirement already satisfied: fastavro<0.22,>=0.21.4 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.21.24)
Requirement already satisfied: future<1.0.0,>=0.16.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.16.0)
Requirement already satisfied: grpcio<2,>=1.8 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (1.22.0)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (2.5.8)
Requirement already satisfied: httplib2<=0.12.0,>=0.8 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.12.0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (2.0.0)
Requirement already satisfied: pymongo<4.0.0,>=3.8.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (3.8.0)
Requirement already satisfied: oauth2client<4,>=2.0.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (3.0.0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (3.9.0)
Requirement already satisfied: pydot<2,>=1.2.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (1.4.1)
Requirement already satisfied: python-dateutil<3,>=2.8.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (2.8.0)
Requirement already satisfied: pytz>=2018.3 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (2019.2)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (3.13)
Requirement already satisfied: avro<2.0.0,>=1.8.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (1.9.0)
Requirement already satisfied: futures<4.0.0,>=3.2.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (3.3.0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.6.8)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (3.6.6)
Requirement already satisfied: pyarrow<0.15.0,>=0.11.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.14.1)
Requirement already satisfied: cachetools<4,>=3.1.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (3.1.1)
Requirement already satisfied: google-apitools<0.5.29,>=0.5.28 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.5.28)
Requirement already satisfied: google-cloud-datastore<1.8.0,>=1.7.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (1.7.4)
Requirement already satisfied: google-cloud-pubsub<0.40.0,>=0.39.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.39.1)
Requirement already satisfied: google-cloud-bigquery<1.18.0,>=1.6.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (1.17.0)
Requirement already satisfied: google-cloud-core<2,>=0.28.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (1.0.3)
Requirement already satisfied: google-cloud-bigtable<0.33.0,>=0.31.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.32.2)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.90.4)
Requirement already satisfied: googledatastore<7.1,>=7.0.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (7.0.2)
Requirement already satisfied: nose>=1.3.7 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (1.3.7)
Requirement already satisfied: nose_xunitmp>=0.4.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.4.1)
Requirement already satisfied: numpy<2,>=1.14.3 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (1.16.4)
Requirement already satisfied: pandas<0.24,>=0.23.4 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.23.4)
Requirement already satisfied: parameterized<0.7.0,>=0.6.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (0.6.3)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (1.9.0)
Requirement already satisfied: tenacity<6.0,>=5.0.2 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from apache-beam==2.16.0.dev0) (5.0.4)
Requirement already satisfied: six>=1.5.2 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from grpcio<2,>=1.8->apache-beam==2.16.0.dev0) (1.12.0)
Requirement already satisfied: enum34>=1.0.4; python_version < "3.4" in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from grpcio<2,>=1.8->apache-beam==2.16.0.dev0) (1.1.6)
Requirement already satisfied: docopt in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from hdfs<3.0.0,>=2.1.0->apache-beam==2.16.0.dev0) (0.6.2)
Requirement already satisfied: requests>=2.7.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from hdfs<3.0.0,>=2.1.0->apache-beam==2.16.0.dev0) (2.22.0)
Requirement already satisfied: pbr>=0.11 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from mock<3.0.0,>=1.0.1->apache-beam==2.16.0.dev0) (5.4.1)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from mock<3.0.0,>=1.0.1->apache-beam==2.16.0.dev0) (1.0.2)
Requirement already satisfied: pyasn1>=0.1.7 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from oauth2client<4,>=2.0.1->apache-beam==2.16.0.dev0) (0.4.6)
Requirement already satisfied: pyasn1-modules>=0.0.5 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from oauth2client<4,>=2.0.1->apache-beam==2.16.0.dev0) (0.2.6)
Requirement already satisfied: rsa>=3.1.4 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from oauth2client<4,>=2.0.1->apache-beam==2.16.0.dev0) (4.0)
Requirement already satisfied: setuptools in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from protobuf<4,>=3.5.0.post1->apache-beam==2.16.0.dev0) (41.0.1)
Requirement already satisfied: pyparsing>=2.1.4 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from pydot<2,>=1.2.0->apache-beam==2.16.0.dev0) (2.4.2)
Requirement already satisfied: fasteners>=0.14 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from google-apitools<0.5.29,>=0.5.28->apache-beam==2.16.0.dev0) (0.15)
Requirement already satisfied: google-api-core[grpc]<2.0.0dev,>=1.6.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from google-cloud-datastore<1.8.0,>=1.7.1->apache-beam==2.16.0.dev0) (1.14.2)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.4 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from google-cloud-pubsub<0.40.0,>=0.39.0->apache-beam==2.16.0.dev0) (0.11.4)
Requirement already satisfied: google-resumable-media>=0.3.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from google-cloud-bigquery<1.18.0,>=1.6.0->apache-beam==2.16.0.dev0) (0.3.2)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.16.0.dev0) (1.6.0)
Requirement already satisfied: monotonic>=0.6; python_version == "2.7" in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from tenacity<6.0,>=5.0.2->apache-beam==2.16.0.dev0) (1.5)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.16.0.dev0) (1.25.3)
Requirement already satisfied: certifi>=2017.4.17 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.16.0.dev0) (2019.6.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.16.0.dev0) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.16.0.dev0) (2.8)
Requirement already satisfied: google-auth<2.0dev,>=0.4.0 in <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages> (from google-api-core[grpc]<2.0.0dev,>=1.6.0->google-cloud-datastore<1.8.0,>=1.7.1->apache-beam==2.16.0.dev0) (1.6.3)
Installing collected packages: apache-beam
Found existing installation: apache-beam 2.16.0.dev0
Not uninstalling apache-beam at <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/sdks/python,> outside environment <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227>
Can't uninstall 'apache-beam'. No files were found to uninstall.
Running setup.py develop for apache-beam
Successfully installed apache-beam
> Task :sdks:python:apache_beam:testing:load_tests:run
<https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/build/gradleenv/1329484227/local/lib/python2.7/site-packages/setuptools/dist.py>:472: UserWarning: Normalizing '2.16.0.dev' to '2.16.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
testCombineGlobally (apache_beam.testing.load_tests.combine_test.CombineTest) ... ERROR
======================================================================
ERROR: testCombineGlobally (apache_beam.testing.load_tests.combine_test.CombineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 70, in tearDown
result = self.pipeline.run()
File "<https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/sdks/python/apache_beam/testing/test_pipeline.py",> line 109, in run
state = result.wait_until_finish()
File "<https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py",> line 446, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline load-tests-python-flink-batch-combine-1-0802150153_f43d8b75-b1d4-4d42-9a03-b250a8a6cd9a failed in state FAILED: java.net.UnknownHostException: beam-loadtests-python-combine-flink-batch-1-w-10.c.apache-beam-testing.internal
-------------------- >> begin captured logging << --------------------
root: INFO: Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
root: INFO: Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 176
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): www.googleapis.com:443
urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/load_test HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/load_test/tables/python_flink_batch_combine_1 HTTP/1.1" 404 None
urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/load_test/tables HTTP/1.1" 200 None
root: INFO: ==================== <function lift_combiners at 0x7f1a0a367de8> ====================
root: DEBUG: 21 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
root: DEBUG: Stages: ['ref_AppliedPTransform_Read/Impulse_3\n Read/Impulse:beam:transform:impulse:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Split_4\n Read/Split:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Reshuffle/AddRandomKeys_6\n Read/Reshuffle/AddRandomKeys:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Reshuffle/ReshufflePerKey/Map(reify_timestamps)_8\n Read/Reshuffle/ReshufflePerKey/Map(reify_timestamps):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Reshuffle/ReshufflePerKey/GroupByKey_9\n Read/Reshuffle/ReshufflePerKey/GroupByKey:beam:transform:group_by_key:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)_13\n Read/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Reshuffle/RemoveRandomKeys_14\n Read/Reshuffle/RemoveRandomKeys:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/ReadSplits_15\n Read/ReadSplits:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Measure time: Start_16\n Measure time: Start:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/KeyWithVoid_18\n Combine with Top 0/KeyWithVoid:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'Combine with Top 0/CombinePerKey/Precombine\n Combine with Top 0/CombinePerKey/Precombine:beam:transform:combine_per_key_precombine:v1\n must follow: \n downstream_side_inputs: <unknown>', 'Combine with Top 0/CombinePerKey/Group\n Combine with Top 0/CombinePerKey/Group:beam:transform:group_by_key:v1\n must follow: \n downstream_side_inputs: <unknown>', 'Combine with Top 0/CombinePerKey/Merge\n Combine with Top 0/CombinePerKey/Merge:beam:transform:combine_per_key_merge_accumulators:v1\n must follow: \n downstream_side_inputs: <unknown>', 'Combine with Top 0/CombinePerKey/ExtractOutputs\n Combine with Top 0/CombinePerKey/ExtractOutputs:beam:transform:combine_per_key_extract_outputs:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/UnKey_26\n Combine with Top 0/UnKey:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/DoOnce/Impulse_28\n Combine with Top 0/DoOnce/Impulse:beam:transform:impulse:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/DoOnce/FlatMap(<lambda at core.py:2415>)_29\n Combine with Top 0/DoOnce/FlatMap(<lambda at core.py:2415>):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/DoOnce/Map(decode)_31\n Combine with Top 0/DoOnce/Map(decode):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/InjectDefault_32\n Combine with Top 0/InjectDefault:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Consume 0_33\n Consume 0:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Measure time: End 0_34\n Measure time: End 0:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>']
root: INFO: ==================== <function expand_sdf at 0x7f1a0a367e60> ====================
root: DEBUG: 21 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
root: DEBUG: Stages: ['ref_AppliedPTransform_Read/Impulse_3\n Read/Impulse:beam:transform:impulse:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Split_4\n Read/Split:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Reshuffle/AddRandomKeys_6\n Read/Reshuffle/AddRandomKeys:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Reshuffle/ReshufflePerKey/Map(reify_timestamps)_8\n Read/Reshuffle/ReshufflePerKey/Map(reify_timestamps):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Reshuffle/ReshufflePerKey/GroupByKey_9\n Read/Reshuffle/ReshufflePerKey/GroupByKey:beam:transform:group_by_key:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)_13\n Read/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/Reshuffle/RemoveRandomKeys_14\n Read/Reshuffle/RemoveRandomKeys:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Read/ReadSplits_15\n Read/ReadSplits:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Measure time: Start_16\n Measure time: Start:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/KeyWithVoid_18\n Combine with Top 0/KeyWithVoid:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'Combine with Top 0/CombinePerKey/Precombine\n Combine with Top 0/CombinePerKey/Precombine:beam:transform:combine_per_key_precombine:v1\n must follow: \n downstream_side_inputs: <unknown>', 'Combine with Top 0/CombinePerKey/Group\n Combine with Top 0/CombinePerKey/Group:beam:transform:group_by_key:v1\n must follow: \n downstream_side_inputs: <unknown>', 'Combine with Top 0/CombinePerKey/Merge\n Combine with Top 0/CombinePerKey/Merge:beam:transform:combine_per_key_merge_accumulators:v1\n must follow: \n downstream_side_inputs: <unknown>', 'Combine with Top 0/CombinePerKey/ExtractOutputs\n Combine with Top 0/CombinePerKey/ExtractOutputs:beam:transform:combine_per_key_extract_outputs:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/UnKey_26\n Combine with Top 0/UnKey:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/DoOnce/Impulse_28\n Combine with Top 0/DoOnce/Impulse:beam:transform:impulse:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/DoOnce/FlatMap(<lambda at core.py:2415>)_29\n Combine with Top 0/DoOnce/FlatMap(<lambda at core.py:2415>):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/DoOnce/Map(decode)_31\n Combine with Top 0/DoOnce/Map(decode):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Combine with Top 0/InjectDefault_32\n Combine with Top 0/InjectDefault:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Consume 0_33\n Consume 0:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Measure time: End 0_34\n Measure time: End 0:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>']
root: DEBUG: Runner option 'job_name' was already added
root: DEBUG: Runner option 'runner' was already added
root: DEBUG: Runner option 'temp_location' was already added
root: DEBUG: Runner option 'streaming' was already added
root: DEBUG: Runner option 'dataflow_kms_key' was already added
root: DEBUG: Runner option 'enable_streaming_engine' was already added
root: DEBUG: Runner option 'project' was already added
root: DEBUG: Runner option 'zone' was already added
root: DEBUG: Runner option 'environment_cache_millis' was already added
root: DEBUG: Runner option 'job_endpoint' was already added
root: DEBUG: Runner option 'sdk_worker_parallelism' was already added
root: DEBUG: Runner option 'files_to_stage' was already added
root: WARNING: Discarding unparseable args: ['--publish_to_big_query=true', '--metrics_dataset=load_test', '--metrics_table=python_flink_batch_combine_1', '--input_options={"num_records": 200000000,"key_size": 1,"value_size": 9}', '--top_count=20']
root: INFO: Job state changed to RUNNING
root: DEBUG: java.net.UnknownHostException: beam-loadtests-python-combine-flink-batch-1-w-10.c.apache-beam-testing.internal
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at java.net.InetAddress.getByName(InetAddress.java:1077)
at org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils.getRpcUrl(AkkaRpcServiceUtils.java:167)
at org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils.getRpcUrl(AkkaRpcServiceUtils.java:133)
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:89)
at org.apache.flink.client.program.ClusterClient.<init>(ClusterClient.java:159)
at org.apache.flink.client.program.rest.RestClusterClient.<init>(RestClusterClient.java:185)
at org.apache.flink.client.program.rest.RestClusterClient.<init>(RestClusterClient.java:158)
at org.apache.flink.client.RemoteExecutor.start(RemoteExecutor.java:152)
at org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:202)
at org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:187)
at org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:173)
at org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator$BatchTranslationContext.execute(FlinkBatchPortablePipelineTranslator.java:200)
at org.apache.beam.runners.flink.FlinkPipelineRunner.runPipelineWithTranslator(FlinkPipelineRunner.java:87)
at org.apache.beam.runners.flink.FlinkPipelineRunner.run(FlinkPipelineRunner.java:63)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:74)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
root: ERROR: java.net.UnknownHostException: beam-loadtests-python-combine-flink-batch-1-w-10.c.apache-beam-testing.internal
root: INFO: Job state changed to FAILED
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 1.429s
FAILED (errors=1)
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 49
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 5s
3 actionable tasks: 2 executed, 1 up-to-date
Publishing build scan...
https://gradle.com/s/yxb2qthca3q2w
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal :
beam_LoadTests_Python_Combine_Flink_Batch #5
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/5/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_Combine_Flink_Batch
#4
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/4/display/redirect?page=changes>
Changes:
[github] Update design-documents.md
------------------------------------------
[...truncated 59.57 KB...]
> Task :model:pipeline:shadowJar
> Task :model:job-management:shadowJar
> Task :model:fn-execution:shadowJar
> Task :sdks:java:core:compileJava FROM-CACHE
> Task :sdks:java:core:classes
> Task :sdks:java:core:shadowJar
> Task :vendor:sdks-java-extensions-protobuf:compileJava FROM-CACHE
> Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE
> Task :vendor:sdks-java-extensions-protobuf:shadowJar
> Task :sdks:java:extensions:google-cloud-platform-core:compileJava FROM-CACHE
> Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE
> Task :runners:core-construction-java:compileJava FROM-CACHE
> Task :runners:core-construction-java:classes UP-TO-DATE
> Task :sdks:java:fn-execution:compileJava FROM-CACHE
> Task :sdks:java:fn-execution:classes UP-TO-DATE
> Task :sdks:java:io:kafka:compileJava FROM-CACHE
> Task :sdks:java:io:kafka:classes UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:jar
> Task :sdks:java:fn-execution:jar
> Task :sdks:java:io:kafka:jar
> Task :runners:core-construction-java:jar
> Task :runners:core-java:compileJava FROM-CACHE
> Task :runners:core-java:classes UP-TO-DATE
> Task :runners:core-java:jar
> Task :sdks:java:harness:compileJava FROM-CACHE
> Task :sdks:java:harness:classes UP-TO-DATE
> Task :sdks:java:harness:jar
> Task :sdks:java:harness:shadowJar
> Task :runners:java-fn-execution:compileJava FROM-CACHE
> Task :runners:java-fn-execution:classes UP-TO-DATE
> Task :runners:java-fn-execution:jar
> Task :runners:flink:1.7:compileJava FROM-CACHE
> Task :runners:flink:1.7:classes
> Task :runners:flink:1.7:jar
> Task :runners:flink:1.7:job-server:compileJava NO-SOURCE
> Task :runners:flink:1.7:job-server:classes UP-TO-DATE
> Task :runners:flink:1.7:job-server:shadowJar
> Task :runners:flink:1.7:job-server-container:copyDockerfileDependencies
> Task :runners:flink:1.7:job-server-container:dockerPrepare
> Task :runners:flink:1.7:job-server-container:docker
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 59s
51 actionable tasks: 35 executed, 15 from cache, 1 up-to-date
Publishing build scan...
https://gradle.com/s/6ag5tiqdfaxnm
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins4379933766351899324.sh
+ echo 'Tagging Flink Job Server'\''s image...'
Tagging Flink Job Server's image...
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins3981421185585683550.sh
+ docker tag gcr.io/apache-beam-testing/beam_portability/flink-job-server gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins2931929007549980876.sh
+ echo 'Pushing Flink Job Server'\''s image...'
Pushing Flink Job Server's image...
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins7229094595603842267.sh
+ docker push gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
The push refers to repository [gcr.io/apache-beam-testing/beam_portability/flink-job-server]
b48bfbff76f5: Preparing
b9f7d68b3db3: Preparing
cee72abf8080: Preparing
b575bb71cbb6: Preparing
7955da51da82: Preparing
c64652873162: Preparing
a4e797bc3f15: Preparing
392f356944ff: Preparing
15210a41d4ee: Preparing
e2a8a00a83b2: Preparing
a4e797bc3f15: Waiting
15210a41d4ee: Waiting
392f356944ff: Waiting
c64652873162: Waiting
e2a8a00a83b2: Waiting
7955da51da82: Layer already exists
b575bb71cbb6: Layer already exists
c64652873162: Layer already exists
a4e797bc3f15: Layer already exists
392f356944ff: Layer already exists
15210a41d4ee: Layer already exists
e2a8a00a83b2: Layer already exists
b48bfbff76f5: Pushed
cee72abf8080: Pushed
b9f7d68b3db3: Pushed
latest: digest: sha256:70de3f729a5924135735986ee053415c11d8b6eb0b63b369ae0a7b8985e2a11c size: 2426
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties content
JOB_SERVER_IMAGE=gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
CLUSTER_NAME=beam-loadtests-python-combine-flink-batch-4
DETACHED_MODE=true
HARNESS_IMAGES_TO_PULL=gcr.io/apache-beam-testing/beam_portability/python:latest
FLINK_NUM_WORKERS=16
FLINK_DOWNLOAD_URL=https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz
GCS_BUCKET=gs://beam-flink-cluster
FLINK_TASKMANAGER_SLOTS=1
ARTIFACTS_DIR=gs://beam-flink-cluster/beam-loadtests-python-combine-flink-batch-4
GCLOUD_ZONE=us-central1-a
[EnvInject] - Variables injected successfully.
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins6367970492431917079.sh
+ echo Setting up flink cluster
Setting up flink cluster
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins4501622146011895229.sh
+ cd <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/.test-infra/dataproc/>
+ ./create_flink_cluster.sh
+ GCLOUD_ZONE=us-central1-a
+ DATAPROC_VERSION=1.2
+ MASTER_NAME=beam-loadtests-python-combine-flink-batch-4-m
+ INIT_ACTIONS_FOLDER_NAME=init-actions
+ FLINK_INIT=gs://beam-flink-cluster/init-actions/flink.sh
+ BEAM_INIT=gs://beam-flink-cluster/init-actions/beam.sh
+ DOCKER_INIT=gs://beam-flink-cluster/init-actions/docker.sh
+ FLINK_LOCAL_PORT=8081
+ FLINK_TASKMANAGER_SLOTS=1
+ TASK_MANAGER_MEM=10240
+ YARN_APPLICATION_MASTER=
+ main
+ upload_init_actions
+ echo 'Uploading initialization actions to GCS bucket: gs://beam-flink-cluster'
Uploading initialization actions to GCS bucket: gs://beam-flink-cluster
+ gsutil cp -r init-actions/beam.sh init-actions/docker.sh init-actions/flink.sh gs://beam-flink-cluster/init-actions
Copying file://init-actions/beam.sh [Content-Type=text/x-sh]...
/ [0 files][ 0.0 B/ 2.3 KiB] / [1 files][ 2.3 KiB/ 2.3 KiB] Copying file://init-actions/docker.sh [Content-Type=text/x-sh]...
/ [1 files][ 2.3 KiB/ 6.0 KiB] / [2 files][ 6.0 KiB/ 6.0 KiB] Copying file://init-actions/flink.sh [Content-Type=text/x-sh]...
/ [2 files][ 6.0 KiB/ 13.1 KiB] / [3 files][ 13.1 KiB/ 13.1 KiB] -
Operation completed over 3 objects/13.1 KiB.
+ create_cluster
+ local metadata=flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz,
+ metadata+=flink-start-yarn-session=true,
+ metadata+=flink-taskmanager-slots=1
+ [[ -n gcr.io/apache-beam-testing/beam_portability/python:latest ]]
+ metadata+=,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/python:latest
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest ]]
+ metadata+=,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
+ local image_version=1.2
+ echo 'Starting dataproc cluster. Dataproc version: 1.2'
Starting dataproc cluster. Dataproc version: 1.2
+ local num_dataproc_workers=17
+ gcloud dataproc clusters create beam-loadtests-python-combine-flink-batch-4 --num-workers=17 --initialization-actions gs://beam-flink-cluster/init-actions/docker.sh,gs://beam-flink-cluster/init-actions/beam.sh,gs://beam-flink-cluster/init-actions/flink.sh --metadata flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz,flink-start-yarn-session=true,flink-taskmanager-slots=1,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/python:latest,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest, --image-version=1.2 --zone=us-central1-a --quiet
Waiting on operation [projects/apache-beam-testing/regions/global/operations/a1fdf5e3-b9e4-392f-86fc-3e9fbf3c5651].
Waiting for cluster creation operation...
WARNING: For PD-Standard without local SSDs, we strongly recommend provisioning 1TB or larger to ensure consistently high I/O performance. See https://cloud.google.com/compute/docs/disks/performance for information on disk I/O performance.
.................................................................................................................................................................................................................................................done.
Created [https://dataproc.googleapis.com/v1/projects/apache-beam-testing/regions/global/clusters/beam-loadtests-python-combine-flink-batch-4] Cluster placed in zone [us-central1-a].
+ get_leader
+ local i=0
+ local application_ids
+ local application_masters
+ echo 'Yarn Applications'
Yarn Applications
++ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-4-m '--command=yarn application -list'
++ grep beam-loadtests-python-combine-flink-batch-4
Warning: Permanently added 'compute.8111246912682062362' (ECDSA) to the list of known hosts.
19/08/05 15:40:25 INFO client.RMProxy: Connecting to ResourceManager at beam-loadtests-python-combine-flink-batch-4-m/10.128.0.89:8032
+ read line
+ echo application_1565019509688_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195
application_1565019509688_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195
++ echo application_1565019509688_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195
++ sed 's/ .*//'
+ application_ids[$i]=application_1565019509688_0001
++ echo application_1565019509688_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195
++ sed 's/.*beam-loadtests-python-combine-flink-batch-4/beam-loadtests-python-combine-flink-batch-4/'
++ sed 's/ .*//'
+ application_masters[$i]=beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195
+ i=1
+ read line
+ '[' 1 '!=' 1 ']'
+ YARN_APPLICATION_MASTER=beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195
+ echo 'Using Yarn Application master: beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195'
Using Yarn Application master: beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest ]]
+ start_job_server
+ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-4-m '--command=sudo --user yarn docker run --detach --publish 8099:8099 --publish 8098:8098 --publish 8097:8097 --volume ~/.config/gcloud:/root/.config/gcloud gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest --flink-master-url=beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195 --artifacts-dir=gs://beam-flink-cluster/beam-loadtests-python-combine-flink-batch-4'
36f69e9aa02b7d574f6d1451b72dbe0e25ceddf5d03d6b6cb6d0e55ab7a5bf3d
+ start_tunnel
++ gcloud compute ssh --quiet --zone=us-central1-a yarn@beam-loadtests-python-combine-flink-batch-4-m '--command=curl -s "http://beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195/jobmanager/config"'
+ local 'job_server_config=[{"key":"web.port","value":"0"},{"key":"high-availability.cluster-id","value":"application_1565019509688_0001"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"web.tmpdir","value":"/tmp/flink-web-27eaec62-ebfa-43ba-accf-fc1d96d62de4"},{"key":"jobmanager.rpc.port","value":"34733"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"rest.port","value":"0"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1565019509688_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"16"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]'
+ local key=jobmanager.rpc.port
++ echo beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195
++ cut -d : -f1
+ local yarn_application_master_host=beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal
++ python -c 'import sys, json; print [ e['\''value'\''] for e in json.load(sys.stdin) if e['\''key'\''] == u'\''jobmanager.rpc.port'\''][0]'
++ echo '[{"key":"web.port","value":"0"},{"key":"high-availability.cluster-id","value":"application_1565019509688_0001"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"web.tmpdir","value":"/tmp/flink-web-27eaec62-ebfa-43ba-accf-fc1d96d62de4"},{"key":"jobmanager.rpc.port","value":"34733"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"rest.port","value":"0"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1565019509688_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"16"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]'
+ local jobmanager_rpc_port=34733
++ [[ true == \t\r\u\e ]]
++ echo ' -Nf >& /dev/null'
+ local 'detached_mode_params= -Nf >& /dev/null'
++ [[ -n gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest ]]
++ echo '-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097'
+ local 'job_server_ports_forwarding=-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097'
+ local 'tunnel_command=gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-4-m -- -L 8081:beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195 -L 34733:beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34733 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf >& /dev/null'
+ eval gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-4-m -- -L 8081:beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195 -L 34733:beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34733 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf '>&' /dev/null
++ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-4-m -- -L 8081:beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34195 -L 34733:beam-loadtests-python-combine-flink-batch-4-w-11.c.apache-beam-testing.internal:34733 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins7784045844282754161.sh
+ echo src Combine Python Load test: 2GB Fanout 4 src
src Combine Python Load test: 2GB Fanout 4 src
[Gradle] - Launching build.
[src] $ <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/gradlew> --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g -PloadTest.mainClass=apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally -Prunner=PortableRunner '-PloadTest.args=--job_name=load-tests-python-flink-batch-combine-4-0805150140 --project=apache-beam-testing --publish_to_big_query=true --metrics_dataset=load_test --metrics_table=python_flink_batch_combine_4 --input_options='{"num_records": 5000000,"key_size": 10,"value_size": 90}' --parallelism=16 --job_endpoint=localhost:8099 --environment_config=gcr.io/apache-beam-testing/beam_portability/python:latest --environment_type=DOCKER --fanout=4 --top_count=20 --runner=PortableRunner' :sdks:python:apache_beam:testing:load_tests:run
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy UP-TO-DATE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources UP-TO-DATE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:spotlessGroovy UP-TO-DATE
> Task :buildSrc:spotlessGroovyCheck UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradle UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE
> Task :buildSrc:spotlessCheck UP-TO-DATE
> Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:validateTaskProperties UP-TO-DATE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE
Configuration on demand is an incubating feature.
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 44
* What went wrong:
A problem occurred evaluating project ':sdks:python:apache_beam:testing:load_tests'.
> Could not get unknown property 'files' for task ':sdks:python:apache_beam:testing:load_tests:run' of type org.gradle.api.tasks.Exec.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1s
Publishing build scan...
https://gradle.com/s/x7mt5toikvffg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_Combine_Flink_Batch
#3
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/3/display/redirect>
------------------------------------------
[...truncated 59.57 KB...]
> Task :model:pipeline:shadowJar
> Task :model:job-management:shadowJar
> Task :model:fn-execution:shadowJar
> Task :sdks:java:core:compileJava FROM-CACHE
> Task :sdks:java:core:classes
> Task :sdks:java:core:shadowJar
> Task :vendor:sdks-java-extensions-protobuf:compileJava FROM-CACHE
> Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:compileJava FROM-CACHE
> Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE
> Task :vendor:sdks-java-extensions-protobuf:shadowJar
> Task :sdks:java:extensions:google-cloud-platform-core:jar
> Task :sdks:java:io:kafka:compileJava FROM-CACHE
> Task :sdks:java:io:kafka:classes UP-TO-DATE
> Task :runners:core-construction-java:compileJava FROM-CACHE
> Task :runners:core-construction-java:classes UP-TO-DATE
> Task :sdks:java:io:kafka:jar
> Task :sdks:java:fn-execution:compileJava FROM-CACHE
> Task :sdks:java:fn-execution:classes UP-TO-DATE
> Task :sdks:java:fn-execution:jar
> Task :runners:core-construction-java:jar
> Task :runners:core-java:compileJava FROM-CACHE
> Task :runners:core-java:classes UP-TO-DATE
> Task :runners:core-java:jar
> Task :sdks:java:harness:compileJava FROM-CACHE
> Task :sdks:java:harness:classes UP-TO-DATE
> Task :sdks:java:harness:jar
> Task :sdks:java:harness:shadowJar
> Task :runners:java-fn-execution:compileJava FROM-CACHE
> Task :runners:java-fn-execution:classes UP-TO-DATE
> Task :runners:java-fn-execution:jar
> Task :runners:flink:1.7:compileJava FROM-CACHE
> Task :runners:flink:1.7:classes
> Task :runners:flink:1.7:jar
> Task :runners:flink:1.7:job-server:compileJava NO-SOURCE
> Task :runners:flink:1.7:job-server:classes UP-TO-DATE
> Task :runners:flink:1.7:job-server:shadowJar
> Task :runners:flink:1.7:job-server-container:copyDockerfileDependencies
> Task :runners:flink:1.7:job-server-container:dockerPrepare
> Task :runners:flink:1.7:job-server-container:docker
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 54s
51 actionable tasks: 35 executed, 15 from cache, 1 up-to-date
Publishing build scan...
https://gradle.com/s/reatfrbgfxx76
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins1034311204291355357.sh
+ echo 'Tagging Flink Job Server'\''s image...'
Tagging Flink Job Server's image...
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins3340861697000850210.sh
+ docker tag gcr.io/apache-beam-testing/beam_portability/flink-job-server gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins3523142666276261565.sh
+ echo 'Pushing Flink Job Server'\''s image...'
Pushing Flink Job Server's image...
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins4711868129174197409.sh
+ docker push gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
The push refers to repository [gcr.io/apache-beam-testing/beam_portability/flink-job-server]
9e186053f681: Preparing
1f8e270426e6: Preparing
f8290fd7980d: Preparing
b575bb71cbb6: Preparing
7955da51da82: Preparing
c64652873162: Preparing
a4e797bc3f15: Preparing
392f356944ff: Preparing
15210a41d4ee: Preparing
e2a8a00a83b2: Preparing
c64652873162: Waiting
a4e797bc3f15: Waiting
392f356944ff: Waiting
15210a41d4ee: Waiting
e2a8a00a83b2: Waiting
7955da51da82: Layer already exists
b575bb71cbb6: Layer already exists
c64652873162: Layer already exists
a4e797bc3f15: Layer already exists
392f356944ff: Layer already exists
15210a41d4ee: Layer already exists
e2a8a00a83b2: Layer already exists
9e186053f681: Pushed
f8290fd7980d: Pushed
1f8e270426e6: Pushed
latest: digest: sha256:bd4681d0d31cb00f5c9249286617a596643576b0c24d2cfaf546e8156954a7b5 size: 2426
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties content
JOB_SERVER_IMAGE=gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
CLUSTER_NAME=beam-loadtests-python-combine-flink-batch-3
DETACHED_MODE=true
HARNESS_IMAGES_TO_PULL=gcr.io/apache-beam-testing/beam_portability/python:latest
FLINK_NUM_WORKERS=16
FLINK_DOWNLOAD_URL=https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz
GCS_BUCKET=gs://beam-flink-cluster
FLINK_TASKMANAGER_SLOTS=1
ARTIFACTS_DIR=gs://beam-flink-cluster/beam-loadtests-python-combine-flink-batch-3
GCLOUD_ZONE=us-central1-a
[EnvInject] - Variables injected successfully.
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins625636766282560839.sh
+ echo Setting up flink cluster
Setting up flink cluster
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins605748809702234222.sh
+ cd <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/.test-infra/dataproc/>
+ ./create_flink_cluster.sh
+ GCLOUD_ZONE=us-central1-a
+ DATAPROC_VERSION=1.2
+ MASTER_NAME=beam-loadtests-python-combine-flink-batch-3-m
+ INIT_ACTIONS_FOLDER_NAME=init-actions
+ FLINK_INIT=gs://beam-flink-cluster/init-actions/flink.sh
+ BEAM_INIT=gs://beam-flink-cluster/init-actions/beam.sh
+ DOCKER_INIT=gs://beam-flink-cluster/init-actions/docker.sh
+ FLINK_LOCAL_PORT=8081
+ FLINK_TASKMANAGER_SLOTS=1
+ TASK_MANAGER_MEM=10240
+ YARN_APPLICATION_MASTER=
+ main
+ upload_init_actions
+ echo 'Uploading initialization actions to GCS bucket: gs://beam-flink-cluster'
Uploading initialization actions to GCS bucket: gs://beam-flink-cluster
+ gsutil cp -r init-actions/beam.sh init-actions/docker.sh init-actions/flink.sh gs://beam-flink-cluster/init-actions
Copying file://init-actions/beam.sh [Content-Type=text/x-sh]...
/ [0 files][ 0.0 B/ 2.3 KiB] / [1 files][ 2.3 KiB/ 2.3 KiB] Copying file://init-actions/docker.sh [Content-Type=text/x-sh]...
/ [1 files][ 2.3 KiB/ 6.0 KiB] / [2 files][ 6.0 KiB/ 6.0 KiB] Copying file://init-actions/flink.sh [Content-Type=text/x-sh]...
/ [2 files][ 6.0 KiB/ 13.1 KiB] / [3 files][ 13.1 KiB/ 13.1 KiB] -
Operation completed over 3 objects/13.1 KiB.
+ create_cluster
+ local metadata=flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz,
+ metadata+=flink-start-yarn-session=true,
+ metadata+=flink-taskmanager-slots=1
+ [[ -n gcr.io/apache-beam-testing/beam_portability/python:latest ]]
+ metadata+=,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/python:latest
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest ]]
+ metadata+=,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
+ local image_version=1.2
+ echo 'Starting dataproc cluster. Dataproc version: 1.2'
Starting dataproc cluster. Dataproc version: 1.2
+ local num_dataproc_workers=17
+ gcloud dataproc clusters create beam-loadtests-python-combine-flink-batch-3 --num-workers=17 --initialization-actions gs://beam-flink-cluster/init-actions/docker.sh,gs://beam-flink-cluster/init-actions/beam.sh,gs://beam-flink-cluster/init-actions/flink.sh --metadata flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz,flink-start-yarn-session=true,flink-taskmanager-slots=1,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/python:latest,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest, --image-version=1.2 --zone=us-central1-a --quiet
Waiting on operation [projects/apache-beam-testing/regions/global/operations/635be986-0edb-3bb7-afd0-d54be719e2a8].
Waiting for cluster creation operation...
WARNING: For PD-Standard without local SSDs, we strongly recommend provisioning 1TB or larger to ensure consistently high I/O performance. See https://cloud.google.com/compute/docs/disks/performance for information on disk I/O performance.
........................................................................................................................................................................................................................done.
Created [https://dataproc.googleapis.com/v1/projects/apache-beam-testing/regions/global/clusters/beam-loadtests-python-combine-flink-batch-3] Cluster placed in zone [us-central1-a].
+ get_leader
+ local i=0
+ local application_ids
+ local application_masters
+ echo 'Yarn Applications'
Yarn Applications
++ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-3-m '--command=yarn application -list'
++ grep beam-loadtests-python-combine-flink-batch-3
Warning: Permanently added 'compute.6314017186246247812' (ECDSA) to the list of known hosts.
19/08/04 15:39:48 INFO client.RMProxy: Connecting to ResourceManager at beam-loadtests-python-combine-flink-batch-3-m/10.128.0.97:8032
+ read line
+ echo application_1564933097097_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 0% http://beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125
application_1564933097097_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 0% http://beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125
++ echo application_1564933097097_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 0% http://beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125
++ sed 's/ .*//'
+ application_ids[$i]=application_1564933097097_0001
++ sed 's/.*beam-loadtests-python-combine-flink-batch-3/beam-loadtests-python-combine-flink-batch-3/'
++ echo application_1564933097097_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 0% http://beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125
++ sed 's/ .*//'
+ application_masters[$i]=beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125
+ i=1
+ read line
+ '[' 1 '!=' 1 ']'
+ YARN_APPLICATION_MASTER=beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125
+ echo 'Using Yarn Application master: beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125'
Using Yarn Application master: beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest ]]
+ start_job_server
+ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-3-m '--command=sudo --user yarn docker run --detach --publish 8099:8099 --publish 8098:8098 --publish 8097:8097 --volume ~/.config/gcloud:/root/.config/gcloud gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest --flink-master-url=beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125 --artifacts-dir=gs://beam-flink-cluster/beam-loadtests-python-combine-flink-batch-3'
a073fa83afe477b13a2dfcd95079001ba399fad64af8f39fe5ba9c5ee1342e6f
+ start_tunnel
++ gcloud compute ssh --quiet --zone=us-central1-a yarn@beam-loadtests-python-combine-flink-batch-3-m '--command=curl -s "http://beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125/jobmanager/config"'
+ local 'job_server_config=[{"key":"web.port","value":"0"},{"key":"high-availability.cluster-id","value":"application_1564933097097_0001"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"web.tmpdir","value":"/tmp/flink-web-ab7dfd80-39cb-4df2-8f81-afaed20cd70a"},{"key":"jobmanager.rpc.port","value":"44451"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"rest.port","value":"0"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1564933097097_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"16"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]'
+ local key=jobmanager.rpc.port
++ echo beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125
++ cut -d : -f1
+ local yarn_application_master_host=beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal
++ echo '[{"key":"web.port","value":"0"},{"key":"high-availability.cluster-id","value":"application_1564933097097_0001"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"web.tmpdir","value":"/tmp/flink-web-ab7dfd80-39cb-4df2-8f81-afaed20cd70a"},{"key":"jobmanager.rpc.port","value":"44451"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"rest.port","value":"0"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1564933097097_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"16"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]'
++ python -c 'import sys, json; print [ e['\''value'\''] for e in json.load(sys.stdin) if e['\''key'\''] == u'\''jobmanager.rpc.port'\''][0]'
+ local jobmanager_rpc_port=44451
++ [[ true == \t\r\u\e ]]
++ echo ' -Nf >& /dev/null'
+ local 'detached_mode_params= -Nf >& /dev/null'
++ [[ -n gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest ]]
++ echo '-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097'
+ local 'job_server_ports_forwarding=-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097'
+ local 'tunnel_command=gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-3-m -- -L 8081:beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125 -L 44451:beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:44451 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf >& /dev/null'
+ eval gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-3-m -- -L 8081:beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125 -L 44451:beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:44451 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf '>&' /dev/null
++ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-3-m -- -L 8081:beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:35125 -L 44451:beam-loadtests-python-combine-flink-batch-3-w-4.c.apache-beam-testing.internal:44451 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins8779099992150462489.sh
+ echo src Combine Python Load test: 2GB Fanout 4 src
src Combine Python Load test: 2GB Fanout 4 src
[Gradle] - Launching build.
[src] $ <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/gradlew> --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g -PloadTest.mainClass=apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally -Prunner=PortableRunner '-PloadTest.args=--job_name=load-tests-python-flink-batch-combine-4-0804150139 --project=apache-beam-testing --publish_to_big_query=true --metrics_dataset=load_test --metrics_table=python_flink_batch_combine_4 --input_options='{"num_records": 5000000,"key_size": 10,"value_size": 90}' --parallelism=16 --job_endpoint=localhost:8099 --environment_config=gcr.io/apache-beam-testing/beam_portability/python:latest --environment_type=DOCKER --fanout=4 --top_count=20 --runner=PortableRunner' :sdks:python:apache_beam:testing:load_tests:run
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy UP-TO-DATE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources UP-TO-DATE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:spotlessGroovy UP-TO-DATE
> Task :buildSrc:spotlessGroovyCheck UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradle UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE
> Task :buildSrc:spotlessCheck UP-TO-DATE
> Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:validateTaskProperties UP-TO-DATE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE
Configuration on demand is an incubating feature.
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 44
* What went wrong:
A problem occurred evaluating project ':sdks:python:apache_beam:testing:load_tests'.
> Could not get unknown property 'files' for task ':sdks:python:apache_beam:testing:load_tests:run' of type org.gradle.api.tasks.Exec.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 0s
Publishing build scan...
https://gradle.com/s/ukoeee72dcj5i
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_Combine_Flink_Batch
#2
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/2/display/redirect?page=changes>
Changes:
[github] Retry Datastore writes on [Errno 32] Broken pipe
[github] Update Python dependencies page for 2.14.0
[michal.walenia] [BEAM-7079] Add Chicago Taxi Example runner script with Python scripts
[ankurgoenka] [BEAM-7868] optionally skip hidden pipeline options
[michal.walenia] [BEAM-7079] Add Chicago Taxi Example gradle task and Jenkins job
[ehudm] [BEAM-7814] Fix BigqueryMatcher._matches when called more than once.
[michal.walenia] [BEAM-7869] Add load testing jobs to README.md
[alireza4263] [BEAM-7844] Implementing NodeStat Estimations for all the nodes
[pabloem] Fix minor typos (#9192)
[udim] [BEAM-7577] Allow ValueProviders in Datastore Query filters (#8950)
[heejong] [BEAM-7877] Change the log level when deleting unknown temoprary files
[github] [BEAM-7060] Introduce Python3-only test modules (#9223)
------------------------------------------
[...truncated 59.57 KB...]
> Task :model:pipeline:shadowJar
> Task :model:job-management:shadowJar
> Task :model:fn-execution:shadowJar
> Task :sdks:java:core:compileJava FROM-CACHE
> Task :sdks:java:core:classes
> Task :sdks:java:core:shadowJar
> Task :runners:core-construction-java:compileJava FROM-CACHE
> Task :runners:core-construction-java:classes UP-TO-DATE
> Task :vendor:sdks-java-extensions-protobuf:compileJava FROM-CACHE
> Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:compileJava FROM-CACHE
> Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE
> Task :sdks:java:io:kafka:compileJava FROM-CACHE
> Task :sdks:java:io:kafka:classes UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:jar
> Task :vendor:sdks-java-extensions-protobuf:shadowJar
> Task :sdks:java:io:kafka:jar
> Task :runners:core-construction-java:jar
> Task :sdks:java:fn-execution:compileJava FROM-CACHE
> Task :sdks:java:fn-execution:classes UP-TO-DATE
> Task :sdks:java:fn-execution:jar
> Task :runners:core-java:compileJava FROM-CACHE
> Task :runners:core-java:classes UP-TO-DATE
> Task :runners:core-java:jar
> Task :sdks:java:harness:compileJava FROM-CACHE
> Task :sdks:java:harness:classes UP-TO-DATE
> Task :sdks:java:harness:jar
> Task :sdks:java:harness:shadowJar
> Task :runners:java-fn-execution:compileJava FROM-CACHE
> Task :runners:java-fn-execution:classes UP-TO-DATE
> Task :runners:java-fn-execution:jar
> Task :runners:flink:1.7:compileJava FROM-CACHE
> Task :runners:flink:1.7:classes
> Task :runners:flink:1.7:jar
> Task :runners:flink:1.7:job-server:compileJava NO-SOURCE
> Task :runners:flink:1.7:job-server:classes UP-TO-DATE
> Task :runners:flink:1.7:job-server:shadowJar
> Task :runners:flink:1.7:job-server-container:copyDockerfileDependencies
> Task :runners:flink:1.7:job-server-container:dockerPrepare
> Task :runners:flink:1.7:job-server-container:docker
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 1m 3s
51 actionable tasks: 35 executed, 15 from cache, 1 up-to-date
Publishing build scan...
https://gradle.com/s/osh257qrit762
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins4368884718617325786.sh
+ echo 'Tagging Flink Job Server'\''s image...'
Tagging Flink Job Server's image...
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins372089533284393080.sh
+ docker tag gcr.io/apache-beam-testing/beam_portability/flink-job-server gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins2817232568276999693.sh
+ echo 'Pushing Flink Job Server'\''s image...'
Pushing Flink Job Server's image...
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins1862354513236147827.sh
+ docker push gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
The push refers to repository [gcr.io/apache-beam-testing/beam_portability/flink-job-server]
099a716387b4: Preparing
663ddeb48b9d: Preparing
8fd4beffea61: Preparing
b575bb71cbb6: Preparing
7955da51da82: Preparing
c64652873162: Preparing
a4e797bc3f15: Preparing
392f356944ff: Preparing
15210a41d4ee: Preparing
e2a8a00a83b2: Preparing
392f356944ff: Waiting
15210a41d4ee: Waiting
e2a8a00a83b2: Waiting
c64652873162: Waiting
a4e797bc3f15: Waiting
7955da51da82: Layer already exists
b575bb71cbb6: Layer already exists
a4e797bc3f15: Layer already exists
c64652873162: Layer already exists
392f356944ff: Layer already exists
15210a41d4ee: Layer already exists
e2a8a00a83b2: Layer already exists
099a716387b4: Pushed
8fd4beffea61: Pushed
663ddeb48b9d: Pushed
latest: digest: sha256:592077e08aa588ac39f274c44e984e4e4b1db1a07df1792625311973f6118977 size: 2426
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties content
JOB_SERVER_IMAGE=gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
CLUSTER_NAME=beam-loadtests-python-combine-flink-batch-2
DETACHED_MODE=true
HARNESS_IMAGES_TO_PULL=gcr.io/apache-beam-testing/beam_portability/python:latest
FLINK_NUM_WORKERS=16
FLINK_DOWNLOAD_URL=https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz
GCS_BUCKET=gs://beam-flink-cluster
FLINK_TASKMANAGER_SLOTS=1
ARTIFACTS_DIR=gs://beam-flink-cluster/beam-loadtests-python-combine-flink-batch-2
GCLOUD_ZONE=us-central1-a
[EnvInject] - Variables injected successfully.
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins5706354185471744364.sh
+ echo Setting up flink cluster
Setting up flink cluster
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins1891823339946988347.sh
+ cd <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/.test-infra/dataproc/>
+ ./create_flink_cluster.sh
+ GCLOUD_ZONE=us-central1-a
+ DATAPROC_VERSION=1.2
+ MASTER_NAME=beam-loadtests-python-combine-flink-batch-2-m
+ INIT_ACTIONS_FOLDER_NAME=init-actions
+ FLINK_INIT=gs://beam-flink-cluster/init-actions/flink.sh
+ BEAM_INIT=gs://beam-flink-cluster/init-actions/beam.sh
+ DOCKER_INIT=gs://beam-flink-cluster/init-actions/docker.sh
+ FLINK_LOCAL_PORT=8081
+ FLINK_TASKMANAGER_SLOTS=1
+ TASK_MANAGER_MEM=10240
+ YARN_APPLICATION_MASTER=
+ main
+ upload_init_actions
+ echo 'Uploading initialization actions to GCS bucket: gs://beam-flink-cluster'
Uploading initialization actions to GCS bucket: gs://beam-flink-cluster
+ gsutil cp -r init-actions/beam.sh init-actions/docker.sh init-actions/flink.sh gs://beam-flink-cluster/init-actions
Copying file://init-actions/beam.sh [Content-Type=text/x-sh]...
/ [0 files][ 0.0 B/ 2.3 KiB] / [1 files][ 2.3 KiB/ 2.3 KiB] Copying file://init-actions/docker.sh [Content-Type=text/x-sh]...
/ [1 files][ 2.3 KiB/ 6.0 KiB] / [2 files][ 6.0 KiB/ 6.0 KiB] Copying file://init-actions/flink.sh [Content-Type=text/x-sh]...
/ [2 files][ 6.0 KiB/ 13.1 KiB] / [3 files][ 13.1 KiB/ 13.1 KiB] -
Operation completed over 3 objects/13.1 KiB.
+ create_cluster
+ local metadata=flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz,
+ metadata+=flink-start-yarn-session=true,
+ metadata+=flink-taskmanager-slots=1
+ [[ -n gcr.io/apache-beam-testing/beam_portability/python:latest ]]
+ metadata+=,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/python:latest
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest ]]
+ metadata+=,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest
+ local image_version=1.2
+ echo 'Starting dataproc cluster. Dataproc version: 1.2'
Starting dataproc cluster. Dataproc version: 1.2
+ local num_dataproc_workers=17
+ gcloud dataproc clusters create beam-loadtests-python-combine-flink-batch-2 --num-workers=17 --initialization-actions gs://beam-flink-cluster/init-actions/docker.sh,gs://beam-flink-cluster/init-actions/beam.sh,gs://beam-flink-cluster/init-actions/flink.sh --metadata flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz,flink-start-yarn-session=true,flink-taskmanager-slots=1,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/python:latest,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest, --image-version=1.2 --zone=us-central1-a --quiet
Waiting on operation [projects/apache-beam-testing/regions/global/operations/d0f6a20a-d860-38ae-9b4b-5e9a36809902].
Waiting for cluster creation operation...
WARNING: For PD-Standard without local SSDs, we strongly recommend provisioning 1TB or larger to ensure consistently high I/O performance. See https://cloud.google.com/compute/docs/disks/performance for information on disk I/O performance.
........................................................................................................................................................................................................................done.
Created [https://dataproc.googleapis.com/v1/projects/apache-beam-testing/regions/global/clusters/beam-loadtests-python-combine-flink-batch-2] Cluster placed in zone [us-central1-a].
+ get_leader
+ local i=0
+ local application_ids
+ local application_masters
+ echo 'Yarn Applications'
Yarn Applications
++ grep beam-loadtests-python-combine-flink-batch-2
++ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-2-m '--command=yarn application -list'
Warning: Permanently added 'compute.349777332730860750' (ECDSA) to the list of known hosts.
19/08/03 15:40:44 INFO client.RMProxy: Connecting to ResourceManager at beam-loadtests-python-combine-flink-batch-2-m/10.128.0.72:8032
+ read line
+ echo application_1564846749583_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561
application_1564846749583_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561
++ echo application_1564846749583_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561
++ sed 's/ .*//'
+ application_ids[$i]=application_1564846749583_0001
++ echo application_1564846749583_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561
++ sed 's/.*beam-loadtests-python-combine-flink-batch-2/beam-loadtests-python-combine-flink-batch-2/'
++ sed 's/ .*//'
+ application_masters[$i]=beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561
+ i=1
+ read line
+ '[' 1 '!=' 1 ']'
+ YARN_APPLICATION_MASTER=beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561
+ echo 'Using Yarn Application master: beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561'
Using Yarn Application master: beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest ]]
+ start_job_server
+ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-2-m '--command=sudo --user yarn docker run --detach --publish 8099:8099 --publish 8098:8098 --publish 8097:8097 --volume ~/.config/gcloud:/root/.config/gcloud gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest --flink-master-url=beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561 --artifacts-dir=gs://beam-flink-cluster/beam-loadtests-python-combine-flink-batch-2'
b335dc4d5b2da27f7e7bd55de4b7bccdac8d18a21294615aa88324603b3ad0e4
+ start_tunnel
++ gcloud compute ssh --quiet --zone=us-central1-a yarn@beam-loadtests-python-combine-flink-batch-2-m '--command=curl -s "http://beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561/jobmanager/config"'
+ local 'job_server_config=[{"key":"web.port","value":"0"},{"key":"high-availability.cluster-id","value":"application_1564846749583_0001"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"web.tmpdir","value":"/tmp/flink-web-7ac4201e-a831-485f-bd5d-983e9f3213e2"},{"key":"jobmanager.rpc.port","value":"34461"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"rest.port","value":"0"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1564846749583_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"16"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]'
+ local key=jobmanager.rpc.port
++ echo beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561
++ cut -d : -f1
+ local yarn_application_master_host=beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal
++ echo '[{"key":"web.port","value":"0"},{"key":"high-availability.cluster-id","value":"application_1564846749583_0001"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"web.tmpdir","value":"/tmp/flink-web-7ac4201e-a831-485f-bd5d-983e9f3213e2"},{"key":"jobmanager.rpc.port","value":"34461"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"rest.port","value":"0"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1564846749583_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"16"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]'
++ python -c 'import sys, json; print [ e['\''value'\''] for e in json.load(sys.stdin) if e['\''key'\''] == u'\''jobmanager.rpc.port'\''][0]'
+ local jobmanager_rpc_port=34461
++ [[ true == \t\r\u\e ]]
++ echo ' -Nf >& /dev/null'
+ local 'detached_mode_params= -Nf >& /dev/null'
++ [[ -n gcr.io/apache-beam-testing/beam_portability/flink-job-server:latest ]]
++ echo '-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097'
+ local 'job_server_ports_forwarding=-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097'
+ local 'tunnel_command=gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-2-m -- -L 8081:beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561 -L 34461:beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:34461 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf >& /dev/null'
+ eval gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-2-m -- -L 8081:beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561 -L 34461:beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:34461 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf '>&' /dev/null
++ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-combine-flink-batch-2-m -- -L 8081:beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:38561 -L 34461:beam-loadtests-python-combine-flink-batch-2-w-0.c.apache-beam-testing.internal:34461 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf
[beam_LoadTests_Python_Combine_Flink_Batch] $ /bin/bash -xe /tmp/jenkins3741157637774281000.sh
+ echo src Combine Python Load test: 2GB Fanout 4 src
src Combine Python Load test: 2GB Fanout 4 src
[Gradle] - Launching build.
[src] $ <https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/gradlew> --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g -PloadTest.mainClass=apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally -Prunner=PortableRunner '-PloadTest.args=--job_name=load-tests-python-flink-batch-combine-4-0803150138 --project=apache-beam-testing --publish_to_big_query=true --metrics_dataset=load_test --metrics_table=python_flink_batch_combine_4 --input_options='{"num_records": 5000000,"key_size": 10,"value_size": 90}' --parallelism=16 --job_endpoint=localhost:8099 --environment_config=gcr.io/apache-beam-testing/beam_portability/python:latest --environment_type=DOCKER --fanout=4 --top_count=20 --runner=PortableRunner' :sdks:python:apache_beam:testing:load_tests:run
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy UP-TO-DATE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources UP-TO-DATE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:spotlessGroovy UP-TO-DATE
> Task :buildSrc:spotlessGroovyCheck UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradle UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE
> Task :buildSrc:spotlessCheck UP-TO-DATE
> Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:validateTaskProperties UP-TO-DATE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE
Configuration on demand is an incubating feature.
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 44
* What went wrong:
A problem occurred evaluating project ':sdks:python:apache_beam:testing:load_tests'.
> Could not get unknown property 'files' for task ':sdks:python:apache_beam:testing:load_tests:run' of type org.gradle.api.tasks.Exec.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 2s
Publishing build scan...
https://gradle.com/s/mj6duz56ageng
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org