You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2019/05/16 00:56:31 UTC
Build failed in Jenkins:
beam_PerformanceTests_ManyFiles_TextIOIT_HDFS #543
See <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/543/display/redirect?page=changes>
Changes:
[daniel.o.programmer] [BEAM-7154] Updating Go SDK errors (Part 3)
[iemejia] Categorize tests from UsesTestStream to UsesTestStreamWithProcessingTime
[huangry] Create PAssert util function for late panes.
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on apache-beam-jenkins-4 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 194576262c41f2502189c52565e2a91ca2204562 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 194576262c41f2502189c52565e2a91ca2204562
Commit message: "[BEAM-7331] Create PAssert util function for late panes."
> git rev-list --no-walk 0294039234f636cce77346b74ba58b4fe16b296c # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
SPARK_LOCAL_IP=127.0.0.1
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins5789323531129367359.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a --verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: [--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format: "default"
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2399133897939325455.sh
+ cp /home/jenkins/.kube/config <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-543>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins4479858691655249012.sh
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-543> create namespace beam-performancetests-manyfiles-textioit-hdfs-543
namespace/beam-performancetests-manyfiles-textioit-hdfs-543 created
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins7892460324870529108.sh
++ kubectl config current-context
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-543> config set-context gke_apache-beam-testing_us-central1-a_io-datastores --namespace=beam-performancetests-manyfiles-textioit-hdfs-543
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins7212959124612878081.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8705268532662449018.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins84949179212842908.sh
+ virtualenv <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env> --python=python2.7
New python executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python2.7>
Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python>
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins5255952175863200059.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install --upgrade setuptools pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (41.0.1)
Requirement already up-to-date: pip in ./env/.perfkit_env/lib/python2.7/site-packages (19.1.1)
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins7108631896585526209.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
Cloning into '<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker'...>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3705642489597620173.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting absl-py (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Collecting jinja2>=2.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 16)) (41.0.1)
Collecting colorlog[windows]==2.6.0 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 18))
Collecting futures>=3.0.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 19))
Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting PyYAML==3.12 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 20))
Collecting pint>=0.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/15/9d/bf177ebbc57d25e9e296addc14a1303d1e34d7964af5df428a8332349c42/Pint-0.9-py2.py3-none-any.whl
Collecting numpy==1.13.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 22))
Using cached https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting functools32 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 23))
Collecting contextlib2>=0.5.1 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 24))
Using cached https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Collecting timeout-decorator (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 26))
Collecting enum34; python_version < "3.4" (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting six (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2>=2.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/fb/40/f3adb7cf24a8012813c5edb20329eb22d5d8e2a0ecf73d21d6b85865da11/MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl
Collecting funcsigs; python_version == "2.7" (from pint>=0.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Collecting requests>=2.9.1 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting xmltodict (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl
Collecting cryptography>=1.3 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/c3/c1/cf8665c955c9393e9ff0872ba6cd3dc6f46ef915e94afcf6e0410508ca69/cryptography-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/d5/3d/1c54e92f62bbc747a638da94adb439f99dc2d2f3041fe41a06b0da4f2808/ntlm_auth-1.3.0-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/01/11/525b02e4acc0c747de8b6ccdab376331597c569c42ea66ab0a1dbd36eca2/urllib3-1.24.3-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/8d/e9/0c8afd1579e5cf7bc0f06fbcd7cdb954cbc0baadd505973949a99337da1c/cffi-1.12.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Installing collected packages: enum34, six, absl-py, MarkupSafe, jinja2, colorama, colorlog, blinker, futures, PyYAML, funcsigs, pint, numpy, functools32, contextlib2, certifi, chardet, idna, urllib3, requests, pycparser, cffi, asn1crypto, ipaddress, cryptography, ntlm-auth, requests-ntlm, xmltodict, pywinrm, timeout-decorator
Successfully installed MarkupSafe-1.1.1 PyYAML-3.12 absl-py-0.7.1 asn1crypto-0.24.0 blinker-1.4 certifi-2019.3.9 cffi-1.12.3 chardet-3.0.4 colorama-0.4.1 colorlog-2.6.0 contextlib2-0.5.5 cryptography-2.6.1 enum34-1.1.6 funcsigs-1.0.2 functools32-3.2.3.post2 futures-3.2.0 idna-2.8 ipaddress-1.0.22 jinja2-2.10.1 ntlm-auth-1.3.0 numpy-1.13.3 pint-0.9 pycparser-2.19 pywinrm-0.3.0 requests-2.21.0 requests-ntlm-1.1.0 six-1.12.0 timeout-decorator-0.4.1 urllib3-1.24.3 xmltodict-0.12.0
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3839881153377571453.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python> <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/pkb.py> --project=apache-beam-testing --dpb_log_level=INFO --bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results --k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/> --beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src> --official=true --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-543> --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 --beam_prebuilt=false --beam_sdk=java --beam_it_module=sdks/java/io/file-based-io-tests --beam_it_class=org.apache.beam.sdk.io.text.TextIOIT '--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]' '--beam_extra_properties=[filesystem=hdfs]' --beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml> --beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
2019-05-16 00:56:30,632 8c3a5e23 MainThread INFO Verbose logging to: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/8c3a5e23/pkb.log>
2019-05-16 00:56:30,633 8c3a5e23 MainThread INFO PerfKitBenchmarker version: v1.12.0-1183-g0bd23a1
2019-05-16 00:56:30,634 8c3a5e23 MainThread INFO Flag values:
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_it_module=sdks/java/io/file-based-io-tests
--beam_sdk=java
--k8s_get_wait_interval=10
--bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results
--temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]
--nobeam_prebuilt
--kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-543>
--beam_extra_properties=[filesystem=hdfs]
--project=apache-beam-testing
--beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
--official
--dpb_log_level=INFO
--beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml>
--beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src>
--k8s_get_retry_count=36
--benchmarks=beam_integration_benchmark
2019-05-16 00:56:30,814 8c3a5e23 MainThread INFO Setting --max_concurrent_threads=200.
2019-05-16 00:56:30,836 8c3a5e23 MainThread WARNING The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2019-05-16 00:56:30,858 8c3a5e23 MainThread beam_integration_benchmark(1/1) INFO Provisioning resources for benchmark beam_integration_benchmark
2019-05-16 00:56:30,858 8c3a5e23 MainThread beam_integration_benchmark(1/1) ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-16 00:56:30,860 8c3a5e23 MainThread beam_integration_benchmark(1/1) ERROR Exception running benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 903, in RunBenchmarkTask
RunBenchmark(spec, collector)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-16 00:56:30,861 8c3a5e23 MainThread beam_integration_benchmark(1/1) ERROR Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue.
2019-05-16 00:56:30,861 8c3a5e23 MainThread beam_integration_benchmark(1/1) INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-05-16 00:56:30,861 8c3a5e23 MainThread beam_integration_benchmark(1/1) INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/8c3a5e23/pkb.log>
2019-05-16 00:56:30,861 8c3a5e23 MainThread beam_integration_benchmark(1/1) INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/8c3a5e23/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal :
beam_PerformanceTests_ManyFiles_TextIOIT_HDFS #551
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/551/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_ManyFiles_TextIOIT_HDFS #550
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/550/display/redirect?page=changes>
Changes:
[25622840+adude3141] [BEAM-7302] ensure artifactId set on pom also with unconfigured project
[github] Fix a typo in setting.gradle
[markliu] [BEAM-7347] Pass dpb_service_zone flag to benchmarks to fix breakage
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on apache-beam-jenkins-1 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 15deadff871e55676a5f708fee95b49d0688af83 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 15deadff871e55676a5f708fee95b49d0688af83
Commit message: "Merge pull request #8602 from markflyhigh/fix-perf"
> git rev-list --no-walk d77474d16734cf63095765cbced97ecd4b016b54 # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
SPARK_LOCAL_IP=127.0.0.1
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3001246559067269337.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a --verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: [--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format: "default"
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3604787104986072938.sh
+ cp /home/jenkins/.kube/config <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-550>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins1909369963371471843.sh
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-550> create namespace beam-performancetests-manyfiles-textioit-hdfs-550
namespace/beam-performancetests-manyfiles-textioit-hdfs-550 created
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins258102300456289062.sh
++ kubectl config current-context
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-550> config set-context gke_apache-beam-testing_us-central1-a_io-datastores --namespace=beam-performancetests-manyfiles-textioit-hdfs-550
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins9099213701729977358.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2144499359018905516.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins4295057757194592254.sh
+ virtualenv <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env> --python=python2.7
New python executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python2.7>
Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python>
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins5951804982147610417.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install --upgrade setuptools pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (41.0.1)
Requirement already up-to-date: pip in ./env/.perfkit_env/lib/python2.7/site-packages (19.1.1)
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins7140093588005853610.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
Cloning into '<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker'...>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins1404689839663241124.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting absl-py (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Collecting jinja2>=2.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 16)) (41.0.1)
Collecting colorlog[windows]==2.6.0 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 18))
Collecting futures>=3.0.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 19))
Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting PyYAML==3.12 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 20))
Collecting pint>=0.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/15/9d/bf177ebbc57d25e9e296addc14a1303d1e34d7964af5df428a8332349c42/Pint-0.9-py2.py3-none-any.whl
Collecting numpy==1.13.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 22))
Using cached https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting functools32 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 23))
Collecting contextlib2>=0.5.1 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 24))
Using cached https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Collecting timeout-decorator (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 26))
Collecting enum34; python_version < "3.4" (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting six (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2>=2.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/fb/40/f3adb7cf24a8012813c5edb20329eb22d5d8e2a0ecf73d21d6b85865da11/MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl
Collecting funcsigs; python_version == "2.7" (from pint>=0.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Collecting requests>=2.9.1 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting xmltodict (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl
Collecting cryptography>=1.3 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/c3/c1/cf8665c955c9393e9ff0872ba6cd3dc6f46ef915e94afcf6e0410508ca69/cryptography-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/d5/3d/1c54e92f62bbc747a638da94adb439f99dc2d2f3041fe41a06b0da4f2808/ntlm_auth-1.3.0-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/39/ec/d93dfc69617a028915df914339ef66936ea976ef24fa62940fd86ba0326e/urllib3-1.25.2-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/8d/e9/0c8afd1579e5cf7bc0f06fbcd7cdb954cbc0baadd505973949a99337da1c/cffi-1.12.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Installing collected packages: enum34, six, absl-py, MarkupSafe, jinja2, colorama, colorlog, blinker, futures, PyYAML, funcsigs, pint, numpy, functools32, contextlib2, urllib3, certifi, chardet, idna, requests, pycparser, cffi, asn1crypto, ipaddress, cryptography, ntlm-auth, requests-ntlm, xmltodict, pywinrm, timeout-decorator
Successfully installed MarkupSafe-1.1.1 PyYAML-3.12 absl-py-0.7.1 asn1crypto-0.24.0 blinker-1.4 certifi-2019.3.9 cffi-1.12.3 chardet-3.0.4 colorama-0.4.1 colorlog-2.6.0 contextlib2-0.5.5 cryptography-2.6.1 enum34-1.1.6 funcsigs-1.0.2 functools32-3.2.3.post2 futures-3.2.0 idna-2.8 ipaddress-1.0.22 jinja2-2.10.1 ntlm-auth-1.3.0 numpy-1.13.3 pint-0.9 pycparser-2.19 pywinrm-0.3.0 requests-2.22.0 requests-ntlm-1.1.0 six-1.12.0 timeout-decorator-0.4.1 urllib3-1.25.2 xmltodict-0.12.0
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8138700056655761922.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python> <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/pkb.py> --project=apache-beam-testing --dpb_log_level=INFO --bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results --k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/> --beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src> --official=true --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-550> --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 --beam_prebuilt=false --beam_sdk=java --beam_it_module=sdks/java/io/file-based-io-tests --beam_it_class=org.apache.beam.sdk.io.text.TextIOIT '--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]' '--beam_extra_properties=[filesystem=hdfs]' --beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml> --beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
2019-05-17 18:57:11,347 8340bf59 MainThread INFO Verbose logging to: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/8340bf59/pkb.log>
2019-05-17 18:57:11,347 8340bf59 MainThread INFO PerfKitBenchmarker version: v1.12.0-1183-g0bd23a1
2019-05-17 18:57:11,348 8340bf59 MainThread INFO Flag values:
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_it_module=sdks/java/io/file-based-io-tests
--beam_sdk=java
--k8s_get_wait_interval=10
--bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results
--temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]
--nobeam_prebuilt
--kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-550>
--beam_extra_properties=[filesystem=hdfs]
--project=apache-beam-testing
--beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
--official
--dpb_log_level=INFO
--beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml>
--beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src>
--k8s_get_retry_count=36
--benchmarks=beam_integration_benchmark
2019-05-17 18:57:11,550 8340bf59 MainThread INFO Setting --max_concurrent_threads=200.
2019-05-17 18:57:11,572 8340bf59 MainThread WARNING The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2019-05-17 18:57:11,597 8340bf59 MainThread beam_integration_benchmark(1/1) INFO Provisioning resources for benchmark beam_integration_benchmark
2019-05-17 18:57:11,597 8340bf59 MainThread beam_integration_benchmark(1/1) ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-17 18:57:11,599 8340bf59 MainThread beam_integration_benchmark(1/1) ERROR Exception running benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 903, in RunBenchmarkTask
RunBenchmark(spec, collector)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-17 18:57:11,599 8340bf59 MainThread beam_integration_benchmark(1/1) ERROR Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue.
2019-05-17 18:57:11,600 8340bf59 MainThread beam_integration_benchmark(1/1) INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-05-17 18:57:11,600 8340bf59 MainThread beam_integration_benchmark(1/1) INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/8340bf59/pkb.log>
2019-05-17 18:57:11,600 8340bf59 MainThread beam_integration_benchmark(1/1) INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/8340bf59/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_ManyFiles_TextIOIT_HDFS #549
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/549/display/redirect>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on apache-beam-jenkins-6 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d77474d16734cf63095765cbced97ecd4b016b54 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f d77474d16734cf63095765cbced97ecd4b016b54
Commit message: "[BEAM-6908] Refactor Python performance test groovy file for easy configuration (#8518)"
> git rev-list --no-walk d77474d16734cf63095765cbced97ecd4b016b54 # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
SPARK_LOCAL_IP=127.0.0.1
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins4499312520848575877.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a --verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: [--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format: "default"
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2830398782925180357.sh
+ cp /home/jenkins/.kube/config <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-549>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2508037300219642441.sh
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-549> create namespace beam-performancetests-manyfiles-textioit-hdfs-549
namespace/beam-performancetests-manyfiles-textioit-hdfs-549 created
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2400927242144401973.sh
++ kubectl config current-context
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-549> config set-context gke_apache-beam-testing_us-central1-a_io-datastores --namespace=beam-performancetests-manyfiles-textioit-hdfs-549
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2544287572355816173.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3476956340250228199.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins983763782780504237.sh
+ virtualenv <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env> --python=python2.7
New python executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python2.7>
Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python>
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins829885755482066721.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install --upgrade setuptools pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (41.0.1)
Requirement already up-to-date: pip in ./env/.perfkit_env/lib/python2.7/site-packages (19.1.1)
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3391837599489930521.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
Cloning into '<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker'...>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins5184602781584508854.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting absl-py (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Collecting jinja2>=2.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 16)) (41.0.1)
Collecting colorlog[windows]==2.6.0 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 18))
Collecting futures>=3.0.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 19))
Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting PyYAML==3.12 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 20))
Collecting pint>=0.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/15/9d/bf177ebbc57d25e9e296addc14a1303d1e34d7964af5df428a8332349c42/Pint-0.9-py2.py3-none-any.whl
Collecting numpy==1.13.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 22))
Using cached https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting functools32 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 23))
Collecting contextlib2>=0.5.1 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 24))
Using cached https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Collecting timeout-decorator (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 26))
Collecting enum34; python_version < "3.4" (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting six (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2>=2.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/fb/40/f3adb7cf24a8012813c5edb20329eb22d5d8e2a0ecf73d21d6b85865da11/MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl
Collecting funcsigs; python_version == "2.7" (from pint>=0.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Collecting requests>=2.9.1 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting xmltodict (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl
Collecting cryptography>=1.3 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/c3/c1/cf8665c955c9393e9ff0872ba6cd3dc6f46ef915e94afcf6e0410508ca69/cryptography-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/d5/3d/1c54e92f62bbc747a638da94adb439f99dc2d2f3041fe41a06b0da4f2808/ntlm_auth-1.3.0-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/39/ec/d93dfc69617a028915df914339ef66936ea976ef24fa62940fd86ba0326e/urllib3-1.25.2-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/8d/e9/0c8afd1579e5cf7bc0f06fbcd7cdb954cbc0baadd505973949a99337da1c/cffi-1.12.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Installing collected packages: enum34, six, absl-py, MarkupSafe, jinja2, colorama, colorlog, blinker, futures, PyYAML, funcsigs, pint, numpy, functools32, contextlib2, urllib3, certifi, chardet, idna, requests, pycparser, cffi, asn1crypto, ipaddress, cryptography, ntlm-auth, requests-ntlm, xmltodict, pywinrm, timeout-decorator
Successfully installed MarkupSafe-1.1.1 PyYAML-3.12 absl-py-0.7.1 asn1crypto-0.24.0 blinker-1.4 certifi-2019.3.9 cffi-1.12.3 chardet-3.0.4 colorama-0.4.1 colorlog-2.6.0 contextlib2-0.5.5 cryptography-2.6.1 enum34-1.1.6 funcsigs-1.0.2 functools32-3.2.3.post2 futures-3.2.0 idna-2.8 ipaddress-1.0.22 jinja2-2.10.1 ntlm-auth-1.3.0 numpy-1.13.3 pint-0.9 pycparser-2.19 pywinrm-0.3.0 requests-2.22.0 requests-ntlm-1.1.0 six-1.12.0 timeout-decorator-0.4.1 urllib3-1.25.2 xmltodict-0.12.0
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8886648918533423063.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python> <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/pkb.py> --project=apache-beam-testing --dpb_log_level=INFO --bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results --k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/> --beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src> --official=true --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-549> --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 --beam_prebuilt=false --beam_sdk=java --beam_it_module=sdks/java/io/file-based-io-tests --beam_it_class=org.apache.beam.sdk.io.text.TextIOIT '--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]' '--beam_extra_properties=[filesystem=hdfs]' --beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml> --beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
2019-05-17 12:56:34,922 9efd4f2d MainThread INFO Verbose logging to: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/9efd4f2d/pkb.log>
2019-05-17 12:56:34,923 9efd4f2d MainThread INFO PerfKitBenchmarker version: v1.12.0-1183-g0bd23a1
2019-05-17 12:56:34,924 9efd4f2d MainThread INFO Flag values:
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_it_module=sdks/java/io/file-based-io-tests
--beam_sdk=java
--k8s_get_wait_interval=10
--bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results
--temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]
--nobeam_prebuilt
--kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-549>
--beam_extra_properties=[filesystem=hdfs]
--project=apache-beam-testing
--beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
--official
--dpb_log_level=INFO
--beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml>
--beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src>
--k8s_get_retry_count=36
--benchmarks=beam_integration_benchmark
2019-05-17 12:56:35,144 9efd4f2d MainThread INFO Setting --max_concurrent_threads=200.
2019-05-17 12:56:35,165 9efd4f2d MainThread WARNING The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2019-05-17 12:56:35,187 9efd4f2d MainThread beam_integration_benchmark(1/1) INFO Provisioning resources for benchmark beam_integration_benchmark
2019-05-17 12:56:35,187 9efd4f2d MainThread beam_integration_benchmark(1/1) ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-17 12:56:35,189 9efd4f2d MainThread beam_integration_benchmark(1/1) ERROR Exception running benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 903, in RunBenchmarkTask
RunBenchmark(spec, collector)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-17 12:56:35,189 9efd4f2d MainThread beam_integration_benchmark(1/1) ERROR Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue.
2019-05-17 12:56:35,190 9efd4f2d MainThread beam_integration_benchmark(1/1) INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-05-17 12:56:35,190 9efd4f2d MainThread beam_integration_benchmark(1/1) INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/9efd4f2d/pkb.log>
2019-05-17 12:56:35,190 9efd4f2d MainThread beam_integration_benchmark(1/1) INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/9efd4f2d/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_ManyFiles_TextIOIT_HDFS #548
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/548/display/redirect>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on apache-beam-jenkins-7 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d77474d16734cf63095765cbced97ecd4b016b54 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f d77474d16734cf63095765cbced97ecd4b016b54
Commit message: "[BEAM-6908] Refactor Python performance test groovy file for easy configuration (#8518)"
> git rev-list --no-walk d77474d16734cf63095765cbced97ecd4b016b54 # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
SPARK_LOCAL_IP=127.0.0.1
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8530116607467968728.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a --verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: [--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format: "default"
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins1137394453077549642.sh
+ cp /home/jenkins/.kube/config <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-548>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2971413879807609167.sh
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-548> create namespace beam-performancetests-manyfiles-textioit-hdfs-548
namespace/beam-performancetests-manyfiles-textioit-hdfs-548 created
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3115621561680926084.sh
++ kubectl config current-context
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-548> config set-context gke_apache-beam-testing_us-central1-a_io-datastores --namespace=beam-performancetests-manyfiles-textioit-hdfs-548
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2333554801113232432.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins7963591368800333007.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins4475650116595987458.sh
+ virtualenv <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env> --python=python2.7
New python executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python2.7>
Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python>
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2853089472285293056.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install --upgrade setuptools pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (41.0.1)
Requirement already up-to-date: pip in ./env/.perfkit_env/lib/python2.7/site-packages (19.1.1)
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins6362884170934338488.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
Cloning into '<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker'...>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2567273117029498637.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting absl-py (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Collecting jinja2>=2.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 16)) (41.0.1)
Collecting colorlog[windows]==2.6.0 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 18))
Collecting futures>=3.0.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 19))
Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting PyYAML==3.12 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 20))
Collecting pint>=0.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/15/9d/bf177ebbc57d25e9e296addc14a1303d1e34d7964af5df428a8332349c42/Pint-0.9-py2.py3-none-any.whl
Collecting numpy==1.13.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 22))
Using cached https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting functools32 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 23))
Collecting contextlib2>=0.5.1 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 24))
Using cached https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Collecting timeout-decorator (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 26))
Collecting enum34; python_version < "3.4" (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting six (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2>=2.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/fb/40/f3adb7cf24a8012813c5edb20329eb22d5d8e2a0ecf73d21d6b85865da11/MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl
Collecting funcsigs; python_version == "2.7" (from pint>=0.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Collecting requests>=2.9.1 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting xmltodict (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl
Collecting cryptography>=1.3 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/c3/c1/cf8665c955c9393e9ff0872ba6cd3dc6f46ef915e94afcf6e0410508ca69/cryptography-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/d5/3d/1c54e92f62bbc747a638da94adb439f99dc2d2f3041fe41a06b0da4f2808/ntlm_auth-1.3.0-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/39/ec/d93dfc69617a028915df914339ef66936ea976ef24fa62940fd86ba0326e/urllib3-1.25.2-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/8d/e9/0c8afd1579e5cf7bc0f06fbcd7cdb954cbc0baadd505973949a99337da1c/cffi-1.12.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Installing collected packages: enum34, six, absl-py, MarkupSafe, jinja2, colorama, colorlog, blinker, futures, PyYAML, funcsigs, pint, numpy, functools32, contextlib2, urllib3, certifi, chardet, idna, requests, pycparser, cffi, asn1crypto, ipaddress, cryptography, ntlm-auth, requests-ntlm, xmltodict, pywinrm, timeout-decorator
Successfully installed MarkupSafe-1.1.1 PyYAML-3.12 absl-py-0.7.1 asn1crypto-0.24.0 blinker-1.4 certifi-2019.3.9 cffi-1.12.3 chardet-3.0.4 colorama-0.4.1 colorlog-2.6.0 contextlib2-0.5.5 cryptography-2.6.1 enum34-1.1.6 funcsigs-1.0.2 functools32-3.2.3.post2 futures-3.2.0 idna-2.8 ipaddress-1.0.22 jinja2-2.10.1 ntlm-auth-1.3.0 numpy-1.13.3 pint-0.9 pycparser-2.19 pywinrm-0.3.0 requests-2.22.0 requests-ntlm-1.1.0 six-1.12.0 timeout-decorator-0.4.1 urllib3-1.25.2 xmltodict-0.12.0
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins5664514606446097863.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python> <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/pkb.py> --project=apache-beam-testing --dpb_log_level=INFO --bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results --k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/> --beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src> --official=true --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-548> --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 --beam_prebuilt=false --beam_sdk=java --beam_it_module=sdks/java/io/file-based-io-tests --beam_it_class=org.apache.beam.sdk.io.text.TextIOIT '--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]' '--beam_extra_properties=[filesystem=hdfs]' --beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml> --beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
2019-05-17 06:56:27,156 c9c28a84 MainThread INFO Verbose logging to: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/c9c28a84/pkb.log>
2019-05-17 06:56:27,157 c9c28a84 MainThread INFO PerfKitBenchmarker version: v1.12.0-1183-g0bd23a1
2019-05-17 06:56:27,158 c9c28a84 MainThread INFO Flag values:
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_it_module=sdks/java/io/file-based-io-tests
--beam_sdk=java
--k8s_get_wait_interval=10
--bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results
--temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]
--nobeam_prebuilt
--kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-548>
--beam_extra_properties=[filesystem=hdfs]
--project=apache-beam-testing
--beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
--official
--dpb_log_level=INFO
--beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml>
--beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src>
--k8s_get_retry_count=36
--benchmarks=beam_integration_benchmark
2019-05-17 06:56:27,375 c9c28a84 MainThread INFO Setting --max_concurrent_threads=200.
2019-05-17 06:56:27,396 c9c28a84 MainThread WARNING The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2019-05-17 06:56:27,417 c9c28a84 MainThread beam_integration_benchmark(1/1) INFO Provisioning resources for benchmark beam_integration_benchmark
2019-05-17 06:56:27,417 c9c28a84 MainThread beam_integration_benchmark(1/1) ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-17 06:56:27,419 c9c28a84 MainThread beam_integration_benchmark(1/1) ERROR Exception running benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 903, in RunBenchmarkTask
RunBenchmark(spec, collector)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-17 06:56:27,419 c9c28a84 MainThread beam_integration_benchmark(1/1) ERROR Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue.
2019-05-17 06:56:27,419 c9c28a84 MainThread beam_integration_benchmark(1/1) INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-05-17 06:56:27,420 c9c28a84 MainThread beam_integration_benchmark(1/1) INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/c9c28a84/pkb.log>
2019-05-17 06:56:27,420 c9c28a84 MainThread beam_integration_benchmark(1/1) INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/c9c28a84/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_ManyFiles_TextIOIT_HDFS #547
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/547/display/redirect?page=changes>
Changes:
[kcweaver] [BEAM-6959] Go SDK: support integration tests on Flink
[kcweaver] run job server on an unused port
[boyuanz] Add retry strategy when TableRow or TableCell is null from BQ response.
[markliu] [BEAM-7339] Make input and checksum configurable for Python WordCountIT
[github] Remove unused attemptNum
[github] [BEAM-6908] Refactor Python performance test groovy file for easy
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on apache-beam-jenkins-7 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d77474d16734cf63095765cbced97ecd4b016b54 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f d77474d16734cf63095765cbced97ecd4b016b54
Commit message: "[BEAM-6908] Refactor Python performance test groovy file for easy configuration (#8518)"
> git rev-list --no-walk adb6d0c9f790c9cda363dd5d14f03fb11362f4d1 # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
SPARK_LOCAL_IP=127.0.0.1
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins4440235918923461446.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a --verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: [--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format: "default"
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins9176158424515380759.sh
+ cp /home/jenkins/.kube/config <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-547>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3381698398767848182.sh
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-547> create namespace beam-performancetests-manyfiles-textioit-hdfs-547
namespace/beam-performancetests-manyfiles-textioit-hdfs-547 created
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8146243043903956845.sh
++ kubectl config current-context
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-547> config set-context gke_apache-beam-testing_us-central1-a_io-datastores --namespace=beam-performancetests-manyfiles-textioit-hdfs-547
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2243861627921961433.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins1026793275125282108.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins7500052957179442722.sh
+ virtualenv <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env> --python=python2.7
New python executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python2.7>
Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python>
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins1468530147506093185.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install --upgrade setuptools pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (41.0.1)
Requirement already up-to-date: pip in ./env/.perfkit_env/lib/python2.7/site-packages (19.1.1)
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins1589683611882592053.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
Cloning into '<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker'...>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins6815900581197908987.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting absl-py (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Collecting jinja2>=2.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 16)) (41.0.1)
Collecting colorlog[windows]==2.6.0 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 18))
Collecting futures>=3.0.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 19))
Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting PyYAML==3.12 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 20))
Collecting pint>=0.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/15/9d/bf177ebbc57d25e9e296addc14a1303d1e34d7964af5df428a8332349c42/Pint-0.9-py2.py3-none-any.whl
Collecting numpy==1.13.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 22))
Using cached https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting functools32 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 23))
Collecting contextlib2>=0.5.1 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 24))
Using cached https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Collecting timeout-decorator (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 26))
Collecting enum34; python_version < "3.4" (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting six (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2>=2.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/fb/40/f3adb7cf24a8012813c5edb20329eb22d5d8e2a0ecf73d21d6b85865da11/MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl
Collecting funcsigs; python_version == "2.7" (from pint>=0.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Collecting requests>=2.9.1 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting xmltodict (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl
Collecting cryptography>=1.3 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/c3/c1/cf8665c955c9393e9ff0872ba6cd3dc6f46ef915e94afcf6e0410508ca69/cryptography-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/d5/3d/1c54e92f62bbc747a638da94adb439f99dc2d2f3041fe41a06b0da4f2808/ntlm_auth-1.3.0-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/39/ec/d93dfc69617a028915df914339ef66936ea976ef24fa62940fd86ba0326e/urllib3-1.25.2-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/8d/e9/0c8afd1579e5cf7bc0f06fbcd7cdb954cbc0baadd505973949a99337da1c/cffi-1.12.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Installing collected packages: enum34, six, absl-py, MarkupSafe, jinja2, colorama, colorlog, blinker, futures, PyYAML, funcsigs, pint, numpy, functools32, contextlib2, urllib3, certifi, chardet, idna, requests, pycparser, cffi, asn1crypto, ipaddress, cryptography, ntlm-auth, requests-ntlm, xmltodict, pywinrm, timeout-decorator
Successfully installed MarkupSafe-1.1.1 PyYAML-3.12 absl-py-0.7.1 asn1crypto-0.24.0 blinker-1.4 certifi-2019.3.9 cffi-1.12.3 chardet-3.0.4 colorama-0.4.1 colorlog-2.6.0 contextlib2-0.5.5 cryptography-2.6.1 enum34-1.1.6 funcsigs-1.0.2 functools32-3.2.3.post2 futures-3.2.0 idna-2.8 ipaddress-1.0.22 jinja2-2.10.1 ntlm-auth-1.3.0 numpy-1.13.3 pint-0.9 pycparser-2.19 pywinrm-0.3.0 requests-2.22.0 requests-ntlm-1.1.0 six-1.12.0 timeout-decorator-0.4.1 urllib3-1.25.2 xmltodict-0.12.0
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3564675913631141020.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python> <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/pkb.py> --project=apache-beam-testing --dpb_log_level=INFO --bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results --k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/> --beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src> --official=true --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-547> --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 --beam_prebuilt=false --beam_sdk=java --beam_it_module=sdks/java/io/file-based-io-tests --beam_it_class=org.apache.beam.sdk.io.text.TextIOIT '--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]' '--beam_extra_properties=[filesystem=hdfs]' --beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml> --beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
2019-05-17 00:56:31,620 951738bd MainThread INFO Verbose logging to: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/951738bd/pkb.log>
2019-05-17 00:56:31,620 951738bd MainThread INFO PerfKitBenchmarker version: v1.12.0-1183-g0bd23a1
2019-05-17 00:56:31,621 951738bd MainThread INFO Flag values:
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_it_module=sdks/java/io/file-based-io-tests
--beam_sdk=java
--k8s_get_wait_interval=10
--bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results
--temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]
--nobeam_prebuilt
--kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-547>
--beam_extra_properties=[filesystem=hdfs]
--project=apache-beam-testing
--beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
--official
--dpb_log_level=INFO
--beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml>
--beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src>
--k8s_get_retry_count=36
--benchmarks=beam_integration_benchmark
2019-05-17 00:56:31,842 951738bd MainThread INFO Setting --max_concurrent_threads=200.
2019-05-17 00:56:31,863 951738bd MainThread WARNING The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2019-05-17 00:56:31,884 951738bd MainThread beam_integration_benchmark(1/1) INFO Provisioning resources for benchmark beam_integration_benchmark
2019-05-17 00:56:31,884 951738bd MainThread beam_integration_benchmark(1/1) ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-17 00:56:31,886 951738bd MainThread beam_integration_benchmark(1/1) ERROR Exception running benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 903, in RunBenchmarkTask
RunBenchmark(spec, collector)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-17 00:56:31,886 951738bd MainThread beam_integration_benchmark(1/1) ERROR Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue.
2019-05-17 00:56:31,886 951738bd MainThread beam_integration_benchmark(1/1) INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-05-17 00:56:31,886 951738bd MainThread beam_integration_benchmark(1/1) INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/951738bd/pkb.log>
2019-05-17 00:56:31,886 951738bd MainThread beam_integration_benchmark(1/1) INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/951738bd/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_ManyFiles_TextIOIT_HDFS #546
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/546/display/redirect?page=changes>
Changes:
[lcwik] [BEAM-7283] Update aggregated javadoc versions during linking to point
[lcwik] [BEAM-7283] Build javadocs using online links for all dependencies
[frederik.bode] Add Postcommit test suite for Python 3.7
[iemejia] [website] Add design doc on portable representation of schemas
[athomas1] Modified KinesisIO to use WatermarkPolicyFactory for watermark
[25622840+adude3141] [BEAM-7287] Change JAVA_HOME to match new Jenkins agents (#8584)
[iemejia] [BEAM-7230] Add javadoc on provided poolable data sources on JdbcIO
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on apache-beam-jenkins-4 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision adb6d0c9f790c9cda363dd5d14f03fb11362f4d1 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f adb6d0c9f790c9cda363dd5d14f03fb11362f4d1
Commit message: "Merge pull request #8564 from fredo838/add-suites"
> git rev-list --no-walk 02ac8717bf73f7942e5cf1d1ace7f1247c702a9b # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
SPARK_LOCAL_IP=127.0.0.1
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8539949180437039744.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a --verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: [--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format: "default"
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8825709317051144158.sh
+ cp /home/jenkins/.kube/config <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-546>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3028965902923087535.sh
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-546> create namespace beam-performancetests-manyfiles-textioit-hdfs-546
namespace/beam-performancetests-manyfiles-textioit-hdfs-546 created
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3762485219877461136.sh
++ kubectl config current-context
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-546> config set-context gke_apache-beam-testing_us-central1-a_io-datastores --namespace=beam-performancetests-manyfiles-textioit-hdfs-546
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3631523158744293763.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2912287018844265151.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins1542629850893808535.sh
+ virtualenv <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env> --python=python2.7
New python executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python2.7>
Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python>
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8434490122595815447.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install --upgrade setuptools pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (41.0.1)
Requirement already up-to-date: pip in ./env/.perfkit_env/lib/python2.7/site-packages (19.1.1)
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins31204664757452556.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
Cloning into '<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker'...>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins4060787352689130092.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting absl-py (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Collecting jinja2>=2.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 16)) (41.0.1)
Collecting colorlog[windows]==2.6.0 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 18))
Collecting futures>=3.0.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 19))
Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting PyYAML==3.12 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 20))
Collecting pint>=0.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/15/9d/bf177ebbc57d25e9e296addc14a1303d1e34d7964af5df428a8332349c42/Pint-0.9-py2.py3-none-any.whl
Collecting numpy==1.13.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 22))
Using cached https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting functools32 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 23))
Collecting contextlib2>=0.5.1 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 24))
Using cached https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Collecting timeout-decorator (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 26))
Collecting enum34; python_version < "3.4" (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting six (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2>=2.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/fb/40/f3adb7cf24a8012813c5edb20329eb22d5d8e2a0ecf73d21d6b85865da11/MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl
Collecting funcsigs; python_version == "2.7" (from pint>=0.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Collecting requests>=2.9.1 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting xmltodict (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl
Collecting cryptography>=1.3 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/c3/c1/cf8665c955c9393e9ff0872ba6cd3dc6f46ef915e94afcf6e0410508ca69/cryptography-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/d5/3d/1c54e92f62bbc747a638da94adb439f99dc2d2f3041fe41a06b0da4f2808/ntlm_auth-1.3.0-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/39/ec/d93dfc69617a028915df914339ef66936ea976ef24fa62940fd86ba0326e/urllib3-1.25.2-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/8d/e9/0c8afd1579e5cf7bc0f06fbcd7cdb954cbc0baadd505973949a99337da1c/cffi-1.12.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Installing collected packages: enum34, six, absl-py, MarkupSafe, jinja2, colorama, colorlog, blinker, futures, PyYAML, funcsigs, pint, numpy, functools32, contextlib2, urllib3, certifi, chardet, idna, requests, pycparser, cffi, asn1crypto, ipaddress, cryptography, ntlm-auth, requests-ntlm, xmltodict, pywinrm, timeout-decorator
Successfully installed MarkupSafe-1.1.1 PyYAML-3.12 absl-py-0.7.1 asn1crypto-0.24.0 blinker-1.4 certifi-2019.3.9 cffi-1.12.3 chardet-3.0.4 colorama-0.4.1 colorlog-2.6.0 contextlib2-0.5.5 cryptography-2.6.1 enum34-1.1.6 funcsigs-1.0.2 functools32-3.2.3.post2 futures-3.2.0 idna-2.8 ipaddress-1.0.22 jinja2-2.10.1 ntlm-auth-1.3.0 numpy-1.13.3 pint-0.9 pycparser-2.19 pywinrm-0.3.0 requests-2.22.0 requests-ntlm-1.1.0 six-1.12.0 timeout-decorator-0.4.1 urllib3-1.25.2 xmltodict-0.12.0
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8309287140648296059.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python> <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/pkb.py> --project=apache-beam-testing --dpb_log_level=INFO --bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results --k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/> --beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src> --official=true --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-546> --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 --beam_prebuilt=false --beam_sdk=java --beam_it_module=sdks/java/io/file-based-io-tests --beam_it_class=org.apache.beam.sdk.io.text.TextIOIT '--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]' '--beam_extra_properties=[filesystem=hdfs]' --beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml> --beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
2019-05-16 18:56:31,042 d09fad54 MainThread INFO Verbose logging to: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/d09fad54/pkb.log>
2019-05-16 18:56:31,042 d09fad54 MainThread INFO PerfKitBenchmarker version: v1.12.0-1183-g0bd23a1
2019-05-16 18:56:31,043 d09fad54 MainThread INFO Flag values:
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_it_module=sdks/java/io/file-based-io-tests
--beam_sdk=java
--k8s_get_wait_interval=10
--bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results
--temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]
--nobeam_prebuilt
--kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-546>
--beam_extra_properties=[filesystem=hdfs]
--project=apache-beam-testing
--beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
--official
--dpb_log_level=INFO
--beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml>
--beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src>
--k8s_get_retry_count=36
--benchmarks=beam_integration_benchmark
2019-05-16 18:56:31,258 d09fad54 MainThread INFO Setting --max_concurrent_threads=200.
2019-05-16 18:56:31,280 d09fad54 MainThread WARNING The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2019-05-16 18:56:31,305 d09fad54 MainThread beam_integration_benchmark(1/1) INFO Provisioning resources for benchmark beam_integration_benchmark
2019-05-16 18:56:31,305 d09fad54 MainThread beam_integration_benchmark(1/1) ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-16 18:56:31,307 d09fad54 MainThread beam_integration_benchmark(1/1) ERROR Exception running benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 903, in RunBenchmarkTask
RunBenchmark(spec, collector)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-16 18:56:31,307 d09fad54 MainThread beam_integration_benchmark(1/1) ERROR Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue.
2019-05-16 18:56:31,307 d09fad54 MainThread beam_integration_benchmark(1/1) INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-05-16 18:56:31,308 d09fad54 MainThread beam_integration_benchmark(1/1) INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/d09fad54/pkb.log>
2019-05-16 18:56:31,308 d09fad54 MainThread beam_integration_benchmark(1/1) INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/d09fad54/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_ManyFiles_TextIOIT_HDFS #545
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/545/display/redirect>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on apache-beam-jenkins-4 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 02ac8717bf73f7942e5cf1d1ace7f1247c702a9b (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 02ac8717bf73f7942e5cf1d1ace7f1247c702a9b
Commit message: "Merge #8589: Simple fix: make use of Reify in Latest to replace duplicate code"
> git rev-list --no-walk 02ac8717bf73f7942e5cf1d1ace7f1247c702a9b # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
SPARK_LOCAL_IP=127.0.0.1
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins7752610300707100586.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a --verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: [--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format: "default"
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins7117581053406401916.sh
+ cp /home/jenkins/.kube/config <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-545>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3270152659297394787.sh
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-545> create namespace beam-performancetests-manyfiles-textioit-hdfs-545
namespace/beam-performancetests-manyfiles-textioit-hdfs-545 created
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins6355772281345516910.sh
++ kubectl config current-context
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-545> config set-context gke_apache-beam-testing_us-central1-a_io-datastores --namespace=beam-performancetests-manyfiles-textioit-hdfs-545
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins845737229022018465.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8066925616179560022.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins527979785844256874.sh
+ virtualenv <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env> --python=python2.7
New python executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python2.7>
Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python>
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins6074262177316806124.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install --upgrade setuptools pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (41.0.1)
Requirement already up-to-date: pip in ./env/.perfkit_env/lib/python2.7/site-packages (19.1.1)
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3129701400896581845.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
Cloning into '<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker'...>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2442984007263564594.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting absl-py (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Collecting jinja2>=2.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 16)) (41.0.1)
Collecting colorlog[windows]==2.6.0 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 18))
Collecting futures>=3.0.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 19))
Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting PyYAML==3.12 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 20))
Collecting pint>=0.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/15/9d/bf177ebbc57d25e9e296addc14a1303d1e34d7964af5df428a8332349c42/Pint-0.9-py2.py3-none-any.whl
Collecting numpy==1.13.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 22))
Using cached https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting functools32 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 23))
Collecting contextlib2>=0.5.1 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 24))
Using cached https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Collecting timeout-decorator (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 26))
Collecting enum34; python_version < "3.4" (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting six (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2>=2.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/fb/40/f3adb7cf24a8012813c5edb20329eb22d5d8e2a0ecf73d21d6b85865da11/MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl
Collecting funcsigs; python_version == "2.7" (from pint>=0.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Collecting requests>=2.9.1 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting xmltodict (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl
Collecting cryptography>=1.3 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/c3/c1/cf8665c955c9393e9ff0872ba6cd3dc6f46ef915e94afcf6e0410508ca69/cryptography-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/d5/3d/1c54e92f62bbc747a638da94adb439f99dc2d2f3041fe41a06b0da4f2808/ntlm_auth-1.3.0-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/01/11/525b02e4acc0c747de8b6ccdab376331597c569c42ea66ab0a1dbd36eca2/urllib3-1.24.3-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/8d/e9/0c8afd1579e5cf7bc0f06fbcd7cdb954cbc0baadd505973949a99337da1c/cffi-1.12.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Installing collected packages: enum34, six, absl-py, MarkupSafe, jinja2, colorama, colorlog, blinker, futures, PyYAML, funcsigs, pint, numpy, functools32, contextlib2, certifi, chardet, idna, urllib3, requests, pycparser, cffi, asn1crypto, ipaddress, cryptography, ntlm-auth, requests-ntlm, xmltodict, pywinrm, timeout-decorator
Successfully installed MarkupSafe-1.1.1 PyYAML-3.12 absl-py-0.7.1 asn1crypto-0.24.0 blinker-1.4 certifi-2019.3.9 cffi-1.12.3 chardet-3.0.4 colorama-0.4.1 colorlog-2.6.0 contextlib2-0.5.5 cryptography-2.6.1 enum34-1.1.6 funcsigs-1.0.2 functools32-3.2.3.post2 futures-3.2.0 idna-2.8 ipaddress-1.0.22 jinja2-2.10.1 ntlm-auth-1.3.0 numpy-1.13.3 pint-0.9 pycparser-2.19 pywinrm-0.3.0 requests-2.21.0 requests-ntlm-1.1.0 six-1.12.0 timeout-decorator-0.4.1 urllib3-1.24.3 xmltodict-0.12.0
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins528467566999959245.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python> <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/pkb.py> --project=apache-beam-testing --dpb_log_level=INFO --bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results --k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/> --beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src> --official=true --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-545> --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 --beam_prebuilt=false --beam_sdk=java --beam_it_module=sdks/java/io/file-based-io-tests --beam_it_class=org.apache.beam.sdk.io.text.TextIOIT '--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]' '--beam_extra_properties=[filesystem=hdfs]' --beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml> --beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
2019-05-16 12:56:29,053 88a725bc MainThread INFO Verbose logging to: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/88a725bc/pkb.log>
2019-05-16 12:56:29,054 88a725bc MainThread INFO PerfKitBenchmarker version: v1.12.0-1183-g0bd23a1
2019-05-16 12:56:29,055 88a725bc MainThread INFO Flag values:
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_it_module=sdks/java/io/file-based-io-tests
--beam_sdk=java
--k8s_get_wait_interval=10
--bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results
--temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]
--nobeam_prebuilt
--kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-545>
--beam_extra_properties=[filesystem=hdfs]
--project=apache-beam-testing
--beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
--official
--dpb_log_level=INFO
--beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml>
--beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src>
--k8s_get_retry_count=36
--benchmarks=beam_integration_benchmark
2019-05-16 12:56:29,326 88a725bc MainThread INFO Setting --max_concurrent_threads=200.
2019-05-16 12:56:29,348 88a725bc MainThread WARNING The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2019-05-16 12:56:29,372 88a725bc MainThread beam_integration_benchmark(1/1) INFO Provisioning resources for benchmark beam_integration_benchmark
2019-05-16 12:56:29,372 88a725bc MainThread beam_integration_benchmark(1/1) ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-16 12:56:29,375 88a725bc MainThread beam_integration_benchmark(1/1) ERROR Exception running benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 903, in RunBenchmarkTask
RunBenchmark(spec, collector)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-16 12:56:29,375 88a725bc MainThread beam_integration_benchmark(1/1) ERROR Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue.
2019-05-16 12:56:29,375 88a725bc MainThread beam_integration_benchmark(1/1) INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-05-16 12:56:29,375 88a725bc MainThread beam_integration_benchmark(1/1) INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/88a725bc/pkb.log>
2019-05-16 12:56:29,376 88a725bc MainThread beam_integration_benchmark(1/1) INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/88a725bc/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_ManyFiles_TextIOIT_HDFS #544
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/544/display/redirect?page=changes>
Changes:
[robinyqiu] Simple fix: make use of Reify in Latest to replace duplicate code
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on apache-beam-jenkins-4 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 02ac8717bf73f7942e5cf1d1ace7f1247c702a9b (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 02ac8717bf73f7942e5cf1d1ace7f1247c702a9b
Commit message: "Merge #8589: Simple fix: make use of Reify in Latest to replace duplicate code"
> git rev-list --no-walk 194576262c41f2502189c52565e2a91ca2204562 # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
SPARK_LOCAL_IP=127.0.0.1
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2246834361661447248.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a --verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: [--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format: "default"
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins5432931579822090504.sh
+ cp /home/jenkins/.kube/config <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-544>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2988082016143820682.sh
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-544> create namespace beam-performancetests-manyfiles-textioit-hdfs-544
namespace/beam-performancetests-manyfiles-textioit-hdfs-544 created
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins9095213051039324093.sh
++ kubectl config current-context
+ kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-544> config set-context gke_apache-beam-testing_us-central1-a_io-datastores --namespace=beam-performancetests-manyfiles-textioit-hdfs-544
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8448021546125553817.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins6685854994813447953.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3690457240156239061.sh
+ virtualenv <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env> --python=python2.7
New python executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python2.7>
Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python>
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins5076600438491648724.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install --upgrade setuptools pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (41.0.1)
Requirement already up-to-date: pip in ./env/.perfkit_env/lib/python2.7/site-packages (19.1.1)
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins4734182197776904051.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker>
Cloning into '<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker'...>
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins5039974578385124331.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/pip> install -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting absl-py (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Collecting jinja2>=2.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 16)) (41.0.1)
Collecting colorlog[windows]==2.6.0 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 18))
Collecting futures>=3.0.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 19))
Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting PyYAML==3.12 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 20))
Collecting pint>=0.7 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/15/9d/bf177ebbc57d25e9e296addc14a1303d1e34d7964af5df428a8332349c42/Pint-0.9-py2.py3-none-any.whl
Collecting numpy==1.13.3 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 22))
Using cached https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting functools32 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 23))
Collecting contextlib2>=0.5.1 (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 24))
Using cached https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Collecting timeout-decorator (from -r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 26))
Collecting enum34; python_version < "3.4" (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting six (from absl-py->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 14))
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2>=2.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 15))
Using cached https://files.pythonhosted.org/packages/fb/40/f3adb7cf24a8012813c5edb20329eb22d5d8e2a0ecf73d21d6b85865da11/MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 17))
Using cached https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl
Collecting funcsigs; python_version == "2.7" (from pint>=0.7->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 21))
Using cached https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Collecting requests>=2.9.1 (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting xmltodict (from pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl
Collecting cryptography>=1.3 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/c3/c1/cf8665c955c9393e9ff0872ba6cd3dc6f46ef915e94afcf6e0410508ca69/cryptography-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/d5/3d/1c54e92f62bbc747a638da94adb439f99dc2d2f3041fe41a06b0da4f2808/ntlm_auth-1.3.0-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests>=2.9.1->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/01/11/525b02e4acc0c747de8b6ccdab376331597c569c42ea66ab0a1dbd36eca2/urllib3-1.24.3-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/8d/e9/0c8afd1579e5cf7bc0f06fbcd7cdb954cbc0baadd505973949a99337da1c/cffi-1.12.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/requirements.txt> (line 25))
Installing collected packages: enum34, six, absl-py, MarkupSafe, jinja2, colorama, colorlog, blinker, futures, PyYAML, funcsigs, pint, numpy, functools32, contextlib2, certifi, chardet, idna, urllib3, requests, pycparser, cffi, asn1crypto, ipaddress, cryptography, ntlm-auth, requests-ntlm, xmltodict, pywinrm, timeout-decorator
Successfully installed MarkupSafe-1.1.1 PyYAML-3.12 absl-py-0.7.1 asn1crypto-0.24.0 blinker-1.4 certifi-2019.3.9 cffi-1.12.3 chardet-3.0.4 colorama-0.4.1 colorlog-2.6.0 contextlib2-0.5.5 cryptography-2.6.1 enum34-1.1.6 funcsigs-1.0.2 functools32-3.2.3.post2 futures-3.2.0 idna-2.8 ipaddress-1.0.22 jinja2-2.10.1 ntlm-auth-1.3.0 numpy-1.13.3 pint-0.9 pycparser-2.19 pywinrm-0.3.0 requests-2.21.0 requests-ntlm-1.1.0 six-1.12.0 timeout-decorator-0.4.1 urllib3-1.24.3 xmltodict-0.12.0
[beam_PerformanceTests_ManyFiles_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins5880898122929366546.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/env/.perfkit_env/bin/python> <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/pkb.py> --project=apache-beam-testing --dpb_log_level=INFO --bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results --k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/> --beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src> --official=true --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-544> --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 --beam_prebuilt=false --beam_sdk=java --beam_it_module=sdks/java/io/file-based-io-tests --beam_it_class=org.apache.beam.sdk.io.text.TextIOIT '--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]' '--beam_extra_properties=[filesystem=hdfs]' --beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml> --beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
2019-05-16 06:56:30,656 883a33b5 MainThread INFO Verbose logging to: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/883a33b5/pkb.log>
2019-05-16 06:56:30,656 883a33b5 MainThread INFO PerfKitBenchmarker version: v1.12.0-1183-g0bd23a1
2019-05-16 06:56:30,658 883a33b5 MainThread INFO Flag values:
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_it_module=sdks/java/io/file-based-io-tests
--beam_sdk=java
--k8s_get_wait_interval=10
--bigquery_table=beam_performance.many_files_textioit_hdfs_pkb_results
--temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/>
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--bigQueryDataset=beam_performance,--bigQueryTable=many_files_textioit_hdfs_results,--reportGcsPerformanceMetrics=true,--gcsPerformanceMetrics=true,--numberOfRecords=1000000,--numberOfShards=1000]
--nobeam_prebuilt
--kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/config-beam-performancetests-manyfiles-textioit-hdfs-544>
--beam_extra_properties=[filesystem=hdfs]
--project=apache-beam-testing
--beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml>
--official
--dpb_log_level=INFO
--beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml>
--beam_location=<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src>
--k8s_get_retry_count=36
--benchmarks=beam_integration_benchmark
2019-05-16 06:56:31,050 883a33b5 MainThread INFO Setting --max_concurrent_threads=200.
2019-05-16 06:56:31,072 883a33b5 MainThread WARNING The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2019-05-16 06:56:31,095 883a33b5 MainThread beam_integration_benchmark(1/1) INFO Provisioning resources for benchmark beam_integration_benchmark
2019-05-16 06:56:31,096 883a33b5 MainThread beam_integration_benchmark(1/1) ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-16 06:56:31,098 883a33b5 MainThread beam_integration_benchmark(1/1) ERROR Exception running benchmark
Traceback (most recent call last):
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 903, in RunBenchmarkTask
RunBenchmark(spec, collector)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 752, in RunBenchmark
DoProvisionPhase(spec, detailed_timer)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 538, in DoProvisionPhase
spec.ConstructDpbService()
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py",> line 209, in ConstructDpbService
self.dpb_service = dpb_service_class(self.config.dpb_service)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",> line 53, in __init__
super(GcpDpbDataflow, self).__init__(dpb_service_spec)
File "<https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/dpb_service.py",> line 127, in __init__
'The flag dpb_service_zone must be provided, for provisioning.')
InvalidFlagConfigurationError: The flag dpb_service_zone must be provided, for provisioning.
2019-05-16 06:56:31,099 883a33b5 MainThread beam_integration_benchmark(1/1) ERROR Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue.
2019-05-16 06:56:31,099 883a33b5 MainThread beam_integration_benchmark(1/1) INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-05-16 06:56:31,099 883a33b5 MainThread beam_integration_benchmark(1/1) INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/883a33b5/pkb.log>
2019-05-16 06:56:31,099 883a33b5 MainThread beam_integration_benchmark(1/1) INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/runs/883a33b5/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org