You are viewing a plain text version of this content. The canonical link for it is here.

Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2019/11/20 19:03:50 UTC

Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py36 #729

See <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/729/display/redirect?page=changes>

Changes:

[katarzyna.kucharczyk] [BEAM-6335] Added streaming GroupByKey Test that reads SyntheticSource

[katarzyna.kucharczyk] [BEAM-6335] Changed SyntheticDataPublisher to publish String UTF values

[katarzyna.kucharczyk] [BEAM-6335] Added custom PubSub Matcher that stops pipeline after

[valentyn] Restore original behavior of evaluating worker host on Windows until a

[echauchot] [BEAM-8470] Add an empty spark-structured-streaming runner project

[echauchot] [BEAM-8470] Fix missing dep

[echauchot] [BEAM-8470] Add SparkPipelineOptions

[echauchot] [BEAM-8470] Start pipeline translation

[echauchot] [BEAM-8470] Add global pipeline translation structure

[echauchot] [BEAM-8470] Add nodes translators structure

[echauchot] [BEAM-8470] Wire node translators with pipeline translator

[echauchot] [BEAM-8470] Renames: better differenciate pipeline translator for

[echauchot] [BEAM-8470] Organise methods in PipelineTranslator

[echauchot] [BEAM-8470] Initialise BatchTranslationContext

[echauchot] [BEAM-8470] Refactoring: -move batch/streaming common translation

[echauchot] [BEAM-8470] Make transform translation clearer: renaming, comments

[echauchot] [BEAM-8470] Improve javadocs

[echauchot] [BEAM-8470] Move SparkTransformOverrides to correct package

[echauchot] [BEAM-8470] Move common translation context components to superclass

[echauchot] [BEAM-8470] apply spotless

[echauchot] [BEAM-8470] Make codestyle and firebug happy

[echauchot] [BEAM-8470] Add TODOs

[echauchot] [BEAM-8470] Post-pone batch qualifier in all classes names for

[echauchot] [BEAM-8470] Add precise TODO for multiple TransformTranslator per

[echauchot] [BEAM-8470] Added SparkRunnerRegistrar

[echauchot] [BEAM-8470] Add basic pipeline execution. Refactor translatePipeline()

[echauchot] [BEAM-8470] Create PCollections manipulation methods

[echauchot] [BEAM-8470] Create Datasets manipulation methods

[echauchot] [BEAM-8470] Add Flatten transformation translator

[echauchot] [BEAM-8470] Add primitive GroupByKeyTranslatorBatch implementation

[echauchot] [BEAM-8470] Use Iterators.transform() to return Iterable

[echauchot] [BEAM-8470] Implement read transform

[echauchot] [BEAM-8470] update TODO

[echauchot] [BEAM-8470] Apply spotless

[echauchot] [BEAM-8470] start source instanciation

[echauchot] [BEAM-8470] Improve exception flow

[echauchot] [BEAM-8470] Improve type enforcement in ReadSourceTranslator

[echauchot] [BEAM-8470] Experiment over using spark Catalog to pass in Beam Source

[echauchot] [BEAM-8470] Add source mocks

[echauchot] [BEAM-8470] fix mock, wire mock in translators and create a main test.

[echauchot] [BEAM-8470] Use raw WindowedValue so that spark Encoders could work

[echauchot] [BEAM-8470] clean deps

[echauchot] [BEAM-8470] Move DatasetSourceMock to proper batch mode

[echauchot] [BEAM-8470] Run pipeline in batch mode or in streaming mode

[echauchot] [BEAM-8470] Split batch and streaming sources and translators

[echauchot] [BEAM-8470] Use raw Encoder<WindowedValue> also in regular

[echauchot] [BEAM-8470] Clean

[echauchot] [BEAM-8470] Add ReadSourceTranslatorStreaming

[echauchot] [BEAM-8470] Move Source and translator mocks to a mock package.

[echauchot] [BEAM-8470] Pass Beam Source and PipelineOptions to the spark DataSource

[echauchot] [BEAM-8470] Refactor DatasetSource fields

[echauchot] [BEAM-8470] Wire real SourceTransform and not mock and update the test

[echauchot] [BEAM-8470] Add missing 0-arg public constructor

[echauchot] [BEAM-8470] Use new PipelineOptionsSerializationUtils

[echauchot] [BEAM-8470] Apply spotless and fix  checkstyle

[echauchot] [BEAM-8470] Add a dummy schema for reader

[echauchot] [BEAM-8470] Add empty 0-arg constructor for mock source

[echauchot] [BEAM-8470] Clean

[echauchot] [BEAM-8470] Checkstyle and Findbugs

[echauchot] [BEAM-8470] Refactor SourceTest to a UTest instaed of a main

[echauchot] [BEAM-8470] Fix pipeline triggering: use a spark action instead of

[echauchot] [BEAM-8470] improve readability of options passing to the source

[echauchot] [BEAM-8470] Clean unneeded fields in DatasetReader

[echauchot] [BEAM-8470] Fix serialization issues

[echauchot] [BEAM-8470] Add SerializationDebugger

[echauchot] [BEAM-8470] Add serialization test

[echauchot] [BEAM-8470] Move SourceTest to same package as tested class

[echauchot] [BEAM-8470] Fix SourceTest

[echauchot] [BEAM-8470] Simplify beam reader creation as it created once the source

[echauchot] [BEAM-8470] Put all transform translators Serializable

[echauchot] [BEAM-8470] Enable test mode

[echauchot] [BEAM-8470] Enable gradle build scan

[echauchot] [BEAM-8470] Add flatten test

[echauchot] [BEAM-8470] First attempt for ParDo primitive implementation

[echauchot] [BEAM-8470] Serialize windowedValue to byte[] in source to be able to

[echauchot] [BEAM-8470] Comment schema choices

[echauchot] [BEAM-8470] Fix errorprone

[echauchot] [BEAM-8470] Fix testMode output to comply with new binary schema

[echauchot] [BEAM-8470] Cleaning

[echauchot] [BEAM-8470] Remove bundleSize parameter and always use spark default

[echauchot] [BEAM-8470] Fix split bug

[echauchot] [BEAM-8470] Clean

[echauchot] [BEAM-8470] Add ParDoTest

[echauchot] [BEAM-8470] Address minor review notes

[echauchot] [BEAM-8470] Clean

[echauchot] [BEAM-8470] Add GroupByKeyTest

[echauchot] [BEAM-8470] Add comments and TODO to GroupByKeyTranslatorBatch

[echauchot] [BEAM-8470] Fix type checking with Encoder of WindowedValue<T>

[echauchot] [BEAM-8470] Port latest changes of ReadSourceTranslatorBatch to

[echauchot] [BEAM-8470] Remove no more needed putDatasetRaw

[echauchot] [BEAM-8470] Add ComplexSourceTest

[echauchot] [BEAM-8470] Fail in case of having SideInouts or State/Timers

[echauchot] [BEAM-8470] Fix Encoders: create an Encoder for every manipulated type

[echauchot] [BEAM-8470] Apply spotless

[echauchot] [BEAM-8470] Fixed Javadoc error

[echauchot] [BEAM-8470] Rename SparkSideInputReader class and rename pruneOutput()

[echauchot] [BEAM-8470] Don't use deprecated

[echauchot] [BEAM-8470] Simplify logic of ParDo translator

[echauchot] [BEAM-8470] Fix kryo issue in GBK translator with a workaround

[echauchot] [BEAM-8470] Rename SparkOutputManager for consistency

[echauchot] [BEAM-8470] Fix for test elements container in GroupByKeyTest

[echauchot] [BEAM-8470] Added "testTwoPardoInRow"

[echauchot] [BEAM-8470] Add a test for the most simple possible Combine

[echauchot] [BEAM-8470] Rename SparkDoFnFilterFunction to DoFnFilterFunction for

[echauchot] [BEAM-8470] Generalize the use of SerializablePipelineOptions in place

[echauchot] [BEAM-8470] Fix getSideInputs

[echauchot] [BEAM-8470] Extract binary schema creation in a helper class

[echauchot] [BEAM-8470] First version of combinePerKey

[echauchot] [BEAM-8470] Improve type checking of Tuple2 encoder

[echauchot] [BEAM-8470] Introduce WindowingHelpers (and helpers package) and use it

[echauchot] [BEAM-8470] Fix combiner using KV as input, use binary encoders in place

[echauchot] [BEAM-8470] Add combinePerKey and CombineGlobally tests

[echauchot] [BEAM-8470] Introduce RowHelpers

[echauchot] [BEAM-8470] Add CombineGlobally translation to avoid translating

[echauchot] [BEAM-8470] Cleaning

[echauchot] [BEAM-8470] Get back to classes in translators resolution because URNs

[echauchot] [BEAM-8470] Fix various type checking issues in Combine.Globally

[echauchot] [BEAM-8470] Update test with Long

[echauchot] [BEAM-8470] Fix combine. For unknown reason GenericRowWithSchema is used

[echauchot] [BEAM-8470] Use more generic Row instead of GenericRowWithSchema

[echauchot] [BEAM-8470] Add explanation about receiving a Row as input in the

[echauchot] [BEAM-8470] Fix encoder bug in combinePerkey

[echauchot] [BEAM-8470] Cleaning

[echauchot] [BEAM-8470] Implement WindowAssignTranslatorBatch

[echauchot] [BEAM-8470] Implement WindowAssignTest

[echauchot] [BEAM-8470] Fix javadoc

[echauchot] [BEAM-8470] Added SideInput support

[echauchot] [BEAM-8470] Fix CheckStyle violations

[echauchot] [BEAM-8470] Don't use Reshuffle translation

[echauchot] [BEAM-8470] Added using CachedSideInputReader

[echauchot] [BEAM-8470] Added TODO comment for ReshuffleTranslatorBatch

[echauchot] [BEAM-8470] And unchecked warning suppression

[echauchot] [BEAM-8470] Add streaming source initialisation

[echauchot] [BEAM-8470] Implement first streaming source

[echauchot] [BEAM-8470] Add a TODO on spark output modes

[echauchot] [BEAM-8470] Add transformators registry in PipelineTranslatorStreaming

[echauchot] [BEAM-8470] Add source streaming test

[echauchot] [BEAM-8470] Specify checkpointLocation at the pipeline start

[echauchot] [BEAM-8470] Clean unneeded 0 arg constructor in batch source

[echauchot] [BEAM-8470] Clean streaming source

[echauchot] [BEAM-8470] Continue impl of offsets for streaming source

[echauchot] [BEAM-8470] Deal with checkpoint and offset based read

[echauchot] [BEAM-8470] Apply spotless and fix spotbugs warnings

[echauchot] [BEAM-8470] Disable never ending test

[echauchot] [BEAM-8470] Fix access level issues, typos and modernize code to Java 8

[echauchot] [BEAM-8470] Merge Spark Structured Streaming runner into main Spark

[echauchot] [BEAM-8470] Fix non-vendored imports from Spark Streaming Runner classes

[echauchot] [BEAM-8470] Pass doFnSchemaInformation to ParDo batch translation

[echauchot] [BEAM-8470] Fix spotless issues after rebase

[echauchot] [BEAM-8470] Fix logging levels in Spark Structured Streaming translation

[echauchot] [BEAM-8470] Add SparkStructuredStreamingPipelineOptions and

[echauchot] [BEAM-8470] Rename SparkPipelineResult to

[echauchot] [BEAM-8470] Use PAssert in Spark Structured Streaming transform tests

[echauchot] [BEAM-8470] Ignore spark offsets (cf javadoc)

[echauchot] [BEAM-8470] implement source.stop

[echauchot] [BEAM-8470] Update javadoc

[echauchot] [BEAM-8470] Apply Spotless

[echauchot] [BEAM-8470] Enable batch Validates Runner tests for Structured Streaming

[echauchot] [BEAM-8470] Limit the number of partitions to make tests go 300% faster

[echauchot] [BEAM-8470] Fixes ParDo not calling setup and not tearing down if

[echauchot] [BEAM-8470] Pass transform based doFnSchemaInformation in ParDo

[echauchot] [BEAM-8470] Consider null object case on RowHelpers, fixes empty side

[echauchot] [BEAM-8470] Put back batch/simpleSourceTest.testBoundedSource

[echauchot] [BEAM-8470] Update windowAssignTest

[echauchot] [BEAM-8470] Add comment about checkpoint mark

[echauchot] [BEAM-8470] Re-code GroupByKeyTranslatorBatch to conserve windowing

[echauchot] [BEAM-8470] re-enable reduceFnRunner timers for output

[echauchot] [BEAM-8470] Improve visibility of debug messages

[echauchot] [BEAM-8470] Add a test that GBK preserves windowing

[echauchot] [BEAM-8470] Add TODO in Combine translations

[echauchot] [BEAM-8470] Update KVHelpers.extractKey() to deal with WindowedValue and

[echauchot] [BEAM-8470] Fix comment about schemas

[echauchot] [BEAM-8470] Implement reduce part of CombineGlobally translation with

[echauchot] [BEAM-8470] Output data after combine

[echauchot] [BEAM-8470] Implement merge accumulators part of CombineGlobally

[echauchot] [BEAM-8470] Fix encoder in combine call

[echauchot] [BEAM-8470] Revert extractKey while combinePerKey is not done (so that

[echauchot] [BEAM-8470] Apply a groupByKey avoids for some reason that the spark

[echauchot] [BEAM-8470] Fix case when a window does not merge into any other window

[echauchot] [BEAM-8470] Fix wrong encoder in combineGlobally GBK

[echauchot] [BEAM-8470] Fix bug in the window merging logic

[echauchot] [BEAM-8470] Remove the mapPartition that adds a key per partition

[echauchot] [BEAM-8470] Remove CombineGlobally translation because it is less

[echauchot] [BEAM-8470] Now that there is only Combine.PerKey translation, make only

[echauchot] [BEAM-8470] Clean no more needed KVHelpers

[echauchot] [BEAM-8470] Clean not more needed RowHelpers

[echauchot] [BEAM-8470] Clean not more needed WindowingHelpers

[echauchot] [BEAM-8470] Fix javadoc of AggregatorCombiner

[echauchot] [BEAM-8470] Fixed immutable list bug

[echauchot] [BEAM-8470] add comment in combine globally test

[echauchot] [BEAM-8470] Clean groupByKeyTest

[echauchot] [BEAM-8470] Add a test that combine per key preserves windowing

[echauchot] [BEAM-8470] Ignore for now not working test testCombineGlobally

[echauchot] [BEAM-8470] Add metrics support in DoFn

[echauchot] [BEAM-8470] Add missing dependencies to run Spark Structured Streaming

[echauchot] [BEAM-8470] Add setEnableSparkMetricSinks() method

[echauchot] [BEAM-8470] Fix javadoc

[echauchot] [BEAM-8470] Fix accumulators initialization in Combine that prevented

[echauchot] [BEAM-8470] Add a test to check that CombineGlobally preserves windowing

[echauchot] [BEAM-8470] Persist all output Dataset if there are multiple outputs in

[echauchot] [BEAM-8470] Added metrics sinks and tests

[echauchot] [BEAM-8470] Make spotless happy

[echauchot] [BEAM-8470] Add PipelineResults to Spark structured streaming.

[echauchot] [BEAM-8470] Update log4j configuration

[echauchot] [BEAM-8470] Add spark execution plans extended debug messages.

[echauchot] [BEAM-8470] Print number of leaf datasets

[echauchot] [BEAM-8470] fixup! Add PipelineResults to Spark structured streaming.

[echauchot] [BEAM-8470] Remove no more needed AggregatorCombinerPerKey (there is

[echauchot] [BEAM-8470] After testing performance and correctness, launch pipeline

[echauchot] [BEAM-8470] Improve Pardo translation performance: avoid calling a

[echauchot] [BEAM-8470] Use "sparkMaster" in local mode to obtain number of shuffle

[echauchot] [BEAM-8470] Wrap Beam Coders into Spark Encoders using

[echauchot] [BEAM-8470] Wrap Beam Coders into Spark Encoders using

[echauchot] [BEAM-8470] type erasure: spark encoders require a Class<T>, pass Object

[echauchot] [BEAM-8470] Fix scala Product in Encoders to avoid StackEverflow

[echauchot] [BEAM-8470] Conform to spark ExpressionEncoders: pass classTags,

[echauchot] [BEAM-8470] Add a simple spark native test to test Beam coders wrapping

[echauchot] [BEAM-8470] Fix code generation in Beam coder wrapper

[echauchot] [BEAM-8470] Lazy init coder because coder instance cannot be

[echauchot] [BEAM-8470] Fix warning in coder construction by reflexion

[echauchot] [BEAM-8470] Fix ExpressionEncoder generated code: typos, try catch, fqcn

[echauchot] [BEAM-8470] Fix getting the output value in code generation

[echauchot] [BEAM-8470] Fix beam coder lazy init using reflexion: use .clas + try

[echauchot] [BEAM-8470] Remove lazy init of beam coder because there is no generic

[echauchot] [BEAM-8470] Remove example code

[echauchot] [BEAM-8470] Fix equal and hashcode

[echauchot] [BEAM-8470] Fix generated code: uniform exceptions catching, fix

[echauchot] [BEAM-8470] Add an assert of equality in the encoders test

[echauchot] [BEAM-8470] Apply spotless and checkstyle and add javadocs

[echauchot] [BEAM-8470] Wrap exceptions in UserCoderExceptions

[echauchot] [BEAM-8470] Put Encoders expressions serializable

[echauchot] [BEAM-8470] Catch Exception instead of IOException because some coders

[echauchot] [BEAM-8470] Apply new Encoders to CombinePerKey

[echauchot] [BEAM-8470] Apply new Encoders to Read source

[echauchot] [BEAM-8470] Improve performance of source: the mapper already calls

[echauchot] [BEAM-8470] Ignore long time failing test: SparkMetricsSinkTest

[echauchot] [BEAM-8470] Apply new Encoders to Window assign translation

[echauchot] [BEAM-8470] Apply new Encoders to AggregatorCombiner

[echauchot] [BEAM-8470] Create a Tuple2Coder to encode scala tuple2

[echauchot] [BEAM-8470] Apply new Encoders to GroupByKey

[echauchot] [BEAM-8470] Apply new Encoders to Pardo. Replace Tuple2Coder with

[echauchot] [BEAM-8470] Apply spotless, fix typo and javadoc

[echauchot] [BEAM-8470] Use beam encoders also in the output of the source

[echauchot] [BEAM-8470] Remove unneeded cast

[echauchot] [BEAM-8470] Fix: Remove generic hack of using object. Use actual Coder

[echauchot] [BEAM-8470] Remove Encoders based on kryo now that we call Beam coders

[echauchot] [BEAM-8470] Add a jenkins job for validates runner tests in the new

[echauchot] [BEAM-8470] Apply spotless

[echauchot] [BEAM-8470] Rebase on master: pass sideInputMapping in SimpleDoFnRunner

[echauchot] Fix SpotBugs

[echauchot] [BEAM-8470] simplify coders in combinePerKey translation

[echauchot] [BEAM-8470] Fix combiner. Do not reuse instance of accumulator

[echauchot] [BEAM-8470] input windows can arrive exploded (for sliding windows). As

[echauchot] [BEAM-8470] Add a combine test with sliding windows

[echauchot] [BEAM-8470] Add a test to test combine translation on binaryCombineFn

[echauchot] [BEAM-8470] Fix tests: use correct

[echauchot] [BEAM-8470] Fix wrong expected results in

[echauchot] [BEAM-8470] Add disclaimers about this runner being experimental

[echauchot] [BEAM-8470] Fix: create an empty accumulator in

[echauchot] [BEAM-8470] Apply spotless

[echauchot] [BEAM-8470] Add a countPerElement test with sliding windows

[echauchot] [BEAM-8470] Fix the output timestamps of combine: timestamps must be

[echauchot] [BEAM-8470] set log level to info to avoid resource consumption in

[echauchot] [BEAM-8470] Fix CombineTest.testCountPerElementWithSlidingWindows

[aromanenko.dev] [BEAM-8470] Remove "validatesStructuredStreamingRunnerBatch" from

[echauchot] [BEAM-8470] Fix timestamps in combine output: assign the timestamp to

[iemejia] [website] Add Spark Structured Runner VR badge to the github template

[tvalentyn] [BEAM-8575] Add a Python test to test windowing in DoFn finish_bundle()


------------------------------------------
Started by timer
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on apache-beam-jenkins-4 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/>
No credentials specified
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/src> # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d1b513dea046228803a5089860382a5850c667e9 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d1b513dea046228803a5089860382a5850c667e9
Commit message: "[BEAM-8575] Add a Python test to test windowing in DoFn finish_bundle() (#10145)"
 > git rev-list --no-walk 0b415fd7e9dd5c80034ca237e08c3959ec78ffe3 # timeout=10
No emails were triggered.
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_WordCountIT_Py36] $ /bin/bash -xe /tmp/jenkins284860423958160068.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker>
[beam_PerformanceTests_WordCountIT_Py36] $ /bin/bash -xe /tmp/jenkins7090700778039925815.sh
+ rm -rf <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/env/.perfkit_env>
[beam_PerformanceTests_WordCountIT_Py36] $ /bin/bash -xe /tmp/jenkins653948504521483333.sh
+ virtualenv <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/env/.perfkit_env> --python=python2.7
New python executable in <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/env/.perfkit_env/bin/python2.7>
Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/env/.perfkit_env/bin/python>
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7
[beam_PerformanceTests_WordCountIT_Py36] $ /bin/bash -xe /tmp/jenkins4764780150592324939.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/env/.perfkit_env/bin/pip> install --upgrade setuptools pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already up-to-date: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (41.6.0)
Requirement already up-to-date: pip in ./env/.perfkit_env/lib/python2.7/site-packages (19.3.1)
[beam_PerformanceTests_WordCountIT_Py36] $ /bin/bash -xe /tmp/jenkins887326843635075091.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker>
Cloning into '<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker'...>
[beam_PerformanceTests_WordCountIT_Py36] $ /bin/bash -xe /tmp/jenkins3493902573762662044.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/env/.perfkit_env/bin/pip> install -r <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker/requirements.txt>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing /home/jenkins/.cache/pip/wheels/a7/15/a0/0a0561549ad11cdc1bc8fa1191a353efd30facf6bfb507aefc/absl_py-0.8.1-cp27-none-any.whl
Collecting jinja2>=2.7
  Using cached https://files.pythonhosted.org/packages/65/e0/eb35e762802015cab1ccee04e8a277b03f1d8e53da3ec3106882ec42558b/Jinja2-2.10.3-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./env/.perfkit_env/lib/python2.7/site-packages (from -r <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker/requirements.txt> (line 16)) (41.6.0)
Collecting colorlog[windows]==2.6.0
  Using cached https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/92/a0/00/8690a57883956a301d91cf4ec999cc0b258b01e3f548f86e89/blinker-1.4-cp27-none-any.whl
Collecting futures>=3.0.3
  Using cached https://files.pythonhosted.org/packages/d8/a6/f46ae3f1da0cd4361c344888f59ec2f5785e69c872e175a748ef6071cdb5/futures-3.3.0-py2-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f/PyYAML-3.13-cp27-cp27mu-linux_x86_64.whl
Collecting pint>=0.7
  Using cached https://files.pythonhosted.org/packages/15/9d/bf177ebbc57d25e9e296addc14a1303d1e34d7964af5df428a8332349c42/Pint-0.9-py2.py3-none-any.whl
Collecting numpy==1.16.4
  Using cached https://files.pythonhosted.org/packages/1f/c7/198496417c9c2f6226616cff7dedf2115a4f4d0276613bab842ec8ac1e23/numpy-1.16.4-cp27-cp27mu-manylinux1_x86_64.whl
Processing /home/jenkins/.cache/pip/wheels/b5/18/32/77a1030457155606ba5e3ec3a8a57132b1a04b1c4f765177b2/functools32-3.2.3.post2-cp27-none-any.whl
Collecting contextlib2>=0.5.1
  Using cached https://files.pythonhosted.org/packages/85/60/370352f7ef6aa96c52fb001831622f50f923c1d575427d021b8ab3311236/contextlib2-0.6.0.post1-py2.py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/8f/db/f3/3028984b926c1c0165d7be4ead1092d219bee6ef71e9219578/pywinrm-0.4.1-py2.py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/f1/e6/ea/7387e3629cb46ba65140141f972745b823f4486c6fe884ccb8/timeout_decorator-0.4.1-cp27-none-any.whl
Collecting google-cloud-monitoring
  Using cached https://files.pythonhosted.org/packages/0d/24/35da9c690c1d5042d45c40bbce49daf65e8a754b38f78f8c3d394bd12843/google_cloud_monitoring-0.33.0-py2.py3-none-any.whl
Collecting enum34; python_version < "3.4"
  Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting six
  Using cached https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23
  Using cached https://files.pythonhosted.org/packages/fb/40/f3adb7cf24a8012813c5edb20329eb22d5d8e2a0ecf73d21d6b85865da11/MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting colorama; extra == "windows"
  Using cached https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl
Collecting funcsigs; python_version == "2.7"
  Using cached https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0
  Using cached https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Collecting requests>=2.9.1
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting xmltodict
  Using cached https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
  Using cached https://files.pythonhosted.org/packages/29/3a/c528ef37f48d6ffba16f0f3c0426456ba21e0dd32be9c61a2ade93e07faa/google_api_core-1.14.3-py2.py3-none-any.whl
Collecting cryptography>=1.3
  Using cached https://files.pythonhosted.org/packages/c8/52/ad7f2cbe3b6e9340526dc401b38bb67c06160449446e15b0bf5f947fa168/cryptography-2.8-cp27-cp27mu-manylinux2010_x86_64.whl
Collecting ntlm-auth>=1.0.2
  Using cached https://files.pythonhosted.org/packages/50/09/5e397eb18685b14fd8b209e26cdb4fa6451c82c1bcc651fef05fa73e7b27/ntlm_auth-1.4.0-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl
Collecting certifi>=2017.4.17
  Using cached https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/9e/3d/a2/1bec8bb7db80ab3216dbc33092bb7ccd0debfb8ba42b5668d5/googleapis_common_protos-1.6.0-cp27-none-any.whl
Collecting protobuf>=3.4.0
  Using cached https://files.pythonhosted.org/packages/c5/49/ffa7ab9c52ec56b535cffec3bc844254c073888e6d4aeee464671ac97480/protobuf-3.10.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting google-auth<2.0dev,>=0.4.0
  Using cached https://files.pythonhosted.org/packages/7b/cb/786dc53d93494784935a62947643b48250b84a882474e714f9af5e1a1928/google_auth-1.7.1-py2.py3-none-any.whl
Collecting pytz
  Using cached https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl
Collecting grpcio<2.0dev,>=1.8.2; extra == "grpc"
  Using cached https://files.pythonhosted.org/packages/0c/47/35cc9f6fd43f8e5ed74fcc6dd8a0cb2e89c118dd3ef7a8ff25e65bf0909f/grpcio-1.25.0-cp27-cp27mu-manylinux2010_x86_64.whl
Collecting cffi!=1.11.3,>=1.8
  Using cached https://files.pythonhosted.org/packages/93/5d/c4f950891251e478929036ca07b22f0b10324460c1d0a4434c584481db51/cffi-1.13.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ipaddress; python_version < "3"
  Using cached https://files.pythonhosted.org/packages/c2/f8/49697181b1651d8347d24c095ce46c7346c37335ddc7d255833e7cde674d/ipaddress-1.0.23-py2.py3-none-any.whl
Collecting rsa<4.1,>=3.1.4
  Using cached https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl
Collecting cachetools<3.2,>=2.0.0
  Using cached https://files.pythonhosted.org/packages/2f/a6/30b0a0bef12283e83e58c1d6e7b5aabc7acfc4110df81a4471655d33e704/cachetools-3.1.1-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.2.1
  Using cached https://files.pythonhosted.org/packages/52/50/bb4cefca37da63a0c52218ba2cb1b1c36110d84dcbae8aa48cd67c5e95c2/pyasn1_modules-0.2.7-py2.py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/f2/9a/90/de94f8556265ddc9d9c8b271b0f63e57b26fb1d67a45564511/pycparser-2.19-py2.py3-none-any.whl
Collecting pyasn1>=0.1.3
  Using cached https://files.pythonhosted.org/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl
Installing collected packages: enum34, six, absl-py, MarkupSafe, jinja2, colorama, colorlog, blinker, futures, PyYAML, funcsigs, pint, numpy, functools32, contextlib2, urllib3, certifi, chardet, idna, requests, pycparser, cffi, ipaddress, cryptography, ntlm-auth, requests-ntlm, xmltodict, pywinrm, timeout-decorator, protobuf, googleapis-common-protos, pyasn1, rsa, cachetools, pyasn1-modules, google-auth, pytz, grpcio, google-api-core, google-cloud-monitoring
Successfully installed MarkupSafe-1.1.1 PyYAML-3.13 absl-py-0.8.1 blinker-1.4 cachetools-3.1.1 certifi-2019.9.11 cffi-1.13.2 chardet-3.0.4 colorama-0.4.1 colorlog-2.6.0 contextlib2-0.6.0.post1 cryptography-2.8 enum34-1.1.6 funcsigs-1.0.2 functools32-3.2.3.post2 futures-3.3.0 google-api-core-1.14.3 google-auth-1.7.1 google-cloud-monitoring-0.33.0 googleapis-common-protos-1.6.0 grpcio-1.25.0 idna-2.8 ipaddress-1.0.23 jinja2-2.10.3 ntlm-auth-1.4.0 numpy-1.16.4 pint-0.9 protobuf-3.10.0 pyasn1-0.4.8 pyasn1-modules-0.2.7 pycparser-2.19 pytz-2019.3 pywinrm-0.4.1 requests-2.22.0 requests-ntlm-1.1.0 rsa-4.0 six-1.13.0 timeout-decorator-0.4.1 urllib3-1.25.7 xmltodict-0.12.0
[beam_PerformanceTests_WordCountIT_Py36] $ /bin/bash -xe /tmp/jenkins4416635476999561327.sh
+ <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/env/.perfkit_env/bin/python> <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker/pkb.py> --project=apache-beam-testing --dpb_log_level=INFO --bigquery_table=beam_performance.wordcount_py36_pkb_results --k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/> --beam_location=<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/src> --official=true --dpb_service_zone=fake_zone --beam_sdk=python --benchmarks=beam_integration_benchmark --beam_it_class=apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it --beam_it_module=:sdks:python:test-suites:dataflow:py36 --beam_prebuilt=true --beam_python_sdk_location=build/apache-beam.tar.gz --beam_runner=TestDataflowRunner --beam_it_timeout=1200 '--beam_it_args=--project=apache-beam-testing,--staging_location=gs://temp-storage-for-end-to-end-tests/staging-it,--temp_location=gs://temp-storage-for-end-to-end-tests/temp-it,--input=gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000*,--output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output,--expect_checksum=ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710,--num_workers=10,--autoscaling_algorithm=NONE'
2019-11-20 19:03:49,293 a00ac7b4 MainThread INFO     Verbose logging to: <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/runs/a00ac7b4/pkb.log>
2019-11-20 19:03:49,293 a00ac7b4 MainThread INFO     PerfKitBenchmarker version: v1.12.0-1532-g2784642
2019-11-20 19:03:49,295 a00ac7b4 MainThread INFO     Flag values:
--beam_it_class=apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it
--beam_it_timeout=1200
--beam_it_module=:sdks:python:test-suites:dataflow:py36
--beam_sdk=python
--k8s_get_wait_interval=10
--bigquery_table=beam_performance.wordcount_py36_pkb_results
--temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/>
--beam_it_args=--project=apache-beam-testing,--staging_location=gs://temp-storage-for-end-to-end-tests/staging-it,--temp_location=gs://temp-storage-for-end-to-end-tests/temp-it,--input=gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000*,--output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output,--expect_checksum=ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710,--num_workers=10,--autoscaling_algorithm=NONE
--beam_prebuilt
--dpb_service_zone=fake_zone
--project=apache-beam-testing
--official
--dpb_log_level=INFO
--beam_location=<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/src>
--beam_runner=TestDataflowRunner
--k8s_get_retry_count=36
--benchmarks=beam_integration_benchmark
--beam_python_sdk_location=build/apache-beam.tar.gz
2019-11-20 19:03:50,008 a00ac7b4 MainThread INFO     Setting --max_concurrent_threads=200.
2019-11-20 19:03:50,033 a00ac7b4 MainThread WARNING  The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2019-11-20 19:03:50,107 a00ac7b4 MainThread beam_integration_benchmark(1/1) INFO     Provisioning resources for benchmark beam_integration_benchmark
2019-11-20 19:03:50,112 a00ac7b4 MainThread beam_integration_benchmark(1/1) INFO     Preparing benchmark beam_integration_benchmark
2019-11-20 19:03:50,113 a00ac7b4 MainThread beam_integration_benchmark(1/1) INFO     Running benchmark beam_integration_benchmark
2019-11-20 19:03:50,114 a00ac7b4 MainThread beam_integration_benchmark(1/1) ERROR    Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
  File "<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 846, in RunBenchmark
    DoRunPhase(spec, collector, detailed_timer)
  File "<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 689, in DoRunPhase
    samples = spec.BenchmarkRun(spec)
  File "<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",> line 161, in Run
    job_type=job_type)
TypeError: SubmitJob() takes at least 3 arguments (5 given)
2019-11-20 19:03:50,114 a00ac7b4 MainThread beam_integration_benchmark(1/1) INFO     Cleaning up benchmark beam_integration_benchmark
2019-11-20 19:03:50,118 a00ac7b4 MainThread beam_integration_benchmark(1/1) ERROR    Exception running benchmark
Traceback (most recent call last):
  File "<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 995, in RunBenchmarkTask
    RunBenchmark(spec, collector)
  File "<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 846, in RunBenchmark
    DoRunPhase(spec, collector, detailed_timer)
  File "<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",> line 689, in DoRunPhase
    samples = spec.BenchmarkRun(spec)
  File "<https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",> line 161, in Run
    job_type=job_type)
TypeError: SubmitJob() takes at least 3 arguments (5 given)
2019-11-20 19:03:50,118 a00ac7b4 MainThread beam_integration_benchmark(1/1) ERROR    Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue.
2019-11-20 19:03:50,118 a00ac7b4 MainThread beam_integration_benchmark(1/1) INFO     Benchmark run statuses:
---------------------------------------------------------------------------------
Name                        UID                          Status  Failed Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark  beam_integration_benchmark0  FAILED                  
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-11-20 19:03:50,118 a00ac7b4 MainThread beam_integration_benchmark(1/1) INFO     Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/runs/a00ac7b4/pkb.log>
2019-11-20 19:03:50,119 a00ac7b4 MainThread beam_integration_benchmark(1/1) INFO     Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/ws/runs/a00ac7b4/completion_statuses.json>
Build step 'Execute shell' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org