You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2020/01/15 12:46:19 UTC
Build failed in Jenkins:
beam_LoadTests_Java_Combine_Portable_Flink_Batch #2
See <https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch/2/display/redirect?page=changes>
Changes:
[mzobii.baig] Beam-2535 : Pursue pull request 4700 with manual apply changes from
[mzobii.baig] Beam-2535 : Pursue pull request 4700 with manual apply changes from
[mzobii.baig] Beam-2535 : Pursue pull request 4700 with manual apply changes from
[mzobii.baig] Beam-2535 : Replace timeStamp with outputTimeStamp
[mzobii.baig] Beam-2535 : Apply Spotless
[mzobii.baig] Beam-2535 : Pass outputTimestamp param in onTimer method
[mzobii.baig] Beam-2535 : Minor changed
[rehman.muradali] [BEAM-2535] : Add Commit State in ParDoEvaluator
[rehman.muradali] [BEAM-2535] : Add outputTimestamp in compare method, Revert
[mzobii.baig] Beam-2535 : Modifying default minimum target and GC time
[rehman.muradali] BEAM-2535 : Removal of extra lines
[mzobii.baig] Beam-2535 : Proposed changes
[mzobii.baig] Beam-2535 : Added original PR watermark hold functionality.
[rehman.muradali] [BEAM-2535] Apply Spotless
[mzobii.baig] [Beam-2535] Variable renaming and added output timestamp in
[mzobii.baig] Beam-2535 : Apply Spotless
[mzobii.baig] [Beam-2535] Modify test case
[mzobii.baig] [Beam-2535] Added comments
[mzobii.baig] [Beam-2535] Apply Spotless
[mzobii.baig] [Beam-2535] Set Processing Time with outputTimestamp
[mzobii.baig] [Beam-2535] Minor renaming
[rehman.muradali] [BEAM-2535] Revert Processing Time, Addition of OutputTimestamp
[rehman.muradali] [BEAM-2535] Revert TimerReceiver outputTimestamp
[kirillkozlov] Modify AggregateProjectMergeRule to have a condition
[ehudm] [BEAM-8269] Convert from_callable type hints to Beam types
[kirillkozlov] SpotlesApply
[kirillkozlov] Test for a query with a predicate
[kirillkozlov] A list of visited nodes should be unique per onMatch invocation
[rehman.muradali] [BEAM-2535] Revert TimerReceiver outputTimestamp
[kirillkozlov] Make sure all nodes are explored
[dcavazos] [BEAM-7390] Add code snippet for Min
[dcavazos] [BEAM-7390] Add code snippet for Max
[rehman.muradali] [BEAM-2535] Making OnTimer compatible
[rehman.muradali] [BEAM-2535] Making OnTimer compatible
[kirillkozlov] Add a new Jenkins job for SQL perf tests
[kirillkozlov] Test boilerplate
[rehman.muradali] Adding OutputTimestamp in Timer Object
[rehman.muradali] Apply Spotless and checkstyle
[kirillkozlov] Table proxy to add TimeMonitor after the IO
[kirillkozlov] Tests for direct_read w/o push-down and default methods
[mzobii.baig] [Beam-2535] Added watermark functionality for the dataflow runner
[kenn] Use more informative assertions in some py tests
[mzobii.baig] [Beam-2535] Used boolean instead boxed type
[kirillkozlov] Cleanup
[dcavazos] [BEAM-7390] Add code snippet for Sum
[mzobii.baig] [Beam-2535] Modify required watermark hold functionality
[kirillkozlov] Monitor total number of fields read from an IO
[ehudm] Fix _get_args for typing.Tuple in Py3.5.2
[kcweaver] Add FlinkMiniClusterEntryPoint for testing the uber jar submission
[kcweaver] [BEAM-8512] Add integration tests for flink_runner.py.
[kcweaver] Build mini cluster jar excluding unwanted classes.
[kcweaver] Rename to testFlinkUberJarPyX.Y
[kcweaver] Increase timeout on beam_PostCommit_PortableJar_Flink.
[kamil.wasilewski] [BEAM-1440] Provide functions for waiting for BQ job and exporting
[kamil.wasilewski] [BEAM-1440] Create _BigQuerySource that implements iobase.BoundedSource
[kamil.wasilewski] [BEAM-1440] Reorganised BigQuery read IT tests
[kamil.wasilewski] [BEAM-1440] Create postCommitIT jobs running on Flink Runner
[kamil.wasilewski] [BEAM-1440] Convert strings to bytes on Python 3 if field type is BYTES
[kamil.wasilewski] [BEAM-1440]: Support RECORD fields in coder
[kamil.wasilewski] [BEAM-1440] Remove json files after reading
[kamil.wasilewski] [BEAM-1440] Marked classes as private
[kamil.wasilewski] [BEAM-1440] Do not force to create temp dataset when using dry run
[echauchot] [BEAM-5192] Migrate ElasticsearchIO to v7
[echauchot] [BEAM-5192] Minor change of ESIO public configuration API:
[robinyqiu] BeamZetaSqlCalcRel prototype
[valentyn] Install SDK after tarball is generated to avoid a race in proto stubs
[kamil.wasilewski] [BEAM-8671] Add Python 3.7 support for LoadTestBuilder
[kamil.wasilewski] [BEAM-8671] Add ParDo test running on Python 3.7
[ehudm] Fix cleanPython race with :clean
[robinyqiu] Fix bug in SingleRowScanConverter
[robinyqiu] Use BeamBigQuerySqlDialect
[boyuanz] [BEAM-8536] Migrate using requested_execution_time to
[pabloem] Initialize logging configuration in Pipeline object
[daniel.o.programmer] [BEAM-7970] Touch-up on Go protobuf generation instructions.
[kamil.wasilewski] [BEAM-8979] Remove mypy-protobuf dependency
[echauchot] [BEAM-5192] Fix missing ifs for ES7 specificities.
[echauchot] [BEAM-5192] Remove unneeded transitive dependencies, upgrade ES and
[echauchot] [BEAM-5192] Disable MockHttpTransport plugin to enabe http dialog to
[mikhail] Update release docs
[relax] Merge pull request #10311: [BEAM-8810] Detect stuck commits in
[kcweaver] Import freezegun for Python time testing.
[kcweaver] Allow message stream to yield duplicates.
[mikhail] Blogpost stub
[kcweaver] [BEAM-8891] Create and submit Spark portable jar in Python.
[kcweaver] [BEAM-8296] containerize spark job server
[robinyqiu] Address comments
[github] [GoSDK] Make data channel splits idempotent (#10406)
[pabloem] Initialize logging configuration in PipelineOptions object.
[rehman.muradali] EarliestTimestamp Fix for outputTimestamp
[lukasz.gajowy] [BEAM-5495] Make PipelineResourcesDetectorAbstractFactory an inner
[lukasz.gajowy] [BEAM-5495] Change detect() return type to List
[lukasz.gajowy] [BEAM-5495] Minor docs and test fixes
[mxm] [BEAM-8959] Invert metrics flag in Flink Runner
[lukasz.gajowy] [BEAM-5495] Re-add test verifying order of resources detection
[echauchot] [BEAM-5192] Fix util class, elasticsearch changed their json output of
[mikhail] Add blogpost file
[tysonjh] CachingShuffleBatchReader use bytes to limit size.
[mikhail] Add blogpost highlights
[github] Update release guide for cherry picks (#10399)
[heejong] [BEAM-8902] parameterize input type of Java external transform
[lukasz.gajowy] [BEAM-5495] Prevent nested jar scanning (jarfiles in jarfiles)
[ehudm] Dicts are not valid DoFn.process return values
[github] Update release notes version to correct one.
[valentyn] Sickbay VR tests that don't pass
[chamikara] Makes environment ID a top level attribute of PTransform.
[angoenka] [BEAM-8944] Change to use single thread in py sdk bundle progress report
[aaltay] [BEAM-8335] Background caching job (#10405)
[ehudm] Light cleanup of opcodes.py
[chamikara] Setting environment ID for ParDo and Combine transforms
[pawel.pasterz] [BEAM-8978] Publish table size of data written during HadoopFormatIOIT
[echauchot] [BEAM-5192] Set a custom json serializer for document metadata to be
[echauchot] [BEAM-5192] Remove testWritePartialUpdateWithErrors because triggering
[sunjincheng121] [BEAM-7949] Add time-based cache threshold support in the data service
[mxm] [BEAM-8996] Auto-generate pipeline options documentation for FlinkRunner
[mxm] Regenerate Flink options table with the latest master
[mxm] [BEAM-8996] Improvements to the Flink runner page
[ehudm] Upgrade parameterized version to 0.7.0+
[kawaigin] [BEAM-8977] Resolve test flakiness
[dpcollins] Modify PubsubClient to use the proto message throughout.
[lukecwik] [BEAM-9004] Migrate org.mockito.Matchers#anyString to
[suztomo] GenericJsonAssert
[suztomo] Refactoring with assertEqualsAsJson
[chamikara] Fixes Go formatting.
[robertwb] [BEAM-8335] Add a TestStreamService Python Implementation (#10120)
[lukecwik] Minor cleanup of tests using TestStream. (#10188)
[pabloem] [BEAM-2572] Python SDK S3 Filesystem (#9955)
[github] [BEAM-8974] Wait for log messages to be processed before checking them.
[sunjincheng121] [BEAM-7949] Introduce PeriodicThread for time-based cache threshold
[github] Merge pull request #10356: [BEAM-7274] Infer a Beam Schema from a
[echauchot] [BEAM-9019] Improve Encoders: replace as much as possible of catalyst
[lukecwik] [BEAM-8623] Add status_endpoint field to provision api ProvisionInfo
[github] [BEAM-8999] Respect timestamp combiners in PGBKCVOperation. (#10425)
[github] Update dataflow container images to beam-master-20191220 (#10448)
[zyichi] [BEAM-8824] Add support to allow specify window allowed_lateness in
[chamikara] Adds documentation for environment_id fields.
[lukecwik] [BEAM-8846] Update documentation about stream observers and factories,
[apilloud] [BEAM-9023] Upgrade to ZetaSQL 2019.12.1
[lukecwik] [BEAM-7951] Allow runner to configure customization WindowedValue coder.
[chamikara] Sets missing environmentId in several locations.
[bhulette] [BEAM-8988] RangeTracker for _CustomBigQuerySource (#10412)
[ehudm] Set TMPDIR for tox environments
[robinyqiu] Address comments
[robinyqiu] Address comments
[kcweaver] Refactor shared uber jar generation code into common subclass.
[sunjincheng121] [BEAM-8935] Fail fast if sdk harness startup failed.
[echauchot] [BEAM-5192] use <= and >= in version specific code instead of == to be
[relax] Merge pull request #10444: [BEAM-9010] Proper TableRow size calculation
[github] Merge pull request #10449: [BEAM-7274] Implement the Protobuf schema
[sunjincheng121] [BEAM-9030] Bump grpc to 1.26.0
[github] Python example parameters fix
[bhulette] [BEAM-9026] Clean up RuntimeValueProvider.runtime_options (#10457)
[kirillkozlov] Fix BytesValue unparsing
[kirillkozlov] Fix floating point literals
[kirillkozlov] Fix string literals
[kirillkozlov] Add null check for SqlTypeFamily
[kirillkozlov] ZetaSqlCalcRule should be disaled by defualt
[kirillkozlov] spotles
[ehudm] [BEAM-9025] Update Dataflow Java container worker
[heejong] [BEAM-9034] Update environment_id for ExternalTransform in Python SDK
[sunjincheng121] [BEAM-9030] Update the dependencies to make sure the dependency linkage
[mxm] [BEAM-8962] Report Flink metric accumulator only when pipeline ends
[github] Revert "[BEAM-8932] Modify PubsubClient to use the proto message
[ehudm] junitxml_report: Add failure tag support
[github] Catch __module__ is None.
[relax] Merge pull request #10422: [BEAM-2535] TimerData signature update
[rehman.muradali] Rebase TimerData PR
[udim] [BEAM-9012] Change __init__ hints so they work with pytype (#10466)
[github] [BEAM-9039] Fix race on reading channel readErr. (#10456)
[lcwik] [BEAM-5605] Increase precision of fraction used during splitting.
[github] [BEAM-8487] Convert forward references to Any (#9888)
[lukecwik] [BEAM-9020] LengthPrefixUnknownCodersTest to avoid relying on
[lukecwik] [BEAM-7951] Improve the docs for beam_runner_api.proto and
[sunjincheng121] [BEAM-9006] Improve ProcessManager for shutdown hook handling.
[kamil.wasilewski] [BEAM-8671] Fix Python 3.7 ParDo test job name
[github] [BEAM-5600] Add unimplemented split API to Runner side SDF libraries.
[github] [BEAM-5605] Fix type used to describe channel splits to match type used
[github] [BEAM-5605] Ensure that split calls are routed to the active bundle
[suztomo] protobuf 3.11.1
[jeff] BEAM-8745 More fine-grained controls for the size of a BigQuery Load job
[kcweaver] Make Spark REST URL a separate pipeline option.
[kirillkozlov] Address comments
[aaltay] [BEAM-8335] On Unbounded Source change (#10442)
[aaltay] [BEAM-9013] TestStream fix for DataflowRunner (#10445)
[angoenka] [BEAM-8575] Refactor test_do_fn_with_windowing_in_finish_bundle to work
[sunjincheng121] [BEAM-9055] Unify the config names of Fn Data API across languages.
[rehman.muradali] onTimer/setTimer signature updates
[sunjincheng121] fixup
[davidsabater] [BEAM-9053] Improve error message when unable to get the correct
[mxm] [BEAM-8577] Initialize FileSystems during Coder deserialization in
[github] Update _posts_2019-12-16-beam-2.17.0.md
[github] Cleanup formatting.
[suztomo] google_auth_version 0.19.0
[github] Update release date.
[lcwik] [BEAM-9059] Migrate PTransformTranslation to use string constants
[iemejia] [BEAM-5546] Update commons-codec to version 1.14
[iemejia] [BEAM-8701] Remove unused commons-io_1x dependency
[iemejia] [BEAM-8701] Update commons-io to version 2.6
[iemejia] [BEAM-5544] Update cassandra-all dependency to version 3.11.5
[iemejia] [BEAM-8749] Update cassandra-driver-mapping to version 3.8.0
[mxm] Rename FlinkClassloading to Workarounds
[mxm] [BEAM-9060] Restore stdout/stderr in case Flink's
[echauchot] Fix link in javadoc to accumulators
[github] Restrict the upper bound for pyhamcrest, since new version does not work
[apilloud] [BEAM-9027] [SQL] Fix ZetaSQL Byte Literals
[github] [BEAM-9058] Fix line-too-long exclusion regex and re-enable
[altay] Readability/Lint fixes
[hannahjiang] BEAM-8780 reuse RC images instead of recreate images
[iemejia] [BEAM-8716] Update commons-csv to version 1.7
[iemejia] [BEAM-9041] Add missing equals methods for GenericRecord <-> Row
[iemejia] [BEAM-9042] Fix RowToGenericRecordFn Avro schema serialization
[iemejia] [BEAM-9042] Update SchemaCoder doc with info about functions requiring
[iemejia] [BEAM-9042] Test serializability and equality of Row<->GenericRecord
[tvalentyn] [BEAM-9062] Improve assertion error for equal_to (#10504)
[iemejia] [BEAM-8717] Update commons-lang3 to version 3.9
[iemejia] [BEAM-8717] Make non core modules use only the repackaged commons-lang3
[chamikara] [BEAM-8960]: Add an option for user to opt out of using insert id for
[ehudm] Small fixes to verify_release_build.sh
[kirillkozlov] Metric name should not be constant
[36090911+boyuanzz] [BEAM-8932] [BEAM-9036] Revert reverted commit to use PubsubMessage as
[sunjincheng121] fixup
[github] Update ParDoTest.java
[rehman.muradali] Apply spotless
[rehman.muradali] Compilation Fix PardoTest
[rehman.muradali] Reverting outputTimestamp validation
[rehman.muradali] CheckStyle Fix
[rehman.muradali] Adding Category to exclude Flink Runner
[jkai] [BEAM-8496] remove SDF translators in flink streaming transform
[github] Fix blogpost typo (#10532)
[kcweaver] [BEAM-9070] tests use absolute paths for job server jars
[12602502+Ardagan] Fix headings in downloads.md
[github] Add # pytype: skip-file before first import statement in each py file
[apilloud] [BEAM-9027] Unparse DOY/DOW/WEEK Enums properly for ZetaSQL
[33895511+aromanenko-dev] [BEAM-8953] Extend ParquetIO read builders for AvroParquetReader
[brad.g.west] [BEAM-9078] Pass total_size to storage.Upload
[hannahjiang] BEAM-7861 add direct_running_mode option
[github] [BEAM-9075] Disable JoinCommuteRule for ZetaSQL planner (#10542)
[bhulette] [BEAM-9075] add a test case. (#10545)
[12602502+Ardagan] [BEAM-8821] Document Python SDK 2.17.0 deps (#10212)
[kirillkozlov] Missing commit
[hannahjiang] [BEAM-7861] rephrase direct_running_mode option checking
[kcweaver] [BEAM-8337] Hard-code Flink versions.
[echauchot] [BEAM-9019] Remove BeamCoderWrapper to avoid extra object allocation and
[lukecwik] [BEAM-8624] Implement Worker Status FnService in Dataflow runner
[github] [BEAM-5605] Add support for executing pair with restriction, split
[kcweaver] fix indentation
[kcweaver] Update release guide
[lostluck] [BEAM-9080] Support KVs in the Go SDK's Partition
[github] Rephrasing lull logging to avoid alarming users (#10446)
[robertwb] [BEAM-8575] Added counter tests for CombineFn (#10190)
[github] [BEAM-8490] Fix instance_to_type for empty containers (#9894)
[apilloud] [BEAM-8630] Use column numbers for BeamZetaSqlCalRel
[apilloud] [BEAM-9027] Backport BigQuerySqlDialect fixes
[robertwb] [BEAM-8575] Test hot-key fanout with accumulation modes. (#10159)
[github] [BEAM-9059] Use string constants in PTransformTranslation instead of
[iemejia] [BEAM-8956] Begin unifying contributor instructions into a single
[pawel.pasterz] [BEAM-7115] Fix metrics being incorrectly gathered
[mxm] Remove incorrectly tagged test annotation from test case
[mxm] [BEAM-6008] Propagate errors during pipeline execution in Java's
[github] Tighten language and remove distracting link
[pabloem] [BEAM-7390] Add code snippet for Top (#10179)
[bhulette] [BEAM-8993] [SQL] MongoDB predicate push down. (#10417)
[lukecwik] [BEAM-8740] Remove unused dependency from Spark runner (#10564)
[robertwb] [BEAM-6587] Remove hacks due to missing common string coder.
[kirillkozlov] Update data source for SQL performance tests
[github] [BEAM-5605] Add support for channel splitting to the gRPC read "source"
[github] [BEAM-5605] Add support for additional parameters to SplittableDofn
[chadrik] [BEAM-7746] Address changes in code since annotations were introduced
[chadrik] [BEAM-7746] Typing fixes that require runtime code changes
[chadrik] [BEAM-7746] Avoid creating attributes dynamically, so that they can be
[chadrik] [BEAM-7746] Bugfix: coder id is expected to be str in python3
[chadrik] [BEAM-7746] Explicitly unpack tuple to avoid inferring unbounded tuple
[chadrik] [BEAM-7746] Generate files with protobuf urns as part of gen_protos
[chadrik] [BEAM-7746] Move name and coder to base StateSpec class
[chadrik] [BEAM-7746] Remove reference to missing attribute in
[chadrik] [BEAM-7746] Non-Optional arguments cannot default to None
[chadrik] [BEAM-7746] Avoid reusing variables with different data types
[chadrik] [BEAM-7746] Add StateHandler abstract base class
[chadrik] [BEAM-7746] Add TODO about fixing assignment to
[chadrik] [BEAM-7746] Fix functions that were defined twice
[chadrik] [BEAM-7746] Fix tests that have the same name
[iemejia] [BEAM-9040] Add skipQueries option to skip queries in a Nexmark suite
[iemejia] [BEAM-9040] Add Spark Structured Streaming Runner to Nexmark PostCommit
[valentyn] Switch to unittest.SkipTest instead of using nose.
[mxm] [BEAM-6008] Make sure to end stream only after sending all messages and
[chamikara] Sets the correct coder when clustering is enabled for the
[robertwb] Always initalize output processor on construction.
[github] [Go SDK Doc] Update Dead Container Link (#10585)
[github] Merge pull request #10582 for [INFRA-19670] Add .asf.yaml for Github
------------------------------------------
[...truncated 72.60 KB...]
+ metadata+=hadoop-jar-url=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar
+ [[ -n gcr.io/apache-beam-testing/beam_portability/java_sdk:latest ]]
+ metadata+=,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest ]]
+ metadata+=,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest
+ local image_version=1.2
+ echo 'Starting dataproc cluster. Dataproc version: 1.2'
Starting dataproc cluster. Dataproc version: 1.2
+ local num_dataproc_workers=17
+ gcloud dataproc clusters create streaming-2 --region=global --num-workers=17 --initialization-actions gs://beam-flink-cluster/init-actions/docker.sh,gs://beam-flink-cluster/init-actions/beam.sh,gs://beam-flink-cluster/init-actions/flink.sh --metadata flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz,flink-start-yarn-session=true,flink-taskmanager-slots=1,hadoop-jar-url=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest, --image-version=1.2 --zone=us-central1-a --quiet
Waiting on operation [projects/apache-beam-testing/regions/global/operations/2a92c56b-c52a-3afc-8900-c6bafa85a6bb].
Waiting for cluster creation operation...
WARNING: For PD-Standard without local SSDs, we strongly recommend provisioning 1TB or larger to ensure consistently high I/O performance. See https://cloud.google.com/compute/docs/disks/performance for information on disk I/O performance.
................................................................................................................................................................................done.
Created [https://dataproc.googleapis.com/v1/projects/apache-beam-testing/regions/global/clusters/streaming-2] Cluster placed in zone [us-central1-a].
+ get_leader
+ local i=0
+ local application_ids
+ local application_masters
+ echo 'Yarn Applications'
Yarn Applications
++ gcloud compute ssh --zone=us-central1-a --quiet yarn@streaming-2-m '--command=yarn application -list'
++ grep streaming-2
Warning: Permanently added 'compute.3774843644762306931' (ECDSA) to the list of known hosts.
20/01/15 12:40:46 INFO client.RMProxy: Connecting to ResourceManager at streaming-2-m/10.128.1.21:8032
+ read line
+ echo application_1579091962043_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://streaming-2-w-3.c.apache-beam-testing.internal:37227
application_1579091962043_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://streaming-2-w-3.c.apache-beam-testing.internal:37227
++ echo application_1579091962043_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://streaming-2-w-3.c.apache-beam-testing.internal:37227
++ sed 's/ .*//'
+ application_ids[$i]=application_1579091962043_0001
++ echo application_1579091962043_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://streaming-2-w-3.c.apache-beam-testing.internal:37227
++ sed 's/.*streaming-2/streaming-2/'
++ sed 's/ .*//'
+ application_masters[$i]=streaming-2-w-3.c.apache-beam-testing.internal:37227
+ i=1
+ read line
+ '[' 1 '!=' 1 ']'
+ YARN_APPLICATION_MASTER=streaming-2-w-3.c.apache-beam-testing.internal:37227
+ echo 'Using Yarn Application master: streaming-2-w-3.c.apache-beam-testing.internal:37227'
Using Yarn Application master: streaming-2-w-3.c.apache-beam-testing.internal:37227
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest ]]
+ start_job_server
+ gcloud compute ssh --zone=us-central1-a --quiet yarn@streaming-2-m '--command=sudo --user yarn docker run --detach --publish 8099:8099 --publish 8098:8098 --publish 8097:8097 --volume ~/.config/gcloud:/root/.config/gcloud gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest --flink-master=streaming-2-w-3.c.apache-beam-testing.internal:37227 --artifacts-dir=gs://beam-flink-cluster/streaming-2'
fde6d01fb6f4cf67795b9ee8a83c8156a820e592e2a5c6a5c457984b81f7cc3e
+ start_tunnel
++ gcloud compute ssh --quiet --zone=us-central1-a yarn@streaming-2-m '--command=curl -s "http://streaming-2-w-3.c.apache-beam-testing.internal:37227/jobmanager/config"'
+ local 'job_server_config=[{"key":"web.port","value":"0"},{"key":"jobmanager.execution.failover-strategy","value":"region"},{"key":"high-availability.cluster-id","value":"application_1579091962043_0001"},{"key":"jobmanager.rpc.address","value":"streaming-2-w-3.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"FLINK_PLUGINS_DIR","value":"/usr/lib/flink/plugins"},{"key":"web.tmpdir","value":"/tmp/flink-web-eb0ba881-e1ad-4bb3-955e-d982a7cabec0"},{"key":"jobmanager.rpc.port","value":"40557"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1579091962043_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"16"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"streaming-2-w-3.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]'
+ local key=jobmanager.rpc.port
++ echo streaming-2-w-3.c.apache-beam-testing.internal:37227
++ cut -d : -f1
+ local yarn_application_master_host=streaming-2-w-3.c.apache-beam-testing.internal
++ echo '[{"key":"web.port","value":"0"},{"key":"jobmanager.execution.failover-strategy","value":"region"},{"key":"high-availability.cluster-id","value":"application_1579091962043_0001"},{"key":"jobmanager.rpc.address","value":"streaming-2-w-3.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"FLINK_PLUGINS_DIR","value":"/usr/lib/flink/plugins"},{"key":"web.tmpdir","value":"/tmp/flink-web-eb0ba881-e1ad-4bb3-955e-d982a7cabec0"},{"key":"jobmanager.rpc.port","value":"40557"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1579091962043_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"16"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"streaming-2-w-3.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]'
++ python -c 'import sys, json; print [ e['\''value'\''] for e in json.load(sys.stdin) if e['\''key'\''] == u'\''jobmanager.rpc.port'\''][0]'
+ local jobmanager_rpc_port=40557
++ [[ true == \t\r\u\e ]]
++ echo ' -Nf >& /dev/null'
+ local 'detached_mode_params= -Nf >& /dev/null'
++ [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest ]]
++ echo '-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097'
+ local 'job_server_ports_forwarding=-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097'
+ local 'tunnel_command=gcloud compute ssh --zone=us-central1-a --quiet yarn@streaming-2-m -- -L 8081:streaming-2-w-3.c.apache-beam-testing.internal:37227 -L 40557:streaming-2-w-3.c.apache-beam-testing.internal:40557 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf >& /dev/null'
+ eval gcloud compute ssh --zone=us-central1-a --quiet yarn@streaming-2-m -- -L 8081:streaming-2-w-3.c.apache-beam-testing.internal:37227 -L 40557:streaming-2-w-3.c.apache-beam-testing.internal:40557 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf '>&' /dev/null
++ gcloud compute ssh --zone=us-central1-a --quiet yarn@streaming-2-m -- -L 8081:streaming-2-w-3.c.apache-beam-testing.internal:37227 -L 40557:streaming-2-w-3.c.apache-beam-testing.internal:40557 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf
[beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins2728798111001035835.sh
+ echo src Load test: fanout 4 times with 2GB 10-byte records total on Flink in Portable mode src
src Load test: fanout 4 times with 2GB 10-byte records total on Flink in Portable mode src
[Gradle] - Launching build.
[src] $ <https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch/ws/src/gradlew> -PloadTest.mainClass=org.apache.beam.sdk.loadtests.CombineLoadTest -Prunner=:runners:portability:java '-PloadTest.args=--project=apache-beam-testing --appName=load_tests_Java_Portable_Flink_batch_Combine_4 --tempLocation=gs://temp-storage-for-perf-tests/loadtests --publishToBigQuery=true --bigQueryDataset=load_test --bigQueryTable=java_portable_flink_batch_Combine_4 --sourceOptions={"numRecords":5000000,"keySizeBytes":10,"valueSizeBytes":90} --fanout=4 --iterations=1 --topCount=20 --sdkWorkerParallelism=16 --perKeyCombiner=TOP_LARGEST --streaming=false --jobEndpoint=localhost:8099 --defaultEnvironmentConfig=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest --defaultEnvironmentType=DOCKER --runner=PortableRunner' --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g :sdks:java:testing:load-tests:run
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy UP-TO-DATE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources UP-TO-DATE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:spotlessGroovy UP-TO-DATE
> Task :buildSrc:spotlessGroovyCheck UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradle UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE
> Task :buildSrc:spotlessCheck UP-TO-DATE
> Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:validateTaskProperties UP-TO-DATE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE
Configuration on demand is an incubating feature.
> Task :sdks:java:core:generateAvroProtocol NO-SOURCE
> Task :runners:core-construction-java:processResources NO-SOURCE
> Task :vendor:sdks-java-extensions-protobuf:processResources NO-SOURCE
> Task :model:job-management:extractProto UP-TO-DATE
> Task :sdks:java:fn-execution:processResources NO-SOURCE
> Task :sdks:java:extensions:google-cloud-platform-core:processResources NO-SOURCE
> Task :runners:core-java:processResources NO-SOURCE
> Task :sdks:java:harness:processResources NO-SOURCE
> Task :model:fn-execution:extractProto UP-TO-DATE
> Task :runners:java-fn-execution:processResources NO-SOURCE
> Task :runners:direct-java:processResources NO-SOURCE
> Task :runners:local-java:processResources NO-SOURCE
> Task :runners:portability:java:processResources NO-SOURCE
> Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE
> Task :model:job-management:processResources UP-TO-DATE
> Task :sdks:java:core:generateAvroJava NO-SOURCE
> Task :sdks:java:extensions:protobuf:extractProto UP-TO-DATE
> Task :sdks:java:io:kafka:processResources NO-SOURCE
> Task :sdks:java:io:synthetic:processResources NO-SOURCE
> Task :sdks:java:extensions:protobuf:processResources NO-SOURCE
> Task :model:fn-execution:processResources UP-TO-DATE
> Task :sdks:java:io:kinesis:processResources NO-SOURCE
> Task :sdks:java:testing:load-tests:processResources NO-SOURCE
> Task :sdks:java:testing:test-utils:processResources NO-SOURCE
> Task :sdks:java:core:generateGrammarSource UP-TO-DATE
> Task :sdks:java:core:processResources UP-TO-DATE
> Task :model:pipeline:extractIncludeProto UP-TO-DATE
> Task :model:pipeline:extractProto UP-TO-DATE
> Task :model:pipeline:generateProto UP-TO-DATE
> Task :model:pipeline:compileJava UP-TO-DATE
> Task :model:pipeline:processResources UP-TO-DATE
> Task :model:pipeline:classes UP-TO-DATE
> Task :model:pipeline:jar UP-TO-DATE
> Task :model:pipeline:shadowJar UP-TO-DATE
> Task :model:fn-execution:extractIncludeProto UP-TO-DATE
> Task :model:job-management:extractIncludeProto UP-TO-DATE
> Task :model:job-management:generateProto UP-TO-DATE
> Task :model:fn-execution:generateProto UP-TO-DATE
> Task :model:job-management:compileJava UP-TO-DATE
> Task :model:job-management:classes UP-TO-DATE
> Task :model:fn-execution:compileJava UP-TO-DATE
> Task :model:fn-execution:classes UP-TO-DATE
> Task :model:job-management:shadowJar UP-TO-DATE
> Task :model:fn-execution:shadowJar UP-TO-DATE
> Task :sdks:java:core:compileJava UP-TO-DATE
> Task :sdks:java:core:classes UP-TO-DATE
> Task :sdks:java:core:shadowJar UP-TO-DATE
> Task :sdks:java:extensions:protobuf:extractIncludeProto UP-TO-DATE
> Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE
> Task :runners:core-construction-java:compileJava UP-TO-DATE
> Task :runners:core-construction-java:classes UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:compileJava UP-TO-DATE
> Task :sdks:java:fn-execution:compileJava UP-TO-DATE
> Task :runners:local-java:compileJava FROM-CACHE
> Task :vendor:sdks-java-extensions-protobuf:compileJava UP-TO-DATE
> Task :runners:local-java:classes UP-TO-DATE
> Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE
> Task :sdks:java:fn-execution:classes UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE
> Task :sdks:java:fn-execution:jar UP-TO-DATE
> Task :vendor:sdks-java-extensions-protobuf:shadowJar UP-TO-DATE
> Task :runners:core-construction-java:jar UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE
> Task :runners:local-java:jar
> Task :sdks:java:extensions:protobuf:compileJava UP-TO-DATE
> Task :sdks:java:extensions:protobuf:classes UP-TO-DATE
> Task :sdks:java:extensions:protobuf:jar UP-TO-DATE
> Task :runners:core-java:compileJava UP-TO-DATE
> Task :runners:core-java:classes UP-TO-DATE
> Task :runners:core-java:jar UP-TO-DATE
> Task :sdks:java:harness:compileJava UP-TO-DATE
> Task :sdks:java:harness:classes UP-TO-DATE
> Task :sdks:java:harness:jar UP-TO-DATE
> Task :sdks:java:harness:shadowJar UP-TO-DATE
> Task :runners:java-fn-execution:compileJava UP-TO-DATE
> Task :runners:java-fn-execution:classes UP-TO-DATE
> Task :runners:java-fn-execution:jar UP-TO-DATE
> Task :runners:portability:java:compileJava FROM-CACHE
> Task :runners:portability:java:classes UP-TO-DATE
> Task :runners:direct-java:compileJava FROM-CACHE
> Task :runners:direct-java:classes UP-TO-DATE
> Task :runners:portability:java:jar
> Task :sdks:java:io:synthetic:compileJava FROM-CACHE
> Task :sdks:java:io:synthetic:classes UP-TO-DATE
> Task :sdks:java:io:synthetic:jar
> Task :sdks:java:io:kafka:compileJava UP-TO-DATE
> Task :sdks:java:io:kafka:classes UP-TO-DATE
> Task :sdks:java:io:kafka:jar UP-TO-DATE
> Task :sdks:java:testing:test-utils:compileJava FROM-CACHE
> Task :sdks:java:testing:test-utils:classes UP-TO-DATE
> Task :sdks:java:testing:test-utils:jar
> Task :sdks:java:io:google-cloud-platform:compileJava UP-TO-DATE
> Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE
> Task :sdks:java:io:google-cloud-platform:jar UP-TO-DATE
> Task :sdks:java:io:kinesis:compileJava FROM-CACHE
> Task :sdks:java:io:kinesis:classes UP-TO-DATE
> Task :sdks:java:io:kinesis:jar
> Task :runners:direct-java:shadowJar
> Task :sdks:java:testing:load-tests:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :sdks:java:testing:load-tests:classes
> Task :sdks:java:testing:load-tests:jar
> Task :sdks:java:testing:load-tests:run
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: The Runner experienced the following error during execution:
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_e01_1579091962043_0001_01_000002 timed out.
at org.apache.beam.runners.portability.JobServicePipelineResult.waitUntilFinish(JobServicePipelineResult.java:98)
at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:99)
at org.apache.beam.sdk.loadtests.CombineLoadTest.run(CombineLoadTest.java:66)
at org.apache.beam.sdk.loadtests.CombineLoadTest.main(CombineLoadTest.java:169)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: The Runner experienced the following error during execution:
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_e01_1579091962043_0001_01_000002 timed out.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at org.apache.beam.runners.portability.JobServicePipelineResult.waitUntilFinish(JobServicePipelineResult.java:90)
... 3 more
Caused by: java.lang.RuntimeException: The Runner experienced the following error during execution:
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_e01_1579091962043_0001_01_000002 timed out.
at org.apache.beam.runners.portability.JobServicePipelineResult.propagateErrors(JobServicePipelineResult.java:165)
at org.apache.beam.runners.portability.JobServicePipelineResult.waitUntilFinish(JobServicePipelineResult.java:110)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> Task :sdks:java:testing:load-tests:run FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:load-tests:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 5m 23s
61 actionable tasks: 9 executed, 6 from cache, 46 up-to-date
Publishing build scan...
https://gradle.com/s/j6ydgla4wctqa
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal :
beam_LoadTests_Java_Combine_Portable_Flink_Batch #4
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch/4/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Java_Combine_Portable_Flink_Batch #3
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch/3/display/redirect?page=changes>
Changes:
[ehudm] [BEAM-8525] Support Const base in binary_subscr
[ehudm] Do not perform test on Py2.7
[echauchot] [BEAM-9065] Reset MetricsContainerStepMapAccumulator upon initialization
[kamil.wasilewski] [BEAM-8108] Allow run_chicago.sh to take variable argument number
[kamil.wasilewski] [BEAM-8108] Use _ReadFromBigQuery transform
[kamil.wasilewski] [BEAM-8108] Create Jenkins job that runs Chicago Taxi Example on Flink
[kamil.wasilewski] [BEAM-8108] Rewrite Python print statement in dataproc init action
[github] Update release guide with extended information.
[github] Add link to mass comment script
[github] Fix typo
[ankurgoenka] [BEAM-9002] Add test_flatten_same_pcollections to fnapi runner
[github] Address comments
[hannahjiang] [BEAM-9084] cleaning up docker image tag
[iemejia] [BEAM-5086] Update kudu-client to version 1.11.1
[valentyn] Add version guards to requirements file for integration tests.
[robertwb] Automatically convert to with pipeline syntax.
[robertwb] Quick pass through failed auto-conversions.
[robertwb] Automatic conversion of more pipelines.
[robertwb] Fix lint and tests due to autoconversion.
[robertwb] A couple more conversions.
[robertwb] Fix lint and tests due to autoconversion.
[robertwb] Return non-None result for Dataflow dry run.
[robertwb] Fix lint and tests due to autoconversion.
[hannahjiang] [BEAM-9084] fix Java spotless
[kamil.wasilewski] [BEAM-8939] A bash script that cancels stale dataflow jobs
[chadrik] [BEAM-7746] Fix a typing issue where SourceBase was assumed to have a
[marek.simunek] [BEAM-9123] HadoopResourceId returns wrong directoryName bugfix
[iemejia] [website] Added security page
[iemejia] [website] Update the 2.17.0 release blog post to include security issues
[boyuanz] Moving to 2.20.0-SNAPSHOT on master branch.
[github] Update twine install details.
[crites] Changes watermark advance from 1001 to 1000 since Dataflow TestStream
[lukecwik] [BEAM-8676] sdks/java: gax and grpc upgrades (#10554)
[lukecwik] [BEAM-9030] Migrate Beam to use beam-vendor-grpc-1_26_0 (#10578)
[lcwik] [BEAM-9030] Align version of protoc/protoc-gen-grpc-java to vendored
[kirillkozlov] Create initial DataStoreV1Table
[kirillkozlov] Added tests
[kirillkozlov] Implement getTableStatistics
[kirillkozlov] buildIOWriter
[kirillkozlov] Read other types
[kirillkozlov] Better conversion, support complex types
[kirillkozlov] Store DataStore key as VARBINARY
[kirillkozlov] Wrap DoFns in PTransforms
[kirillkozlov] Table property for specifying key field
[kirillkozlov] JavaDoc
[kirillkozlov] Infer schema for RowToEntity
[kirillkozlov] Better conversion performance
[kirillkozlov] Mark Table as `Internal` and `Experimental`
[kirillkozlov] Review changes
[boyuanz] Exclude testOutputTimestamp from flink streaming tests.
[lukecwik] [BEAM-7951] Supports multiple inputs/outputs for wire coder settings.
[mxm] [BEAM-9116] Limit the number of past invocations stored in JobService
[kirillkozlov] Add IT that does not rely on SQL
[lukecwik] [BEAM-9124] Linkage Checker 1.1.2 to use Maven Central HTTPS URL
[robertwb] lint, reviewer comments
[relax] Merge pull request #10316: [BEAM-6857] Support Dynamic Timers
[robertwb] lint
[github] Merge pull request #10577 from Adding Python test for ReadFromBigQuery
[kirillkozlov] fix style
[github] [BEAM-9127] Fix output type declaration in xlang wordcount. (#10605)
[robertwb] fix merge
[apilloud] [BEAM-9140] Upgrade to ZetaSQL 2020.01.1
[angoenka] [BEAM-8625] Implement servlet for exposing sdk harness statuses in Da…
[iemejia] [BEAM-9143] Make RedisIO code follow standard Beam conventions
[iemejia] [BEAM-9143] Add withOutputParallelization to RedisIO.Read/ReadAll
[suztomo] Beam's own TimestampConversion
[suztomo] Link in Javadoc
[ehudm] Migrate HDFS IT to use tox env.
[ehudm] Fix for py2
[lukecwik] [BEAM-8695] Upgrade google-http-client to 1.34.0 (#10614)
[rohde.samuel] Make apply_TestStream not a test b/c Nose thinks it is a test
[chamikara] [BEAM-7246] Add Google Spanner IO Read on Python SDK (#9606)
[boyuanz] [BEAM-9151] Fix misconfigured legacy dataflow tests.
[hannahjiang] [BEAM-7861] update documentation
[github] Merge pull request #10622: [BEAM=6857] Fix timermap test to not use
[sunjincheng121] [BEAM-9153] Fix release guide heading level
[kamil.wasilewski] Report status code 0 when no stale jobs are found
[hsuryawirawan] Update to Python 3.7 and upgrade to Beam v2.17.0
[hsuryawirawan] Change "Built-in IOs" task type to "theory"
[hsuryawirawan] Remove "run_common_tests()" in all tests.py to make it work with Python
[hsuryawirawan] Update the course on Stepik
[hsuryawirawan] Add back the license header
[github] Fix typo in 2017-02-13-stateful-processing.md
[github] cleanup typo
[github] [BEAM-9120] Make WordCountsMatcher extend
[github] Revert "Merge pull request #10582 for [INFRA-19670] Add .asf.yaml for
[angoenka] [BEAM-9122] Add uses_keyed_state step property in python dataflow run…
[github] [BEAM-9120] Make BigqueryMatcher extend TypeSafeMatcher<TableAndQuery>
[github] [BEAM-9120] Make FileChecksumMatcher extend TypeSafeMatcher<ShardedFile>
[pabloem] Merge pull request #10459 from [BEAM-9029]Fix two bugs in Python SDK S3
[lcwik] [BEAM-3419] Support iterable on Dataflow runner when using the unified
[ehudm] [BEAM-9168] Temporary fix for RunnerAPIPTransformHolder usage
[keijiy] Fix comments on bigquery.py * beam.io.gcp.WriteToBigQuery ->
[github] Merge pull request #10534: Beam-2535: Support timer output timestamps in
[github] [BEAM-8992] ignore Go Vet failures. (#10661)
[github] [BEAM-4725] Preallocate CoGBK value buffers (#10660)
[kirillkozlov] Use custom escape method
[github] typehint fixes to DoOutputsTuple (#10494)
[github] [BEAM-3419, BEAM-9173] Add TODO comment
[github] [BEAM-9167] Reduce Go SDK metric overhead (#10654)
[iemejia] [BEAM-9172] Add missing parameter to Nexmark CI execution for Flink
[github] Chnage dataflow container version
[hannahjiang] [BEAM-9084] cleaning up docker image tag
[iemejia] Make Spark use of batch based invocation in Nexmark CI jobs cleaner
[kirillkozlov] Inline, link JIRA
[ehudm] Update beam website for 2.18.0
[apilloud] [BEAM-9027] Unparse LIKE as binary operator
[github] Blog post for release 2.18.0 (#10575)
[kcweaver] upgrade auto-value to version 1.7
[kcweaver] [BEAM-9149] Add SQL query parameters to public API and enable positional
[bhulette] [BEAM-9072] [SQL] Primitive types should fall through (#10681)
[elias.djurfeldt] Added valueprovider support for Datastore query namespaces
[iemejia] [BEAM-9170] Unify Jenkins job names related to Java 11 testing
[github] Add a minor comment.
[aaltay] Bump tensorflow from 1.14.0 to 1.15.0 in /sdks/python/container (#10392)
[github] [BEAM-9093] Log invalid overwrites in pipeline options (#10613)
[github] [BEAM-8492] Allow None, Optional return hints for DoFn.process and
------------------------------------------
[...truncated 39.81 KB...]
03ff63c55220: Preparing
bee1e39d7c3a: Preparing
1f59a4b2e206: Preparing
0ca7f54856c0: Preparing
ebb9ae013834: Preparing
0ca7f54856c0: Waiting
619b5df84312: Waiting
e7fe5541de5f: Waiting
03ff63c55220: Waiting
a6ded049566a: Waiting
ebb9ae013834: Waiting
1f59a4b2e206: Waiting
bee1e39d7c3a: Waiting
7c13b2965ae4: Pushed
468df377a9d9: Pushed
14421810ded7: Pushed
a6ded049566a: Layer already exists
e7fe5541de5f: Layer already exists
03ff63c55220: Layer already exists
bee1e39d7c3a: Layer already exists
1f59a4b2e206: Layer already exists
0ca7f54856c0: Layer already exists
ebb9ae013834: Layer already exists
619b5df84312: Pushed
b9b02e4fbbee: Pushed
88f235ed14ee: Pushed
latest: digest: sha256:f1fa2c52e57c24f525e08f7dc5fe377f06bb134fe20af909f11790e38f884ee4 size: 3056
[Gradle] - Launching build.
[src] $ <https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch/ws/src/gradlew> --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g -Pdocker-repository-root=gcr.io/apache-beam-testing/beam_portability -Pdocker-tag=latest :runners:flink:1.9:job-server-container:docker
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy UP-TO-DATE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources UP-TO-DATE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:spotlessGroovy UP-TO-DATE
> Task :buildSrc:spotlessGroovyCheck UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradle UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE
> Task :buildSrc:spotlessCheck UP-TO-DATE
> Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:validateTaskProperties UP-TO-DATE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE
Configuration on demand is an incubating feature.
> Task :sdks:java:fn-execution:processResources NO-SOURCE
> Task :sdks:java:extensions:google-cloud-platform-core:processResources NO-SOURCE
> Task :model:job-management:extractProto UP-TO-DATE
> Task :sdks:java:harness:processResources NO-SOURCE
> Task :runners:core-construction-java:processResources NO-SOURCE
> Task :runners:core-java:processResources NO-SOURCE
> Task :sdks:java:core:generateAvroProtocol NO-SOURCE
> Task :model:fn-execution:extractProto UP-TO-DATE
> Task :vendor:sdks-java-extensions-protobuf:processResources NO-SOURCE
> Task :runners:java-fn-execution:processResources NO-SOURCE
> Task :runners:flink:1.9:copyResourcesOverrides NO-SOURCE
> Task :runners:flink:1.9:job-server:processResources NO-SOURCE
> Task :sdks:java:core:generateAvroJava NO-SOURCE
> Task :sdks:java:io:kafka:processResources NO-SOURCE
> Task :sdks:java:extensions:protobuf:extractProto
> Task :runners:flink:1.9:job-server-container:dockerClean UP-TO-DATE
> Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE
> Task :sdks:java:extensions:protobuf:processResources NO-SOURCE
> Task :model:fn-execution:processResources UP-TO-DATE
> Task :model:job-management:processResources UP-TO-DATE
> Task :sdks:java:core:generateGrammarSource UP-TO-DATE
> Task :sdks:java:core:processResources UP-TO-DATE
> Task :runners:flink:1.9:copySourceOverrides
> Task :runners:flink:1.9:copyTestResourcesOverrides NO-SOURCE
> Task :model:pipeline:extractIncludeProto UP-TO-DATE
> Task :sdks:java:build-tools:compileJava FROM-CACHE
> Task :model:pipeline:extractProto UP-TO-DATE
> Task :runners:flink:1.9:processResources
> Task :sdks:java:build-tools:processResources
> Task :sdks:java:build-tools:classes
> Task :model:pipeline:generateProto UP-TO-DATE
> Task :sdks:java:build-tools:jar
> Task :model:pipeline:compileJava UP-TO-DATE
> Task :model:pipeline:processResources UP-TO-DATE
> Task :model:pipeline:classes UP-TO-DATE
> Task :model:pipeline:jar UP-TO-DATE
> Task :model:pipeline:shadowJar UP-TO-DATE
> Task :model:job-management:extractIncludeProto UP-TO-DATE
> Task :model:fn-execution:extractIncludeProto UP-TO-DATE
> Task :model:job-management:generateProto UP-TO-DATE
> Task :model:fn-execution:generateProto UP-TO-DATE
> Task :model:job-management:compileJava UP-TO-DATE
> Task :model:job-management:classes UP-TO-DATE
> Task :model:job-management:shadowJar UP-TO-DATE
> Task :model:fn-execution:compileJava UP-TO-DATE
> Task :model:fn-execution:classes UP-TO-DATE
> Task :model:fn-execution:shadowJar UP-TO-DATE
> Task :sdks:java:core:compileJava UP-TO-DATE
> Task :sdks:java:core:classes UP-TO-DATE
> Task :sdks:java:core:shadowJar UP-TO-DATE
> Task :vendor:sdks-java-extensions-protobuf:compileJava FROM-CACHE
> Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE
> Task :sdks:java:fn-execution:compileJava UP-TO-DATE
> Task :sdks:java:fn-execution:classes UP-TO-DATE
> Task :sdks:java:io:kafka:compileJava UP-TO-DATE
> Task :sdks:java:io:kafka:classes UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:compileJava UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE
> Task :sdks:java:fn-execution:jar UP-TO-DATE
> Task :sdks:java:io:kafka:jar UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE
> Task :runners:core-construction-java:compileJava UP-TO-DATE
> Task :runners:core-construction-java:classes UP-TO-DATE
> Task :runners:core-construction-java:jar UP-TO-DATE
> Task :vendor:sdks-java-extensions-protobuf:shadowJar
> Task :runners:core-java:compileJava UP-TO-DATE
> Task :runners:core-java:classes UP-TO-DATE
> Task :runners:core-java:jar UP-TO-DATE
> Task :sdks:java:harness:compileJava UP-TO-DATE
> Task :sdks:java:harness:classes UP-TO-DATE
> Task :sdks:java:harness:jar
> Task :sdks:java:harness:shadowJar UP-TO-DATE
> Task :sdks:java:extensions:protobuf:extractIncludeProto
> Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE
> Task :sdks:java:extensions:protobuf:compileJava FROM-CACHE
> Task :sdks:java:extensions:protobuf:classes UP-TO-DATE
> Task :sdks:java:extensions:protobuf:jar
> Task :sdks:java:io:google-cloud-platform:compileJava FROM-CACHE
> Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE
> Task :sdks:java:io:google-cloud-platform:jar
> Task :runners:java-fn-execution:compileJava FROM-CACHE
> Task :runners:java-fn-execution:classes UP-TO-DATE
> Task :runners:java-fn-execution:jar
> Task :runners:flink:1.9:compileJava FROM-CACHE
> Task :runners:flink:1.9:classes
> Task :runners:flink:1.9:jar
> Task :runners:flink:1.9:job-server:compileJava NO-SOURCE
> Task :runners:flink:1.9:job-server:classes UP-TO-DATE
> Task :runners:flink:1.9:job-server:shadowJar
> Task :runners:flink:1.9:job-server-container:copyDockerfileDependencies
> Task :runners:flink:1.9:job-server-container:dockerPrepare
> Task :runners:flink:1.9:job-server-container:docker
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 58s
58 actionable tasks: 16 executed, 6 from cache, 36 up-to-date
Publishing build scan...
https://gradle.com/s/kqv7h3bws2wdq
[beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins3680403557783725952.sh
+ echo 'Tagging image...'
Tagging image...
[beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins3095438528072116417.sh
+ docker tag gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest
[beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins4489883657229385944.sh
+ echo 'Pushing image...'
Pushing image...
[beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins7217377338939886267.sh
+ docker push gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest
The push refers to repository [gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server]
a093f3591473: Preparing
b191cd8defc8: Preparing
e35c185253ca: Preparing
a6ded049566a: Preparing
e7fe5541de5f: Preparing
03ff63c55220: Preparing
bee1e39d7c3a: Preparing
1f59a4b2e206: Preparing
0ca7f54856c0: Preparing
ebb9ae013834: Preparing
03ff63c55220: Waiting
bee1e39d7c3a: Waiting
1f59a4b2e206: Waiting
0ca7f54856c0: Waiting
ebb9ae013834: Waiting
a6ded049566a: Layer already exists
e7fe5541de5f: Layer already exists
03ff63c55220: Layer already exists
bee1e39d7c3a: Layer already exists
0ca7f54856c0: Layer already exists
1f59a4b2e206: Layer already exists
ebb9ae013834: Layer already exists
a093f3591473: Pushed
e35c185253ca: Pushed
b191cd8defc8: Pushed
latest: digest: sha256:2b1253585340ccbaf1e0d86ef4cfe8afe5727bf1d5f5811014e0fbb0763fdb50 size: 2427
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties content
JOB_SERVER_IMAGE=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest
CLUSTER_NAME=streaming-3
DETACHED_MODE=true
HARNESS_IMAGES_TO_PULL=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest
FLINK_NUM_WORKERS=16
FLINK_DOWNLOAD_URL=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz
GCS_BUCKET=gs://beam-flink-cluster
HADOOP_DOWNLOAD_URL=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar
FLINK_TASKMANAGER_SLOTS=1
ARTIFACTS_DIR=gs://beam-flink-cluster/streaming-3
GCLOUD_ZONE=us-central1-a
[EnvInject] - Variables injected successfully.
[beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins5953650193304024910.sh
+ echo Setting up flink cluster
Setting up flink cluster
[beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins319433620560313628.sh
+ cd <https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch/ws/src/.test-infra/dataproc>
+ ./flink_cluster.sh create
+ GCLOUD_ZONE=us-central1-a
+ DATAPROC_VERSION=1.2
+ MASTER_NAME=streaming-3-m
+ INIT_ACTIONS_FOLDER_NAME=init-actions
+ FLINK_INIT=gs://beam-flink-cluster/init-actions/flink.sh
+ BEAM_INIT=gs://beam-flink-cluster/init-actions/beam.sh
+ DOCKER_INIT=gs://beam-flink-cluster/init-actions/docker.sh
+ FLINK_LOCAL_PORT=8081
+ FLINK_TASKMANAGER_SLOTS=1
+ TASK_MANAGER_MEM=10240
+ YARN_APPLICATION_MASTER=
+ create
+ upload_init_actions
+ echo 'Uploading initialization actions to GCS bucket: gs://beam-flink-cluster'
Uploading initialization actions to GCS bucket: gs://beam-flink-cluster
+ gsutil cp -r init-actions/beam.sh init-actions/docker.sh init-actions/flink.sh gs://beam-flink-cluster/init-actions
Copying file://init-actions/beam.sh [Content-Type=text/x-sh]...
/ [0 files][ 0.0 B/ 2.3 KiB] / [1 files][ 2.3 KiB/ 2.3 KiB] Copying file://init-actions/docker.sh [Content-Type=text/x-sh]...
/ [1 files][ 2.3 KiB/ 6.0 KiB] / [2 files][ 6.0 KiB/ 6.0 KiB] Copying file://init-actions/flink.sh [Content-Type=text/x-sh]...
/ [2 files][ 6.0 KiB/ 13.4 KiB] / [3 files][ 13.4 KiB/ 13.4 KiB]
Operation completed over 3 objects/13.4 KiB.
+ create_cluster
+ local metadata=flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz,
+ metadata+=flink-start-yarn-session=true,
+ metadata+=flink-taskmanager-slots=1,
+ metadata+=hadoop-jar-url=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar
+ [[ -n gcr.io/apache-beam-testing/beam_portability/java_sdk:latest ]]
+ metadata+=,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest ]]
+ metadata+=,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest
+ local image_version=1.2
+ echo 'Starting dataproc cluster. Dataproc version: 1.2'
Starting dataproc cluster. Dataproc version: 1.2
+ local num_dataproc_workers=17
+ gcloud dataproc clusters create streaming-3 --region=global --num-workers=17 --initialization-actions gs://beam-flink-cluster/init-actions/docker.sh,gs://beam-flink-cluster/init-actions/beam.sh,gs://beam-flink-cluster/init-actions/flink.sh --metadata flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz,flink-start-yarn-session=true,flink-taskmanager-slots=1,hadoop-jar-url=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest, --image-version=1.2 --zone=us-central1-a --quiet
ERROR: (gcloud.dataproc.clusters.create) ALREADY_EXISTS: Already exists: Failed to create cluster: Cluster projects/apache-beam-testing/regions/global/clusters/streaming-3
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org