You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2020/11/16 02:09:45 UTC
beam_PerformanceTests_Kafka_IO - Build # 1559 - Aborted!
beam_PerformanceTests_Kafka_IO - Build # 1559 - Aborted:
Check console output at https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1559/ to view the results.
Jenkins build is back to normal : beam_PerformanceTests_Kafka_IO
#1568
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1568/display/redirect>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_Kafka_IO #1567
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1567/display/redirect?page=changes>
Changes:
[heejong] [BEAM-11270] Dataflow Java on runner v2 tests are failing because sdk
[Udi Meiri] Update Beam Dataflow container versions for Python
------------------------------------------
[...truncated 1.07 MB...]
Nov 17, 2020 6:57:08 AM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
Nov 17, 2020 6:57:09 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 218 files. Enable logging at DEBUG level to see which files will be staged.
Nov 17, 2020 6:57:09 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
Nov 17, 2020 6:57:11 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 218 files from PipelineOptions.filesToStage to staging location to prepare for execution.
Nov 17, 2020 6:57:12 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 218 files cached, 0 files newly uploaded in 0 seconds
Nov 17, 2020 6:57:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Generate records/Impulse as step s1
Nov 17, 2020 6:57:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Generate records/ParDo(OutputSingleSource) as step s2
Nov 17, 2020 6:57:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Generate records/ParDo(BoundedSourceAsSDFWrapper) as step s3
Nov 17, 2020 6:57:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Measure write time as step s4
Nov 17, 2020 6:57:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to Kafka/Kafka ProducerRecord/Map as step s5
Nov 17, 2020 6:57:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to Kafka/KafkaIO.WriteRecords/ParDo(KafkaWriter) as step s6
Nov 17, 2020 6:57:12 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://dataflow-staging-us-central1-844138762903/temp/staging/
Nov 17, 2020 6:57:12 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <97742 bytes, hash b9576dc1c5907e1bdfbebfbcc1b169541d597953d280f239c7e851c500a970f5> to gs://dataflow-staging-us-central1-844138762903/temp/staging/pipeline-uVdtwcWQfhvfvr-8wbFpVB1ZeVPSgPI5x-hRxQCpcPU.pb
Nov 17, 2020 6:57:12 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.27.0-SNAPSHOT
Nov 17, 2020 6:57:19 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-11-16_22_57_12-12817168721422767204?project=apache-beam-testing
Nov 17, 2020 6:57:19 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2020-11-16_22_57_12-12817168721422767204
Nov 17, 2020 6:57:19 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2020-11-16_22_57_12-12817168721422767204
Nov 17, 2020 6:57:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2020-11-17T06:57:22.754Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: kafkaioit0testkafkaioreadsandwritescorrectlyinstreaming-je-mvl3. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:32.760Z: Worker configuration: n1-standard-1 in us-central1-f.
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.504Z: Expanding SplittableParDo operations into optimizable parts.
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.539Z: Expanding CollectionToSingleton operations into optimizable parts.
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.651Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.698Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.766Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.797Z: Fusing consumer Generate records/ParDo(OutputSingleSource) into Generate records/Impulse
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.839Z: Fusing consumer s3/PairWithRestriction into Generate records/ParDo(OutputSingleSource)
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.872Z: Fusing consumer s3/SplitWithSizing into s3/PairWithRestriction
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.907Z: Fusing consumer Measure write time into s3/ProcessElementAndRestrictionWithSizing
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.936Z: Fusing consumer Write to Kafka/Kafka ProducerRecord/Map into Measure write time
Nov 17, 2020 6:57:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:33.972Z: Fusing consumer Write to Kafka/KafkaIO.WriteRecords/ParDo(KafkaWriter) into Write to Kafka/Kafka ProducerRecord/Map
Nov 17, 2020 6:57:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:34.735Z: Executing operation Generate records/Impulse+Generate records/ParDo(OutputSingleSource)+s3/PairWithRestriction+s3/SplitWithSizing
Nov 17, 2020 6:57:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:34.817Z: Starting 5 ****s in us-central1-f...
Nov 17, 2020 6:57:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:57:43.354Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Nov 17, 2020 6:58:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:58:06.496Z: Autoscaling: Raised the number of ****s to 3 based on the rate of progress in the currently running stage(s).
Nov 17, 2020 6:58:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:58:06.550Z: Resized **** pool to 3, though goal was 5. This could be a quota issue.
Nov 17, 2020 6:58:16 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:58:16.865Z: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
Nov 17, 2020 6:58:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:58:29.703Z: Workers have started successfully.
Nov 17, 2020 6:58:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:58:29.733Z: Workers have started successfully.
Nov 17, 2020 6:59:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:59:18.209Z: Finished operation Generate records/Impulse+Generate records/ParDo(OutputSingleSource)+s3/PairWithRestriction+s3/SplitWithSizing
Nov 17, 2020 6:59:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:59:18.443Z: Executing operation s3/ProcessElementAndRestrictionWithSizing+Measure write time+Write to Kafka/Kafka ProducerRecord/Map+Write to Kafka/KafkaIO.WriteRecords/ParDo(KafkaWriter)
Nov 17, 2020 6:59:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:59:39.610Z: Finished operation s3/ProcessElementAndRestrictionWithSizing+Measure write time+Write to Kafka/Kafka ProducerRecord/Map+Write to Kafka/KafkaIO.WriteRecords/ParDo(KafkaWriter)
Nov 17, 2020 6:59:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:59:39.799Z: Cleaning up.
Nov 17, 2020 6:59:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T06:59:39.883Z: Stopping **** pool...
Nov 17, 2020 7:00:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:29.929Z: Autoscaling: Resized **** pool from 5 to 0.
Nov 17, 2020 7:00:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:29.977Z: Worker pool stopped.
Nov 17, 2020 7:00:35 AM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2020-11-16_22_57_12-12817168721422767204 finished with status DONE.
Nov 17, 2020 7:00:36 AM org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory tryCreateDefaultBucket
INFO: No tempLocation specified, attempting to use default bucket: dataflow-staging-us-central1-844138762903
Nov 17, 2020 7:00:36 AM org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler handleResponse
WARNING: Request failed with code 409, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://storage.googleapis.com/storage/v1/b?predefinedAcl=projectPrivate&predefinedDefaultObjectAcl=projectPrivate&project=apache-beam-testing.
Nov 17, 2020 7:00:36 AM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
Nov 17, 2020 7:00:36 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 218 files. Enable logging at DEBUG level to see which files will be staged.
Nov 17, 2020 7:00:36 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 218 files from PipelineOptions.filesToStage to staging location to prepare for execution.
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 218 files cached, 0 files newly uploaded in 0 seconds
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from unbounded Kafka/Read(KafkaUnboundedSource)/Impulse as step s1
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from unbounded Kafka/Read(KafkaUnboundedSource)/ParDo(OutputSingleSource) as step s2
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from unbounded Kafka/Read(KafkaUnboundedSource)/ParDo(UnboundedSourceAsSDFWrapper) as step s3
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from unbounded Kafka/Read(KafkaUnboundedSource)/ParDo(StripIds) as step s4
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Measure read time as step s5
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Map records to strings/Map as step s6
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Counting element as step s7
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://dataflow-staging-us-central1-844138762903/temp/staging/
Nov 17, 2020 7:00:38 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <101720 bytes, hash bc71eb5268e513e299c1c55b2a458d8d491d11d620d9df4f728ddc5a62fffb25> to gs://dataflow-staging-us-central1-844138762903/temp/staging/pipeline-vHHrUmjlE-KZwcVbKkWNjUkdEdYg2d9Pco3cWmL_-yU.pb
Nov 17, 2020 7:00:39 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.27.0-SNAPSHOT
Nov 17, 2020 7:00:40 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-11-16_23_00_39-14396083331838786362?project=apache-beam-testing
Nov 17, 2020 7:00:40 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2020-11-16_23_00_39-14396083331838786362
Nov 17, 2020 7:00:40 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2020-11-16_23_00_39-14396083331838786362
Nov 17, 2020 7:00:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2020-11-17T07:00:42.744Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: kafkaioit0testkafkaioreadsandwritescorrectlyinstreaming-je-fw0k. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:52.543Z: Worker configuration: n1-standard-4 in us-central1-f.
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.316Z: Expanding SplittableParDo operations into optimizable parts.
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.350Z: Expanding CollectionToSingleton operations into optimizable parts.
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.593Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.635Z: Expanding SplittableProcessKeyed operations into optimizable parts.
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.678Z: Expanding GroupByKey operations into streaming Read/Write steps
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.714Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.783Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.854Z: Fusing consumer Read from unbounded Kafka/Read(KafkaUnboundedSource)/ParDo(OutputSingleSource) into Read from unbounded Kafka/Read(KafkaUnboundedSource)/Impulse
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.882Z: Fusing consumer s3/PairWithRestriction into Read from unbounded Kafka/Read(KafkaUnboundedSource)/ParDo(OutputSingleSource)
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.910Z: Fusing consumer s3/SplitWithSizing into s3/PairWithRestriction
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.947Z: Fusing consumer Read from unbounded Kafka/Read(KafkaUnboundedSource)/ParDo(StripIds) into s3/ProcessElementAndRestrictionWithSizing
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:53.980Z: Fusing consumer Measure read time into Read from unbounded Kafka/Read(KafkaUnboundedSource)/ParDo(StripIds)
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:54.012Z: Fusing consumer Map records to strings/Map into Measure read time
Nov 17, 2020 7:00:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:54.037Z: Fusing consumer Counting element into Map records to strings/Map
Nov 17, 2020 7:00:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:00:55.517Z: Starting 5 ****s in us-central1-f...
Nov 17, 2020 7:01:09 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:01:08.770Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Nov 17, 2020 7:01:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:01:33.058Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
Nov 17, 2020 7:01:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:01:57.154Z: Workers have started successfully.
Nov 17, 2020 7:01:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-11-17T07:01:57.229Z: Workers have started successfully.
Nov 17, 2020 7:15:40 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
WARNING: No terminal state was returned within allotted timeout. State value RUNNING
Gradle Test Executor 3 finished executing tests.
> Task :sdks:java:io:kafka:integrationTest FAILED
org.apache.beam.sdk.io.kafka.KafkaIOIT > testKafkaIOReadsAndWritesCorrectlyInStreaming FAILED
java.lang.AssertionError: expected:<100000> but was:<200000>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.apache.beam.sdk.io.kafka.KafkaIOIT.testKafkaIOReadsAndWritesCorrectlyInStreaming(KafkaIOIT.java:144)
1 test completed, 1 failed
Finished generating test XML results (0.004 secs) into: <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/kafka/build/test-results/integrationTest>
Generating HTML test report...
Finished generating test html results (0.005 secs) into: <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/kafka/build/reports/tests/integrationTest>
:sdks:java:io:kafka:integrationTest (Thread[Execution **** for ':' Thread 8,5,main]) completed. Took 18 mins 36.235 secs.
:runners:google-cloud-dataflow-java:cleanUpDockerImages (Thread[Execution **** for ':' Thread 8,5,main]) started.
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages
Custom actions are attached to task ':runners:google-cloud-dataflow-java:cleanUpDockerImages'.
Caching disabled for task ':runners:google-cloud-dataflow-java:cleanUpDockerImages' because:
Caching has not been enabled for the task
Task ':runners:google-cloud-dataflow-java:cleanUpDockerImages' is not up-to-date because:
Task has not declared any outputs despite executing actions.
Starting process 'command 'docker''. Working directory: <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/runners/google-cloud-dataflow-java> Command: docker rmi --force us.gcr.io/apache-beam-testing/java-postcommit-it/java:20201117065201
Successfully started process 'command 'docker''
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20201117065201
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9305a3799e3d84710d7b4035bdf038e053fdd86ba57739fe4e1b7556950db3a5
Starting process 'command 'gcloud''. Working directory: <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/runners/google-cloud-dataflow-java> Command: gcloud --quiet container images delete --force-delete-tags us.gcr.io/apache-beam-testing/java-postcommit-it/java:20201117065201
Successfully started process 'command 'gcloud''
Digests:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9305a3799e3d84710d7b4035bdf038e053fdd86ba57739fe4e1b7556950db3a5
Associated tags:
- 20201117065201
Tags:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java:20201117065201
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20201117065201].
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9305a3799e3d84710d7b4035bdf038e053fdd86ba57739fe4e1b7556950db3a5].
:runners:google-cloud-dataflow-java:cleanUpDockerImages (Thread[Execution **** for ':' Thread 8,5,main]) completed. Took 2.728 secs.
:sdks:java:io:kafka:cleanUp (Thread[Execution **** for ':' Thread 8,5,main]) started.
> Task :sdks:java:io:kafka:cleanUp
Skipping task ':sdks:java:io:kafka:cleanUp' as it has no actions.
:sdks:java:io:kafka:cleanUp (Thread[Execution **** for ':' Thread 8,5,main]) completed. Took 0.0 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:io:kafka:integrationTest'.
> There were failing tests. See the report at: file://<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/kafka/build/reports/tests/integrationTest/index.html>
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 23m 44s
136 actionable tasks: 25 executed, 111 up-to-date
Gradle was unable to watch the file system for changes. The inotify watches limit is too low.
Publishing build scan...
https://gradle.com/s/ozo4nlvf2ktmg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_Kafka_IO #1566
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1566/display/redirect?page=changes>
Changes:
[Robert Bradshaw] Further dataframe batch consolidation.
[Robert Bradshaw] Add another test.
[tysonjh] Fix Java ValidatesRunner V2 task dependency and update test exclusion.
[Robert Bradshaw] Use non-underscore methods for series and dataframe comparators.
[tysonjh] [BEAM-11263] Java cleanUpDockerImages now force removes container
[Robert Bradshaw] Only force indexing for multi-input elementwise operations.
[veblush] Upgrade gcsio to 2.1.6
[noreply] Update sdks/python/apache_beam/dataframe/transforms.py
[noreply] [BEAM-11265] Update quickstart-java.md (#13128)
[noreply] Go redundant type cleanup. (#13348)
[noreply] [BEAM-11207] Metric Extraction via proto RPC API (#13272)
[noreply] [BEAM-11262] Remove numSleeps assertion in SpannerIOWriteTest (#13340)
------------------------------------------
Started by timer
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on apache-beam-jenkins-15 (beam) in workspace <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/>
No credentials specified
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
> git init <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src> # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git --version # timeout=10
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
> git fetch --tags --progress https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/* # timeout=10
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6c5011a266ab9f2c75f68813f64e86e6a5689946 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 6c5011a266ab9f2c75f68813f64e86e6a5689946 # timeout=10
Commit message: "[BEAM-11262] Remove numSleeps assertion in SpannerIOWriteTest (#13340)"
> git rev-list --no-walk 2f2ffdafb341458fd626d1135cd87923231b31b8 # timeout=10
No emails were triggered.
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
SPARK_LOCAL_IP=127.0.0.1
SETUPTOOLS_USE_DISTUTILS=stdlib
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Kafka_IO] $ /bin/bash -xe /tmp/jenkins6353885063422705181.sh
+ cp /home/jenkins/.kube/config <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/config-beam-performancetests-kafka-io-1566>
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties content
KUBECONFIG=<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/config-beam-performancetests-kafka-io-1566>
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Kafka_IO] $ /bin/bash -xe /tmp/jenkins4588715154008241323.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a
Fetching cluster endpoint and auth data.
kubeconfig entry generated for io-datastores.
[beam_PerformanceTests_Kafka_IO] $ /bin/bash -xe /tmp/jenkins1407882080059481681.sh
+ <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/.test-infra/kubernetes/kubernetes.sh> createNamespace beam-performancetests-kafka-io-1566
+ KUBECONFIG=<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/config-beam-performancetests-kafka-io-1566>
+ KUBERNETES_NAMESPACE=default
+ KUBECTL='kubectl --kubeconfig=<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/config-beam-performancetests-kafka-io-1566> --namespace=default'
+ createNamespace beam-performancetests-kafka-io-1566
+ eval 'kubectl --kubeconfig=<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/config-beam-performancetests-kafka-io-1566> create namespace beam-performancetests-kafka-io-1566'
++ kubectl --kubeconfig=<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/config-beam-performancetests-kafka-io-1566> create namespace beam-performancetests-kafka-io-1566
namespace/beam-performancetests-kafka-io-1566 created
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties content
KUBERNETES_NAMESPACE=beam-performancetests-kafka-io-1566
[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Kafka_IO] $ /bin/bash -xe /tmp/jenkins2656413785041649960.sh
+ <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/.test-infra/kubernetes/kubernetes.sh> apply <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/.test-infra/kubernetes/kafka-cluster>
+ KUBECONFIG=<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/config-beam-performancetests-kafka-io-1566>
+ KUBERNETES_NAMESPACE=beam-performancetests-kafka-io-1566
+ KUBECTL='kubectl --kubeconfig=<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/config-beam-performancetests-kafka-io-1566> --namespace=beam-performancetests-kafka-io-1566'
+ apply <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/.test-infra/kubernetes/kafka-cluster>
+ eval 'kubectl --kubeconfig=<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/config-beam-performancetests-kafka-io-1566> --namespace=beam-performancetests-kafka-io-1566 apply -R -f <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/.test-infra/kubernetes/kafka-cluster'>
++ kubectl --kubeconfig=<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/config-beam-performancetests-kafka-io-1566> --namespace=beam-performancetests-kafka-io-1566 apply -R -f <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/.test-infra/kubernetes/kafka-cluster>
storageclass.storage.k8s.io/kafka-broker unchanged
storageclass.storage.k8s.io/kafka-zookeeper unchanged
clusterrole.rbac.authorization.k8s.io/node-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/kafka-node-reader configured
role.rbac.authorization.k8s.io/pod-labler created
rolebinding.rbac.authorization.k8s.io/kafka-pod-labler created
configmap/zookeeper-config created
service/pzoo created
service/zookeeper created
statefulset.apps/pzoo created
configmap/broker-config created
service/broker created
service/bootstrap created
statefulset.apps/kafka created
configmap/kafka-config created
job.batch/kafka-config-eff079ec created
Error from server (Invalid): error when creating "<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-0.yml">: Service "outside-0" is invalid: spec.ports[0].nodePort: Invalid value: 32400: provided port is already allocated
Error from server (Invalid): error when creating "<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-1.yml">: Service "outside-1" is invalid: spec.ports[0].nodePort: Invalid value: 32401: provided port is already allocated
Error from server (Invalid): error when creating "<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-2.yml">: Service "outside-2" is invalid: spec.ports[0].nodePort: Invalid value: 32402: provided port is already allocated
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
beam_PerformanceTests_Kafka_IO - Build # 1565 - Aborted!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
beam_PerformanceTests_Kafka_IO - Build # 1565 - Aborted:
Check console output at https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1565/ to view the results.
beam_PerformanceTests_Kafka_IO - Build # 1564 - Aborted!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
beam_PerformanceTests_Kafka_IO - Build # 1564 - Aborted:
Check console output at https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1564/ to view the results.
beam_PerformanceTests_Kafka_IO - Build # 1563 - Aborted!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
beam_PerformanceTests_Kafka_IO - Build # 1563 - Aborted:
Check console output at https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1563/ to view the results.
beam_PerformanceTests_Kafka_IO - Build # 1562 - Aborted!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
beam_PerformanceTests_Kafka_IO - Build # 1562 - Aborted:
Check console output at https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1562/ to view the results.
Build failed in Jenkins: beam_PerformanceTests_Kafka_IO #1561
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1561/display/redirect>
Changes:
------------------------------------------
GitHub pull request #13026 of commit 3309f670077755cf7a44e8e76171835cdd1ca7f1, no merge conflicts.
Running as SYSTEM
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
beam_PerformanceTests_Kafka_IO - Build # 1560 - Aborted!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
beam_PerformanceTests_Kafka_IO - Build # 1560 - Aborted:
Check console output at https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1560/ to view the results.