[jira] [Created] (BEAM-13579) Cannot run python_xlang_kafka_taxi_dataflow validation script on 2.35.0

["Valentyn Tymofieiev (Jira)" <ji...@apache.org>]

Valentyn Tymofieiev created BEAM-13579:
------------------------------------------

             Summary: Cannot run python_xlang_kafka_taxi_dataflow validation script on 2.35.0
                 Key: BEAM-13579
                 URL: https://issues.apache.org/jira/browse/BEAM-13579
             Project: Beam
          Issue Type: Bug
          Components: release
            Reporter: Valentyn Tymofieiev


Dataflow job gets errors like:
{noformat}
Error message from worker: generic::unknown: org.apache.beam.sdk.util.UserCodeException: java.io.IOException: KafkaWriter : failed to send 1 records (since last report)
	org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
	org.apache.beam.sdk.io.kafka.KafkaWriter$DoFnInvoker.invokeProcessElement(Unknown Source)
	org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:758)
	org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:257)
	org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:209)
	org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo(FnApiDoFnRunner.java:1745)
	org.apache.beam.fn.harness.FnApiDoFnRunner.access$2700(FnApiDoFnRunner.java:142)
	org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.outputWithTimestamp(FnApiDoFnRunner.java:2263)
	org.apache.beam.fn.harness.FnApiDoFnRunner$ProcessBundleContextBase.output(FnApiDoFnRunner.java:2432)
	org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:78)
	org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:142)
	org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown Source)
	org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:758)
	org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:257)
	org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:209)
	org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:172)
	org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver2.awaitCompletion(BeamFnDataInboundObserver2.java:126)
	org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:467)
	org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:151)
	org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:116)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: KafkaWriter : failed to send 1 records (since last report)
	org.apache.beam.sdk.io.kafka.KafkaWriter.checkForFailures(KafkaWriter.java:133)
	org.apache.beam.sdk.io.kafka.KafkaWriter.processElement(KafkaWriter.java:58)
Caused by: org.apache.kafka.common.errors.TimeoutException: Topic beam-runnerv2 not present in metadata after 60000 ms.
{noformat}

run_rc_validation.sh fails with errors like:
{noformat}
* Please wait for at least 10 mins to let Dataflow job be launched and results get populated.
* Sleeping for 10 mins
* How to verify results:
* 1. Goto your Dataflow job console and check whether there is any error.
* 2. Check whether valentyn_python_validations_1229_10742.xlang_kafka_taxi has data, retrieving BigQuery data as below: 
BigQuery error in head operation: Not found: Table
google.com:clouddfe:valentyn_python_validations_1229_10742.xlang_kafka_taxi
Couldn't find expected output. Please confirm the output by visiting the console manually.
{noformat}

I suspect something is misconfigured but couldn't figure out what.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)