You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Chamikara Madhusanka Jayalath (Jira)" <ji...@apache.org> on 2022/03/16 01:30:00 UTC
[jira] [Updated] (BEAM-13579) Cannot run python_xlang_kafka_taxi_dataflow validation script on 2.35.0
[ https://issues.apache.org/jira/browse/BEAM-13579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chamikara Madhusanka Jayalath updated BEAM-13579:
-------------------------------------------------
Fix Version/s: Not applicable
Resolution: Fixed
Status: Resolved (was: Open)
> Cannot run python_xlang_kafka_taxi_dataflow validation script on 2.35.0
> -----------------------------------------------------------------------
>
> Key: BEAM-13579
> URL: https://issues.apache.org/jira/browse/BEAM-13579
> Project: Beam
> Issue Type: Bug
> Components: release
> Reporter: Valentyn Tymofieiev
> Assignee: Chamikara Madhusanka Jayalath
> Priority: P1
> Fix For: Not applicable
>
>
> Dataflow job gets errors like:
> {noformat}
> Error message from worker: generic::unknown: org.apache.beam.sdk.util.UserCodeException: java.io.IOException: KafkaWriter : failed to send 1 records (since last report)
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
> org.apache.beam.sdk.io.kafka.KafkaWriter$DoFnInvoker.invokeProcessElement(Unknown Source)
> org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:758)
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:257)
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:209)
> org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo(FnApiDoFnRunner.java:1745)
> org.apache.beam.fn.harness.FnApiDoFnRunner.access$2700(FnApiDoFnRunner.java:142)
> org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.outputWithTimestamp(FnApiDoFnRunner.java:2263)
> org.apache.beam.fn.harness.FnApiDoFnRunner$ProcessBundleContextBase.output(FnApiDoFnRunner.java:2432)
> org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:78)
> org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:142)
> org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown Source)
> org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:758)
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:257)
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:209)
> org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:172)
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver2.awaitCompletion(BeamFnDataInboundObserver2.java:126)
> org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:467)
> org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:151)
> org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:116)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: KafkaWriter : failed to send 1 records (since last report)
> org.apache.beam.sdk.io.kafka.KafkaWriter.checkForFailures(KafkaWriter.java:133)
> org.apache.beam.sdk.io.kafka.KafkaWriter.processElement(KafkaWriter.java:58)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Topic beam-runnerv2 not present in metadata after 60000 ms.
> {noformat}
> run_rc_validation.sh fails with errors like:
> {noformat}
> * Please wait for at least 10 mins to let Dataflow job be launched and results get populated.
> * Sleeping for 10 mins
> * How to verify results:
> * 1. Goto your Dataflow job console and check whether there is any error.
> * 2. Check whether valentyn_python_validations_1229_10742.xlang_kafka_taxi has data, retrieving BigQuery data as below:
> BigQuery error in head operation: Not found: Table
> google.com:clouddfe:valentyn_python_validations_1229_10742.xlang_kafka_taxi
> Couldn't find expected output. Please confirm the output by visiting the console manually.
> {noformat}
> I suspect something is misconfigured but couldn't figure out what.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)