You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Kenneth Knowles (Jira)" <ji...@apache.org> on 2022/05/12 17:56:00 UTC

[jira] [Updated] (BEAM-14120) Running Apache Beam pipeline on Azure Databricks

     [ https://issues.apache.org/jira/browse/BEAM-14120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kenneth Knowles updated BEAM-14120:
-----------------------------------
    Status: Open  (was: Triage Needed)

> Running Apache Beam pipeline on Azure Databricks
> ------------------------------------------------
>
>                 Key: BEAM-14120
>                 URL: https://issues.apache.org/jira/browse/BEAM-14120
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-kafka, runner-spark
>            Reporter: Manoj Kumar
>            Priority: P2
>
> I'm trying to create a simple streaming app with Apache Beam, where it reads data from an Azure event hub and produces messages into another Azure event hub.
>  
> I'm creating and running spark jobs on Azure Databricks. 
> The problem is the consumer (uses SparkRunner) is not receiving any messages from Event hub (topic). There is no activity and no errors on the Spark cluster.
>  I tried to consume event hub messages without using Apache beam on the same cluster and it is working without any issues. In addition to that I'm also able to produce message from same cluster using Apache Beam Kafka IO. 
>  
> I'm not sure is this a issue in Kafka IO or Spark runner. Could anyone help on this?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)