[jira] [Created] (BEAM-14120) Running Apache Beam pipeline on Azure Databricks

["Manoj Kumar (Jira)" <ji...@apache.org>]

Manoj Kumar created BEAM-14120:
----------------------------------

             Summary: Running Apache Beam pipeline on Azure Databricks
                 Key: BEAM-14120
                 URL: https://issues.apache.org/jira/browse/BEAM-14120
             Project: Beam
          Issue Type: Bug
          Components: io-java-kafka, runner-spark
            Reporter: Manoj Kumar


I'm trying to create a simple streaming app with Apache Beam, where it reads data from an Azure event hub and produces messages into another Azure event hub.
 
I'm creating and running spark jobs on Azure Databricks. 
The problem is the consumer (uses SparkRunner) is not receiving any messages from Event hub (topic). There is no activity and no errors on the Spark cluster.
 I tried to consume event hub messages without using Apache beam on the same cluster and it is working without any issues. In addition to that I'm also able to produce message from same cluster using Apache Beam Kafka IO. 
 
I'm not sure is this a issue in Kafka IO or Spark runner. Could anyone help on this?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)