Integrating with a separate kafka cluster

[Euan Hope <ho...@gmail.com>]

Hi Metron community.

I am hoping to get some help relating to the kakfa brokers used by Metron.

We are able to get our dataflows working with the local instance of Kafka
within the Metron environment we have setup. Now, we would like to
integrate with our organizations production Kafka cluster. We have tried to
update the config's in Ambari but we are getting a
"java.lang.IllegalArgumentException: bootstrap servers cannot be null"
error. This error occurs no matter which custom sensor we start up.

We also tried another approach of updating the Kafka broker and Zookeeper
broker config's within the enrichment-splitjoin.properties and the
enrichment-unified.properties configuration files but this also did not
seem to be the right approach.

Is it possible to integrate Metron with a Kafka cluster that is setup on a
separate HDP environment?

Thanking you in advance for any guidance.

Re: Integrating with a separate kafka cluster

[Euan Hope <ho...@gmail.com>]

Hi Tom,
Thank you for your response.

We have managed to publish and consume data from topics on the other kafka
instance (external to our Metron environment) and we do this using nifi.

What we are trying to do now is to get a sensor (created within Metron) to
write data from our production kafka cluster to hbase (which is on the same
cluster as our Metron instance), using the
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter" class (as per
the sensor parser config in the Management UI).

We have tried changing the Kafka config's in Ambari to point to our
production Kafka brokers but we get the
"java.lang.IllegalArgumentException: bootstrap servers cannot be null"
error.

Hoping to get some guidance on this as we are not sure which parameters to
update.

Thanks once again for your mail.

On Thu, Jun 18, 2020 at 11:17 PM Yerex, Tom <to...@ubc.ca> wrote:

> Good afternoon Euan,
>
> In this group and in others I often see Nifi characterized (in my own
> words), as the bridge that can be used to connect
> various places you want to move data. If you haven't explored it yet, then
> it might be worthwhile to take an afternoon
> and have a look.
>
> I'm exploring a similar problem so if you can share what you learn, it
> would be appreciated.
>
> --Tom.
>
> On Thu, 2020-06-18 at 22:15 +0200, Euan Hope wrote:
> > Hi Metron community.
> >
> > I am hoping to get some help relating to the kakfa brokers used by
> Metron.
> >
> > We are able to get our dataflows working with the local instance of
> Kafka within the Metron environment we have setup.
> > Now, we would like to integrate with our organizations production Kafka
> cluster. We have tried to update the config's
> > in Ambari but we are getting a "java.lang.IllegalArgumentException:
> bootstrap servers cannot be null" error. This
> > error occurs no matter which custom sensor we start up.
> >
> > We also tried another approach of updating the Kafka broker and
> Zookeeper broker config's within the enrichment-
> > splitjoin.properties and the enrichment-unified.properties configuration
> files but this also did not seem to be the
> > right approach.
> >
> > Is it possible to integrate Metron with a Kafka cluster that is setup on
> a separate HDP environment?
> >
> > Thanking you in advance for any guidance.
> >
> >
>
>

Re: Integrating with a separate kafka cluster

["Yerex, Tom" <to...@ubc.ca>]

Good afternoon Euan,

In this group and in others I often see Nifi characterized (in my own words), as the bridge that can be used to connect
various places you want to move data. If you haven't explored it yet, then it might be worthwhile to take an afternoon
and have a look.

I'm exploring a similar problem so if you can share what you learn, it would be appreciated.

--Tom.

On Thu, 2020-06-18 at 22:15 +0200, Euan Hope wrote:
> Hi Metron community.
> 
> I am hoping to get some help relating to the kakfa brokers used by Metron. 
> 
> We are able to get our dataflows working with the local instance of Kafka within the Metron environment we have setup.
> Now, we would like to integrate with our organizations production Kafka cluster. We have tried to update the config's
> in Ambari but we are getting a "java.lang.IllegalArgumentException: bootstrap servers cannot be null" error. This
> error occurs no matter which custom sensor we start up. 
> 
> We also tried another approach of updating the Kafka broker and Zookeeper broker config's within the enrichment-
> splitjoin.properties and the enrichment-unified.properties configuration files but this also did not seem to be the
> right approach. 
> 
> Is it possible to integrate Metron with a Kafka cluster that is setup on a separate HDP environment? 
> 
> Thanking you in advance for any guidance. 
> 
>