You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Mohit <mo...@open-insights.co.in> on 2018/07/25 12:16:29 UTC
Unable to see Nifi data lineage in Atlas
Hi all,
I have configured ReportingLineageToAtlas reporting task to send Nifi flow
information to Atlas. Nifi is integrated with Ranger.
I am able to see all the information in the Atlas except the lineage. When I
search for hdfs_path or hive_table, I can only see the hive side
information. I can't figure out anything wrong in the configuration.
Is there something in the Ranger configuration that I'm missing?
Regards,
Mohit
Re: Unable to see Nifi data lineage in Atlas
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Mohit,
From the log message, I assume that you are using an existing
atlas-application.properties copied from somewhere (most likely from
HDP environment) and PLAINTEXTSASL is used in it.
PLAINTEXTSASL is not supported by the ReportLineageToAtlas.
As a work-around, please set 'Create Atlas Configuration File' to true
and let the reporting task generate atlas-application.properties
instead.
SASL_PLAINTEXT is identical to PLAINTEXTSASL.
You may need to restart NiFi to take effect.
Hope this helps,
Koji
On Thu, Jul 26, 2018 at 7:12 PM, Mohit <mo...@open-insights.co.in> wrote:
> Hi,
>
>
>
> While looking at the logs, I found out that ReportingLineageToAtlas is not
> able to construct KafkaProducer.
>
> It throws the following logs –
>
>
>
> org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>
> at
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335)
>
> at
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188)
>
> at
> org.apache.atlas.kafka.KafkaNotification.createProducer(KafkaNotification.java:286)
>
> at
> org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:207)
>
> at
> org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:84)
>
> at
> org.apache.atlas.hook.AtlasHook.notifyEntitiesInternal(AtlasHook.java:133)
>
> at
> org.apache.atlas.hook.AtlasHook.notifyEntities(AtlasHook.java:118)
>
> at
> org.apache.atlas.hook.AtlasHook.notifyEntities(AtlasHook.java:171)
>
> at
> org.apache.nifi.atlas.NiFiAtlasHook.commitMessages(NiFiAtlasHook.java:150)
>
> at
> org.apache.nifi.atlas.reporting.ReportLineageToAtlas.lambda$consumeNiFiProvenanceEvents$6(ReportLineageToAtlas.java:721)
>
> at
> org.apache.nifi.reporting.util.provenance.ProvenanceEventConsumer.consumeEvents(ProvenanceEventConsumer.java:204)
>
> at
> org.apache.nifi.atlas.reporting.ReportLineageToAtlas.consumeNiFiProvenanceEvents(ReportLineageToAtlas.java:712)
>
> at
> org.apache.nifi.atlas.reporting.ReportLineageToAtlas.onTrigger(ReportLineageToAtlas.java:664)
>
> at
> org.apache.nifi.controller.tasks.ReportingTaskWrapper.run(ReportingTaskWrapper.java:41)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.lang.IllegalArgumentException: No enum constant
> org.apache.kafka.common.protocol.SecurityProtocol.PLAINTEXTSASL
>
> at java.lang.Enum.valueOf(Enum.java:238)
>
> at
> org.apache.kafka.common.protocol.SecurityProtocol.valueOf(SecurityProtocol.java:28)
>
> at
> org.apache.kafka.common.protocol.SecurityProtocol.forName(SecurityProtocol.java:89)
>
> at
> org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:79)
>
> at
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:277)
>
> ... 20 common frames omitted
>
>
>
> Thanks,
>
> Mohit
>
>
>
> From: Mohit <mo...@open-insights.co.in>
> Sent: 25 July 2018 17:46
> To: users@nifi.apache.org
> Subject: Unable to see Nifi data lineage in Atlas
>
>
>
> Hi all,
>
>
>
> I have configured ReportingLineageToAtlas reporting task to send Nifi flow
> information to Atlas. Nifi is integrated with Ranger.
>
> I am able to see all the information in the Atlas except the lineage. When I
> search for hdfs_path or hive_table, I can only see the hive side
> information. I can’t figure out anything wrong in the configuration.
>
> Is there something in the Ranger configuration that I’m missing?
>
>
>
> Regards,
>
> Mohit
>
>
>
>
RE: Unable to see Nifi data lineage in Atlas
Posted by Mohit <mo...@open-insights.co.in>.
Hi,
While looking at the logs, I found out that ReportingLineageToAtlas is not
able to construct KafkaProducer.
It throws the following logs -
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at
org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:33
5)
at
org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:18
8)
at
org.apache.atlas.kafka.KafkaNotification.createProducer(KafkaNotification.ja
va:286)
at
org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java
:207)
at
org.apache.atlas.notification.AbstractNotification.send(AbstractNotification
.java:84)
at
org.apache.atlas.hook.AtlasHook.notifyEntitiesInternal(AtlasHook.java:133)
at
org.apache.atlas.hook.AtlasHook.notifyEntities(AtlasHook.java:118)
at
org.apache.atlas.hook.AtlasHook.notifyEntities(AtlasHook.java:171)
at
org.apache.nifi.atlas.NiFiAtlasHook.commitMessages(NiFiAtlasHook.java:150)
at
org.apache.nifi.atlas.reporting.ReportLineageToAtlas.lambda$consumeNiFiProve
nanceEvents$6(ReportLineageToAtlas.java:721)
at
org.apache.nifi.reporting.util.provenance.ProvenanceEventConsumer.consumeEve
nts(ProvenanceEventConsumer.java:204)
at
org.apache.nifi.atlas.reporting.ReportLineageToAtlas.consumeNiFiProvenanceEv
ents(ReportLineageToAtlas.java:712)
at
org.apache.nifi.atlas.reporting.ReportLineageToAtlas.onTrigger(ReportLineage
ToAtlas.java:664)
at
org.apache.nifi.controller.tasks.ReportingTaskWrapper.run(ReportingTaskWrapp
er.java:41)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
49)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
24)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: No enum constant
org.apache.kafka.common.protocol.SecurityProtocol.PLAINTEXTSASL
at java.lang.Enum.valueOf(Enum.java:238)
at
org.apache.kafka.common.protocol.SecurityProtocol.valueOf(SecurityProtocol.j
ava:28)
at
org.apache.kafka.common.protocol.SecurityProtocol.forName(SecurityProtocol.j
ava:89)
at
org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:7
9)
at
org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:27
7)
... 20 common frames omitted
Thanks,
Mohit
From: Mohit <mo...@open-insights.co.in>
Sent: 25 July 2018 17:46
To: users@nifi.apache.org
Subject: Unable to see Nifi data lineage in Atlas
Hi all,
I have configured ReportingLineageToAtlas reporting task to send Nifi flow
information to Atlas. Nifi is integrated with Ranger.
I am able to see all the information in the Atlas except the lineage. When I
search for hdfs_path or hive_table, I can only see the hive side
information. I can't figure out anything wrong in the configuration.
Is there something in the Ranger configuration that I'm missing?
Regards,
Mohit