Re: [Nifi 1.16.1] Conflicting libraries when reading/writing to HDFS and Ozone

[Giorgio <gr...@protonmail.com.INVALID>]

Hello,

to fix the problem we essentially followed the following steps.

1)
Added the relevant dependencies to the pom.xml
<!-- https://mvnrepository.com/artifact/org.apache.ozone/ozone-client -->
<dependency>
<groupId>org.apache.ozone</groupId>
<artifactId>ozone-client</artifactId>
<version>1.2.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.ozone/ozone-filesystem -->
<dependency>
<groupId>org.apache.ozone</groupId>
<artifactId>ozone-filesystem</artifactId>
<version>1.2.1</version>
</dependency>


2)
We extended the class  AbstractHadoopProcessor following what was done for the PutHdfs processesor (https://github.com/apache/nifi/blob/589136ac11288a1d0c0a267c08ce102e45279cfc/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java#L100)

3)
added the annotation @RequiresInstanceClassLoading(cloneAncestorResources = true)

4)
built the NAR and deployed it in the NiFi's lib folder.

Hope this explanation is detailed enough; otherwise, please ask any other question.

Thanks,

Giorgio




Sent with Proton Mail secure email.

------- Original Message -------
On Friday, July 29th, 2022 at 15:22, Bryan Bende <bb...@gmail.com> wrote:


> Yes creating a jira makes sense.
>
> Likely someone needs to look at which version of Ozone client is being
> used, and which version of HDFS client is being used, and then compare
> all the transitive dependencies to see if there are any obvious
> conflicts.
>
> On Fri, Jul 29, 2022 at 8:54 AM Giorgio grgs@protonmail.com wrote:
>
> > Hello,
> >
> > Following up on this, I just wanted to ask if it’s suggested to open an issue.
> >
> > Thanks
> >
> > G
> >
> > Sent from Proton Mail for iOS
> >
> > On Mon, Jul 25, 2022 at 20:04, Giorgio grgs@protonmail.com wrote:
> >
> > Hi Bryan,
> >
> > Thank you for getting back to me. Yes, that’s correct we compiled with the ozone profile.
> >
> > G
> >
> > Sent from Proton Mail for iOS
> >
> > On Mon, Jul 25, 2022 at 18:12, Bryan Bende bbende@gmail.com wrote:
> >
> > Hello,
> >
> > Did you build your own NiFi off 1.16.1 using the profile
> > "include-hadoop-ozone", and then used PutHDFS and that is what
> > generated the error in your attached logs?
> >
> > -Bryan
> >
> > On Mon, Jul 25, 2022 at 11:08 AM Giorgio grgs@protonmail.com.invalid wrote:
> >
> > > Hello,
> > >
> > > I would like to report a potential bug that affects NiFi 1.16.1. Our current setup consists of a cluster of 3 nodes and the aim is to build a workflow that can write to both HDFS and Apache Ozone. To this end, we have configured two distinct PutHDFS processors. The configuration of these two processors is such that one points to HDFS and the other one to Ozone using two different core-sites configuration files. The PutHDFS processor that points to Ozone uses OFS as protocol.
> > >
> > > When we try running the workflow, however, we run into problems. Essentially, this seems to be caused by conflicting libraries that are instantiated by the two processors, thus generating an error.
> > >
> > > We have found a way to circumvent this problem but this required us to build a custom PutOzone processor that better encapsulates the way the libraries are instantiated.
> > >
> > > We are not sure whether this is a known issue. Please find attached the relevant log lines.
> > >
> > > Regards,
> > >
> > > Giorgio
> > >
> > > Sent with Proton Mail secure email.