You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Jeremy Dyer <je...@apache.org> on 2019/08/29 00:29:36 UTC
MiNiFi Python Dependencies (Anaconda)
Hello all,
I'm trying to use the MiNiFi C++ Python processors feature. I have used
this with great results in the past but I'm trying to load some libraries
that are installed in my Python conda virtual environment.
I have activated my environment and then run the ./bin/minifi binary under
the same user but the process doesn't seem to pickup my Python library that
I am trying to import. Does anyone have any thought on how I can make my
virtual environment path visible to the MiNiFi agent binary so those
libraries can be used in MiNiFi?
Thanks,
Jeremy Dyer
Re: MiNiFi Python Dependencies (Anaconda)
Posted by Marc Parisi <ph...@apache.org>.
Hi Jeremy,
The env variable I sent can be set or appended prior to running minifi,
without the need to recompile.
On Thu, Aug 29, 2019, 3:38 PM Jeremy Dyer <jd...@gmail.com> wrote:
> So are you saying I need to build and link against my conda python binary?
>
> On Thu, Aug 29, 2019 at 6:47 AM Marc Parisi <ph...@apache.org> wrote:
>
> > Hi Jeremy,
> >
> > Apache NiFi MiNiFi C++ python processors support virtual environments;
> > however, I don't think the library links, once the binary is built, will
> be
> > overridden through conda's environment activation ( only PATH is updated
> if
> > I recall correctly )
> >
> > I could be off base on the problem you are experiencing -- but one way
> > I've solved this with my Anacaonda environment has been to use
> > LD_LIBRARY_PATH. When I use the following command: export
> > LD_LIBRARY_PATH=/home/0x0000000000000000/anaconda3/lib/
> >
> > After this I no longer link to the system provided version of python
> > libraries but instead to
> > /home/0x0000000000000000/anaconda3/lib/libpython3.7m.so.1.0 .
> >
> > This is actually something I've seen detected via scripts so it is
> > possible for minifi.sh to (potentially) handle this.
> >
> > Let me know if this solves the issue.
> >
> > Best Regards,
> > Marc
> >
> > On Wed, Aug 28, 2019 at 8:29 PM Jeremy Dyer <je...@apache.org>
> wrote:
> >
> > > Hello all,
> > >
> > > I'm trying to use the MiNiFi C++ Python processors feature. I have used
> > > this with great results in the past but I'm trying to load some
> libraries
> > > that are installed in my Python conda virtual environment.
> > >
> > > I have activated my environment and then run the ./bin/minifi binary
> > under
> > > the same user but the process doesn't seem to pickup my Python library
> > that
> > > I am trying to import. Does anyone have any thought on how I can make
> my
> > > virtual environment path visible to the MiNiFi agent binary so those
> > > libraries can be used in MiNiFi?
> > >
> > > Thanks,
> > > Jeremy Dyer
> > >
> >
>
Re: MiNiFi Python Dependencies (Anaconda)
Posted by Jeremy Dyer <jd...@gmail.com>.
So are you saying I need to build and link against my conda python binary?
On Thu, Aug 29, 2019 at 6:47 AM Marc Parisi <ph...@apache.org> wrote:
> Hi Jeremy,
>
> Apache NiFi MiNiFi C++ python processors support virtual environments;
> however, I don't think the library links, once the binary is built, will be
> overridden through conda's environment activation ( only PATH is updated if
> I recall correctly )
>
> I could be off base on the problem you are experiencing -- but one way
> I've solved this with my Anacaonda environment has been to use
> LD_LIBRARY_PATH. When I use the following command: export
> LD_LIBRARY_PATH=/home/0x0000000000000000/anaconda3/lib/
>
> After this I no longer link to the system provided version of python
> libraries but instead to
> /home/0x0000000000000000/anaconda3/lib/libpython3.7m.so.1.0 .
>
> This is actually something I've seen detected via scripts so it is
> possible for minifi.sh to (potentially) handle this.
>
> Let me know if this solves the issue.
>
> Best Regards,
> Marc
>
> On Wed, Aug 28, 2019 at 8:29 PM Jeremy Dyer <je...@apache.org> wrote:
>
> > Hello all,
> >
> > I'm trying to use the MiNiFi C++ Python processors feature. I have used
> > this with great results in the past but I'm trying to load some libraries
> > that are installed in my Python conda virtual environment.
> >
> > I have activated my environment and then run the ./bin/minifi binary
> under
> > the same user but the process doesn't seem to pickup my Python library
> that
> > I am trying to import. Does anyone have any thought on how I can make my
> > virtual environment path visible to the MiNiFi agent binary so those
> > libraries can be used in MiNiFi?
> >
> > Thanks,
> > Jeremy Dyer
> >
>
Re: MiNiFi Python Dependencies (Anaconda)
Posted by Marc Parisi <ph...@apache.org>.
Hi Jeremy,
Apache NiFi MiNiFi C++ python processors support virtual environments;
however, I don't think the library links, once the binary is built, will be
overridden through conda's environment activation ( only PATH is updated if
I recall correctly )
I could be off base on the problem you are experiencing -- but one way
I've solved this with my Anacaonda environment has been to use
LD_LIBRARY_PATH. When I use the following command: export
LD_LIBRARY_PATH=/home/0x0000000000000000/anaconda3/lib/
After this I no longer link to the system provided version of python
libraries but instead to
/home/0x0000000000000000/anaconda3/lib/libpython3.7m.so.1.0 .
This is actually something I've seen detected via scripts so it is
possible for minifi.sh to (potentially) handle this.
Let me know if this solves the issue.
Best Regards,
Marc
On Wed, Aug 28, 2019 at 8:29 PM Jeremy Dyer <je...@apache.org> wrote:
> Hello all,
>
> I'm trying to use the MiNiFi C++ Python processors feature. I have used
> this with great results in the past but I'm trying to load some libraries
> that are installed in my Python conda virtual environment.
>
> I have activated my environment and then run the ./bin/minifi binary under
> the same user but the process doesn't seem to pickup my Python library that
> I am trying to import. Does anyone have any thought on how I can make my
> virtual environment path visible to the MiNiFi agent binary so those
> libraries can be used in MiNiFi?
>
> Thanks,
> Jeremy Dyer
>