Question about nifi-env.sh

[Lars Francke <la...@gmail.com>]

Hi,

I'm trying to use the environment variables NIFI_PID_DIR and NIFI_LOG_DIR
to change those directories.

Unfortunately nifi-env.sh is being called by nifi.sh which overrides them
without any way to override them.

To make it even more confusing the path for NIFI_PID_DIR is different than
the default which is hardcoded in RunNifi.

I'm trying to figure out if I'm missing anything obvious.

Otherwise I'd suggest changing nifi-env.sh to only set those variables when
they have not already been set to another value. Does that make sense?

Thank you,
Lars

Re: Question about nifi-env.sh

[Lars Francke <la...@gmail.com>]

I've opened NIFI-5573 (<https://issues.apache.org/jira/browse/NIFI-5573>)
for this

On Mon, Aug 27, 2018 at 12:38 PM, Lars Francke <la...@gmail.com>
wrote:

> Peter,
>
> thanks for the comments.
> It seems as if Joe and you are in agreement.
> I'd be in favor of changing it to the way Hadoop, HBase and other projects
> do it because for me this behaviour was very surprising (and inconsistent).
> But I understand your reasoning and won't put up an argument :)
>
> The solution you proposed would be fine for me. Having a
> NIFI_DISABLE_NIFIENV (I'm open for naming suggestions) environment variable
> and if that's set disregard whatever we have in nifi-env.sh. If that's
> acceptable I can open an issue and provide a patch.
>
> Cheers,
> Lars
>
> On Mon, Aug 27, 2018 at 10:07 AM, Peter Wilcsinszky <
> peterwilcsinszky@gmail.com> wrote:
>
>> Hi Lars!
>>
>> The idea you proposed - to set only what has not been set already - makes
>> sense, however it may cause unexpected side effects in deployments where
>> there are environment variables hidden by nifi-env.sh. This is
>> theoretical,
>> but I can imagine situations where operators tried to set variables
>> without
>> luck and forgot to remove them later. I mean we should better do this
>> opt-in, for example we may have a variable that can be used to disable
>> nifi-env.sh completely. We did something like that in the docker image
>> where nifi-env.sh is explicitly cleared [1] but would benefit from a more
>> generic solution.
>>
>> [1]
>> https://github.com/apache/nifi/blob/master/nifi-docker/docke
>> rhub/Dockerfile#L82
>>
>> Peter
>>
>> On Fri, Aug 24, 2018 at 8:14 PM Lars Francke <la...@gmail.com>
>> wrote:
>>
>> > Maybe I should explain what I'm doing and where my last two issues came
>> > from as well.[1]
>> >
>> > I'm trying to run NiFi in an environment where the tarball is extracted
>> (as
>> > downloaded from Apache) but I don't have write access to that directory
>> at
>> > all. So _all_ files NiFi writes need to go into another directory.
>> Meaning
>> > I have to point bootstrap.conf somewhere else, PID dir, Log dir, Content
>> > repository, Provenance repository, NAR work dir etc. - this means I
>> cannot
>> > overwrite nivi-env.sh as it's being picked up from wherever nifi.sh
>> lives.
>> >
>> > Maybe I'm missing something but that doesn't seem super trivial. Any
>> hints
>> > are highly appreciated.
>> >
>> >
>> > [1] <https://issues.apache.org/jira/browse/NIFI-5352> & <
>> > https://issues.apache.org/jira/browse/NIFI-5350]
>> >
>> > On Fri, Aug 24, 2018 at 4:30 PM, Lars Francke <la...@gmail.com>
>> > wrote:
>> >
>> > > Thanks for the quick response Joe.
>> > >
>> > > Yes, I assumed as much that's why I wanted to discuss here first
>> before I
>> > > open an issue.
>> > >
>> > > On Fri, Aug 24, 2018 at 4:01 PM, Joe Witt <jo...@gmail.com> wrote:
>> > >
>> > >> Lars
>> > >>
>> > >> I dont know the specifics of what went into various bits there but
>> > >> thanks for being careful/discussing it - need folks like mark payne
>> > >> and yolanda davis and others who wrote some of those bits to comment.
>> > >> I say this because these scripts/start routines are super heavily
>> used
>> > >> for deployment processes that I know a lot of users depend on.
>> > >>
>> > >> Thanks
>> > >>
>> > >> On Fri, Aug 24, 2018 at 9:58 AM Lars Francke <lars.francke@gmail.com
>> >
>> > >> wrote:
>> > >> >
>> > >> > Hi,
>> > >> >
>> > >> > I'm trying to use the environment variables NIFI_PID_DIR and
>> > >> NIFI_LOG_DIR
>> > >> > to change those directories.
>> > >> >
>> > >> > Unfortunately nifi-env.sh is being called by nifi.sh which
>> overrides
>> > >> them
>> > >> > without any way to override them.
>> > >> >
>> > >> > To make it even more confusing the path for NIFI_PID_DIR is
>> different
>> > >> than
>> > >> > the default which is hardcoded in RunNifi.
>> > >> >
>> > >> > I'm trying to figure out if I'm missing anything obvious.
>> > >> >
>> > >> > Otherwise I'd suggest changing nifi-env.sh to only set those
>> variables
>> > >> when
>> > >> > they have not already been set to another value. Does that make
>> sense?
>> > >> >
>> > >> > Thank you,
>> > >> > Lars
>> > >>
>> > >
>> > >
>> >
>>
>
>

Re: Question about nifi-env.sh

[Lars Francke <la...@gmail.com>]

Peter,

thanks for the comments.
It seems as if Joe and you are in agreement.
I'd be in favor of changing it to the way Hadoop, HBase and other projects
do it because for me this behaviour was very surprising (and inconsistent).
But I understand your reasoning and won't put up an argument :)

The solution you proposed would be fine for me. Having a
NIFI_DISABLE_NIFIENV (I'm open for naming suggestions) environment variable
and if that's set disregard whatever we have in nifi-env.sh. If that's
acceptable I can open an issue and provide a patch.

Cheers,
Lars

On Mon, Aug 27, 2018 at 10:07 AM, Peter Wilcsinszky <
peterwilcsinszky@gmail.com> wrote:

> Hi Lars!
>
> The idea you proposed - to set only what has not been set already - makes
> sense, however it may cause unexpected side effects in deployments where
> there are environment variables hidden by nifi-env.sh. This is theoretical,
> but I can imagine situations where operators tried to set variables without
> luck and forgot to remove them later. I mean we should better do this
> opt-in, for example we may have a variable that can be used to disable
> nifi-env.sh completely. We did something like that in the docker image
> where nifi-env.sh is explicitly cleared [1] but would benefit from a more
> generic solution.
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-docker/
> dockerhub/Dockerfile#L82
>
> Peter
>
> On Fri, Aug 24, 2018 at 8:14 PM Lars Francke <la...@gmail.com>
> wrote:
>
> > Maybe I should explain what I'm doing and where my last two issues came
> > from as well.[1]
> >
> > I'm trying to run NiFi in an environment where the tarball is extracted
> (as
> > downloaded from Apache) but I don't have write access to that directory
> at
> > all. So _all_ files NiFi writes need to go into another directory.
> Meaning
> > I have to point bootstrap.conf somewhere else, PID dir, Log dir, Content
> > repository, Provenance repository, NAR work dir etc. - this means I
> cannot
> > overwrite nivi-env.sh as it's being picked up from wherever nifi.sh
> lives.
> >
> > Maybe I'm missing something but that doesn't seem super trivial. Any
> hints
> > are highly appreciated.
> >
> >
> > [1] <https://issues.apache.org/jira/browse/NIFI-5352> & <
> > https://issues.apache.org/jira/browse/NIFI-5350]
> >
> > On Fri, Aug 24, 2018 at 4:30 PM, Lars Francke <la...@gmail.com>
> > wrote:
> >
> > > Thanks for the quick response Joe.
> > >
> > > Yes, I assumed as much that's why I wanted to discuss here first
> before I
> > > open an issue.
> > >
> > > On Fri, Aug 24, 2018 at 4:01 PM, Joe Witt <jo...@gmail.com> wrote:
> > >
> > >> Lars
> > >>
> > >> I dont know the specifics of what went into various bits there but
> > >> thanks for being careful/discussing it - need folks like mark payne
> > >> and yolanda davis and others who wrote some of those bits to comment.
> > >> I say this because these scripts/start routines are super heavily used
> > >> for deployment processes that I know a lot of users depend on.
> > >>
> > >> Thanks
> > >>
> > >> On Fri, Aug 24, 2018 at 9:58 AM Lars Francke <la...@gmail.com>
> > >> wrote:
> > >> >
> > >> > Hi,
> > >> >
> > >> > I'm trying to use the environment variables NIFI_PID_DIR and
> > >> NIFI_LOG_DIR
> > >> > to change those directories.
> > >> >
> > >> > Unfortunately nifi-env.sh is being called by nifi.sh which overrides
> > >> them
> > >> > without any way to override them.
> > >> >
> > >> > To make it even more confusing the path for NIFI_PID_DIR is
> different
> > >> than
> > >> > the default which is hardcoded in RunNifi.
> > >> >
> > >> > I'm trying to figure out if I'm missing anything obvious.
> > >> >
> > >> > Otherwise I'd suggest changing nifi-env.sh to only set those
> variables
> > >> when
> > >> > they have not already been set to another value. Does that make
> sense?
> > >> >
> > >> > Thank you,
> > >> > Lars
> > >>
> > >
> > >
> >
>

Re: Question about nifi-env.sh

[Peter Wilcsinszky <pe...@gmail.com>]

Hi Lars!

The idea you proposed - to set only what has not been set already - makes
sense, however it may cause unexpected side effects in deployments where
there are environment variables hidden by nifi-env.sh. This is theoretical,
but I can imagine situations where operators tried to set variables without
luck and forgot to remove them later. I mean we should better do this
opt-in, for example we may have a variable that can be used to disable
nifi-env.sh completely. We did something like that in the docker image
where nifi-env.sh is explicitly cleared [1] but would benefit from a more
generic solution.

[1]
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/Dockerfile#L82

Peter

On Fri, Aug 24, 2018 at 8:14 PM Lars Francke <la...@gmail.com> wrote:

> Maybe I should explain what I'm doing and where my last two issues came
> from as well.[1]
>
> I'm trying to run NiFi in an environment where the tarball is extracted (as
> downloaded from Apache) but I don't have write access to that directory at
> all. So _all_ files NiFi writes need to go into another directory. Meaning
> I have to point bootstrap.conf somewhere else, PID dir, Log dir, Content
> repository, Provenance repository, NAR work dir etc. - this means I cannot
> overwrite nivi-env.sh as it's being picked up from wherever nifi.sh lives.
>
> Maybe I'm missing something but that doesn't seem super trivial. Any hints
> are highly appreciated.
>
>
> [1] <https://issues.apache.org/jira/browse/NIFI-5352> & <
> https://issues.apache.org/jira/browse/NIFI-5350]
>
> On Fri, Aug 24, 2018 at 4:30 PM, Lars Francke <la...@gmail.com>
> wrote:
>
> > Thanks for the quick response Joe.
> >
> > Yes, I assumed as much that's why I wanted to discuss here first before I
> > open an issue.
> >
> > On Fri, Aug 24, 2018 at 4:01 PM, Joe Witt <jo...@gmail.com> wrote:
> >
> >> Lars
> >>
> >> I dont know the specifics of what went into various bits there but
> >> thanks for being careful/discussing it - need folks like mark payne
> >> and yolanda davis and others who wrote some of those bits to comment.
> >> I say this because these scripts/start routines are super heavily used
> >> for deployment processes that I know a lot of users depend on.
> >>
> >> Thanks
> >>
> >> On Fri, Aug 24, 2018 at 9:58 AM Lars Francke <la...@gmail.com>
> >> wrote:
> >> >
> >> > Hi,
> >> >
> >> > I'm trying to use the environment variables NIFI_PID_DIR and
> >> NIFI_LOG_DIR
> >> > to change those directories.
> >> >
> >> > Unfortunately nifi-env.sh is being called by nifi.sh which overrides
> >> them
> >> > without any way to override them.
> >> >
> >> > To make it even more confusing the path for NIFI_PID_DIR is different
> >> than
> >> > the default which is hardcoded in RunNifi.
> >> >
> >> > I'm trying to figure out if I'm missing anything obvious.
> >> >
> >> > Otherwise I'd suggest changing nifi-env.sh to only set those variables
> >> when
> >> > they have not already been set to another value. Does that make sense?
> >> >
> >> > Thank you,
> >> > Lars
> >>
> >
> >
>

Re: Question about nifi-env.sh

[Lars Francke <la...@gmail.com>]

Maybe I should explain what I'm doing and where my last two issues came
from as well.[1]

I'm trying to run NiFi in an environment where the tarball is extracted (as
downloaded from Apache) but I don't have write access to that directory at
all. So _all_ files NiFi writes need to go into another directory. Meaning
I have to point bootstrap.conf somewhere else, PID dir, Log dir, Content
repository, Provenance repository, NAR work dir etc. - this means I cannot
overwrite nivi-env.sh as it's being picked up from wherever nifi.sh lives.

Maybe I'm missing something but that doesn't seem super trivial. Any hints
are highly appreciated.


[1] <https://issues.apache.org/jira/browse/NIFI-5352> & <
https://issues.apache.org/jira/browse/NIFI-5350]

On Fri, Aug 24, 2018 at 4:30 PM, Lars Francke <la...@gmail.com>
wrote:

> Thanks for the quick response Joe.
>
> Yes, I assumed as much that's why I wanted to discuss here first before I
> open an issue.
>
> On Fri, Aug 24, 2018 at 4:01 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Lars
>>
>> I dont know the specifics of what went into various bits there but
>> thanks for being careful/discussing it - need folks like mark payne
>> and yolanda davis and others who wrote some of those bits to comment.
>> I say this because these scripts/start routines are super heavily used
>> for deployment processes that I know a lot of users depend on.
>>
>> Thanks
>>
>> On Fri, Aug 24, 2018 at 9:58 AM Lars Francke <la...@gmail.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > I'm trying to use the environment variables NIFI_PID_DIR and
>> NIFI_LOG_DIR
>> > to change those directories.
>> >
>> > Unfortunately nifi-env.sh is being called by nifi.sh which overrides
>> them
>> > without any way to override them.
>> >
>> > To make it even more confusing the path for NIFI_PID_DIR is different
>> than
>> > the default which is hardcoded in RunNifi.
>> >
>> > I'm trying to figure out if I'm missing anything obvious.
>> >
>> > Otherwise I'd suggest changing nifi-env.sh to only set those variables
>> when
>> > they have not already been set to another value. Does that make sense?
>> >
>> > Thank you,
>> > Lars
>>
>
>

Re: Question about nifi-env.sh

[Lars Francke <la...@gmail.com>]

Thanks for the quick response Joe.

Yes, I assumed as much that's why I wanted to discuss here first before I
open an issue.

On Fri, Aug 24, 2018 at 4:01 PM, Joe Witt <jo...@gmail.com> wrote:

> Lars
>
> I dont know the specifics of what went into various bits there but
> thanks for being careful/discussing it - need folks like mark payne
> and yolanda davis and others who wrote some of those bits to comment.
> I say this because these scripts/start routines are super heavily used
> for deployment processes that I know a lot of users depend on.
>
> Thanks
>
> On Fri, Aug 24, 2018 at 9:58 AM Lars Francke <la...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I'm trying to use the environment variables NIFI_PID_DIR and NIFI_LOG_DIR
> > to change those directories.
> >
> > Unfortunately nifi-env.sh is being called by nifi.sh which overrides them
> > without any way to override them.
> >
> > To make it even more confusing the path for NIFI_PID_DIR is different
> than
> > the default which is hardcoded in RunNifi.
> >
> > I'm trying to figure out if I'm missing anything obvious.
> >
> > Otherwise I'd suggest changing nifi-env.sh to only set those variables
> when
> > they have not already been set to another value. Does that make sense?
> >
> > Thank you,
> > Lars
>

Re: Question about nifi-env.sh

[Joe Witt <jo...@gmail.com>]

Lars

I dont know the specifics of what went into various bits there but
thanks for being careful/discussing it - need folks like mark payne
and yolanda davis and others who wrote some of those bits to comment.
I say this because these scripts/start routines are super heavily used
for deployment processes that I know a lot of users depend on.

Thanks

On Fri, Aug 24, 2018 at 9:58 AM Lars Francke <la...@gmail.com> wrote:
>
> Hi,
>
> I'm trying to use the environment variables NIFI_PID_DIR and NIFI_LOG_DIR
> to change those directories.
>
> Unfortunately nifi-env.sh is being called by nifi.sh which overrides them
> without any way to override them.
>
> To make it even more confusing the path for NIFI_PID_DIR is different than
> the default which is hardcoded in RunNifi.
>
> I'm trying to figure out if I'm missing anything obvious.
>
> Otherwise I'd suggest changing nifi-env.sh to only set those variables when
> they have not already been set to another value. Does that make sense?
>
> Thank you,
> Lars