You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Charlie Frasure <ch...@gmail.com> on 2016/03/04 16:38:18 UTC
List Files
I'm using the 0.5.1 build and having what I think is odd behavior for
ListFile. The processor supports expression language in the Input
Directory property, however I can't figure out how to configure an
attribute as input.
I tried using UpdateAttribute prior to ListFile, but ListFile doesn't seem
to allow incoming connections. I also tried creating an attribute within
ListFile, but it doesn't seem to be available for referencing within the
same processor.
Is this expected? If so, what is the intended purpose of allowing
expression language in this attribute?
Re: Re: List Files
Posted by Thad Guidry <th...@gmail.com>.
It should be currently possible to use the ExecuteScript processor (with
Groovy language) and set those needed user variables like Paths,etc using
whatever System environment variables you wish to use and retrieve later.
In Groovy scripts you can do this:
System.setProperty("socksProxyPort","2003")
System.setProperty("socksProxyHost","localhost")
System.setProperty("ourPathToHell","DonaldTrump")
Thad
+ThadGuidry <https://www.google.com/+ThadGuidry>
Re: Re: List Files
Posted by Charlie Frasure <ch...@gmail.com>.
Thanks Matthew and Joe,
I must have missed the automatic environment variable mapping in expression
language. I will plan on utilizing this method until the Variable Registry
is available.
On Fri, Mar 4, 2016 at 1:57 PM, Joe Witt <jo...@gmail.com> wrote:
> Charlie,
>
> Yeah you're right that is a viable option now too. And we support
> environment variables from the system too. So for things that support
> expression language that helps.
>
> The variable registry obviously will give us a nice step forward.
>
> Thanks
> Joe
>
> On Fri, Mar 4, 2016 at 1:55 PM, Charlie Frasure
> <ch...@gmail.com> wrote:
> > Thanks Joe,
> >
> > That option is certainly more comprehensive than what I was looking to
> do.
> > An UpdateAttribute at the beginning of a flow (or even before a process
> > group) could establish the variables I need. One case that either
> > UpdateAttribute or the Variable Registry should consider is being able to
> > mark sensitive properties so that things like passwords can remain
> secure.
> >
> >
> > On Fri, Mar 4, 2016 at 1:24 PM, Joe Witt <jo...@gmail.com> wrote:
> >>
> >> No. But we need to and plan to
> >> https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
> >>
> >> On Fri, Mar 4, 2016 at 1:22 PM, Uwe Geercken <uw...@web.de>
> wrote:
> >> > Matthew,
> >> >
> >> > does NiFi itself allow to define such things as constants? The idea
> >> > would be
> >> > to re-use these constants (e.g. a path) in different processors.
> >> >
> >> > Rgds,
> >> >
> >> > Uwe
> >> >
> >> > Gesendet: Freitag, 04. März 2016 um 19:04 Uhr
> >> > Von: "Matthew Clarke" <ma...@gmail.com>
> >> > An: users@nifi.apache.org, "Joe Percivall" <jo...@yahoo.com>
> >> > Betreff: Re: List Files
> >> >
> >> > The expression language can also be used to return values from both
> JVM
> >> > properties set in the NiFi bootstrap.conf and system environment
> >> > variables
> >> > for the user running NiFi.
> >> >
> >> > On Mar 4, 2016 11:00 AM, "Joe Percivall" <jo...@yahoo.com>
> wrote:
> >> >>
> >> >> Hello,
> >> >>
> >> >> ListFile is a source processor so this behavior is expected. It
> >> >> supports
> >> >> expression language so that it can be configured to utilize certains
> >> >> methods. For example, some people may want to get a list of files
> from
> >> >> a
> >> >> rotating directory that gets created every hour. To do that they
> would
> >> >> need
> >> >> to use the date functions included in Expression Language.
> >> >>
> >> >> Does forbidding input hinder a specific use-case you have?
> >> >>
> >> >> Joe
> >> >>
> >> >> - - - - - -
> >> >> Joseph Percivall
> >> >> linkedin.com/in/Percivall
> >> >> e: joepercivall@yahoo.com
> >> >>
> >> >>
> >> >>
> >> >> On Friday, March 4, 2016 10:38 AM, Charlie Frasure
> >> >> <ch...@gmail.com> wrote:
> >> >>
> >> >>
> >> >>
> >> >> I'm using the 0.5.1 build and having what I think is odd behavior for
> >> >> ListFile. The processor supports expression language in the Input
> >> >> Directory
> >> >> property, however I can't figure out how to configure an attribute as
> >> >> input.
> >> >>
> >> >> I tried using UpdateAttribute prior to ListFile, but ListFile doesn't
> >> >> seem
> >> >> to allow incoming connections. I also tried creating an attribute
> >> >> within
> >> >> ListFile, but it doesn't seem to be available for referencing within
> >> >> the
> >> >> same processor.
> >> >>
> >> >> Is this expected? If so, what is the intended purpose of allowing
> >> >> expression language in this attribute?
> >
> >
>
Re: Re: List Files
Posted by Joe Witt <jo...@gmail.com>.
Charlie,
Yeah you're right that is a viable option now too. And we support
environment variables from the system too. So for things that support
expression language that helps.
The variable registry obviously will give us a nice step forward.
Thanks
Joe
On Fri, Mar 4, 2016 at 1:55 PM, Charlie Frasure
<ch...@gmail.com> wrote:
> Thanks Joe,
>
> That option is certainly more comprehensive than what I was looking to do.
> An UpdateAttribute at the beginning of a flow (or even before a process
> group) could establish the variables I need. One case that either
> UpdateAttribute or the Variable Registry should consider is being able to
> mark sensitive properties so that things like passwords can remain secure.
>
>
> On Fri, Mar 4, 2016 at 1:24 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>> No. But we need to and plan to
>> https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>>
>> On Fri, Mar 4, 2016 at 1:22 PM, Uwe Geercken <uw...@web.de> wrote:
>> > Matthew,
>> >
>> > does NiFi itself allow to define such things as constants? The idea
>> > would be
>> > to re-use these constants (e.g. a path) in different processors.
>> >
>> > Rgds,
>> >
>> > Uwe
>> >
>> > Gesendet: Freitag, 04. März 2016 um 19:04 Uhr
>> > Von: "Matthew Clarke" <ma...@gmail.com>
>> > An: users@nifi.apache.org, "Joe Percivall" <jo...@yahoo.com>
>> > Betreff: Re: List Files
>> >
>> > The expression language can also be used to return values from both JVM
>> > properties set in the NiFi bootstrap.conf and system environment
>> > variables
>> > for the user running NiFi.
>> >
>> > On Mar 4, 2016 11:00 AM, "Joe Percivall" <jo...@yahoo.com> wrote:
>> >>
>> >> Hello,
>> >>
>> >> ListFile is a source processor so this behavior is expected. It
>> >> supports
>> >> expression language so that it can be configured to utilize certains
>> >> methods. For example, some people may want to get a list of files from
>> >> a
>> >> rotating directory that gets created every hour. To do that they would
>> >> need
>> >> to use the date functions included in Expression Language.
>> >>
>> >> Does forbidding input hinder a specific use-case you have?
>> >>
>> >> Joe
>> >>
>> >> - - - - - -
>> >> Joseph Percivall
>> >> linkedin.com/in/Percivall
>> >> e: joepercivall@yahoo.com
>> >>
>> >>
>> >>
>> >> On Friday, March 4, 2016 10:38 AM, Charlie Frasure
>> >> <ch...@gmail.com> wrote:
>> >>
>> >>
>> >>
>> >> I'm using the 0.5.1 build and having what I think is odd behavior for
>> >> ListFile. The processor supports expression language in the Input
>> >> Directory
>> >> property, however I can't figure out how to configure an attribute as
>> >> input.
>> >>
>> >> I tried using UpdateAttribute prior to ListFile, but ListFile doesn't
>> >> seem
>> >> to allow incoming connections. I also tried creating an attribute
>> >> within
>> >> ListFile, but it doesn't seem to be available for referencing within
>> >> the
>> >> same processor.
>> >>
>> >> Is this expected? If so, what is the intended purpose of allowing
>> >> expression language in this attribute?
>
>
Re: PutMongo Processor
Posted by Bryan Bende <bb...@gmail.com>.
Uwe,
Personally I don't have that much experience with MongoDB, but the
additional functionality you described sounds like something we would want
to support. Looking through JIRA I only see one ticket related to MongoDB
to add SSL support [1] so I think it would be great to create a new JIRA to
capture these ideas and any specifics about how you envision it working.
Thanks,
Bryan
[1] https://issues.apache.org/jira/browse/NIFI-1197
On Fri, Mar 4, 2016 at 6:43 PM, Uwe Geercken <uw...@web.de> wrote:
> Hello,
>
> I have tried to use the PutMongo processor. I read Json files from a
> folder and send them to a mongo database.
>
> The insert of documents works seamlessly.
>
> Next I tested updates. The problem here is that for the update a complete
> document is required. So if you insert a document with 5 key/value pairs
> and then make an update using a Json with only 3 key/value pairs then 2
> fields are gone after the update. This is a known behavior in mongodb. But
> mongodb also supports doing updates using the $set operator. In this case
> the key/values pairs procided get updated and the others remain untouched.
> But the nifi processor does not seem to support this technique.
>
> I then went on to test upserts. Same problem here as upserts are basically
> the same as updates. But I laso found out that the processor does not
> support some of the other more enhanced operators such as $push (push a
> key/value into an array) or the $inc operator (increment).
>
> The error message from the nifi log says:
> java.lang.IllegalArgumentException: Invalid BSON field name $push - for
> example.
>
> So I think there is currently only limited support for mongodb. But I hope
> this will be enhanced in the near future taking the importance of mongodb.
>
> What is your experiece with this processor? Or maybe you have a workaround?
>
> Rgds,
>
> Uwe
>
>
>
Re: Re: List Files
Posted by Charlie Frasure <ch...@gmail.com>.
Thanks Joe,
That option is certainly more comprehensive than what I was looking to do.
An UpdateAttribute at the beginning of a flow (or even before a process
group) could establish the variables I need. One case that either
UpdateAttribute or the Variable Registry should consider is being able to
mark sensitive properties so that things like passwords can remain secure.
On Fri, Mar 4, 2016 at 1:24 PM, Joe Witt <jo...@gmail.com> wrote:
> No. But we need to and plan to
> https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>
> On Fri, Mar 4, 2016 at 1:22 PM, Uwe Geercken <uw...@web.de> wrote:
> > Matthew,
> >
> > does NiFi itself allow to define such things as constants? The idea
> would be
> > to re-use these constants (e.g. a path) in different processors.
> >
> > Rgds,
> >
> > Uwe
> >
> > Gesendet: Freitag, 04. März 2016 um 19:04 Uhr
> > Von: "Matthew Clarke" <ma...@gmail.com>
> > An: users@nifi.apache.org, "Joe Percivall" <jo...@yahoo.com>
> > Betreff: Re: List Files
> >
> > The expression language can also be used to return values from both JVM
> > properties set in the NiFi bootstrap.conf and system environment
> variables
> > for the user running NiFi.
> >
> > On Mar 4, 2016 11:00 AM, "Joe Percivall" <jo...@yahoo.com> wrote:
> >>
> >> Hello,
> >>
> >> ListFile is a source processor so this behavior is expected. It supports
> >> expression language so that it can be configured to utilize certains
> >> methods. For example, some people may want to get a list of files from a
> >> rotating directory that gets created every hour. To do that they would
> need
> >> to use the date functions included in Expression Language.
> >>
> >> Does forbidding input hinder a specific use-case you have?
> >>
> >> Joe
> >>
> >> - - - - - -
> >> Joseph Percivall
> >> linkedin.com/in/Percivall
> >> e: joepercivall@yahoo.com
> >>
> >>
> >>
> >> On Friday, March 4, 2016 10:38 AM, Charlie Frasure
> >> <ch...@gmail.com> wrote:
> >>
> >>
> >>
> >> I'm using the 0.5.1 build and having what I think is odd behavior for
> >> ListFile. The processor supports expression language in the Input
> Directory
> >> property, however I can't figure out how to configure an attribute as
> input.
> >>
> >> I tried using UpdateAttribute prior to ListFile, but ListFile doesn't
> seem
> >> to allow incoming connections. I also tried creating an attribute
> within
> >> ListFile, but it doesn't seem to be available for referencing within the
> >> same processor.
> >>
> >> Is this expected? If so, what is the intended purpose of allowing
> >> expression language in this attribute?
>
Re: Re: List Files
Posted by Joe Witt <jo...@gmail.com>.
No. But we need to and plan to
https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
On Fri, Mar 4, 2016 at 1:22 PM, Uwe Geercken <uw...@web.de> wrote:
> Matthew,
>
> does NiFi itself allow to define such things as constants? The idea would be
> to re-use these constants (e.g. a path) in different processors.
>
> Rgds,
>
> Uwe
>
> Gesendet: Freitag, 04. März 2016 um 19:04 Uhr
> Von: "Matthew Clarke" <ma...@gmail.com>
> An: users@nifi.apache.org, "Joe Percivall" <jo...@yahoo.com>
> Betreff: Re: List Files
>
> The expression language can also be used to return values from both JVM
> properties set in the NiFi bootstrap.conf and system environment variables
> for the user running NiFi.
>
> On Mar 4, 2016 11:00 AM, "Joe Percivall" <jo...@yahoo.com> wrote:
>>
>> Hello,
>>
>> ListFile is a source processor so this behavior is expected. It supports
>> expression language so that it can be configured to utilize certains
>> methods. For example, some people may want to get a list of files from a
>> rotating directory that gets created every hour. To do that they would need
>> to use the date functions included in Expression Language.
>>
>> Does forbidding input hinder a specific use-case you have?
>>
>> Joe
>>
>> - - - - - -
>> Joseph Percivall
>> linkedin.com/in/Percivall
>> e: joepercivall@yahoo.com
>>
>>
>>
>> On Friday, March 4, 2016 10:38 AM, Charlie Frasure
>> <ch...@gmail.com> wrote:
>>
>>
>>
>> I'm using the 0.5.1 build and having what I think is odd behavior for
>> ListFile. The processor supports expression language in the Input Directory
>> property, however I can't figure out how to configure an attribute as input.
>>
>> I tried using UpdateAttribute prior to ListFile, but ListFile doesn't seem
>> to allow incoming connections. I also tried creating an attribute within
>> ListFile, but it doesn't seem to be available for referencing within the
>> same processor.
>>
>> Is this expected? If so, what is the intended purpose of allowing
>> expression language in this attribute?
Re: List Files
Posted by Matthew Clarke <ma...@gmail.com>.
The expression language can also be used to return values from both JVM
properties set in the NiFi bootstrap.conf and system environment variables
for the user running NiFi.
On Mar 4, 2016 11:00 AM, "Joe Percivall" <jo...@yahoo.com> wrote:
> Hello,
>
> ListFile is a source processor so this behavior is expected. It supports
> expression language so that it can be configured to utilize certains
> methods. For example, some people may want to get a list of files from a
> rotating directory that gets created every hour. To do that they would need
> to use the date functions included in Expression Language.
>
> Does forbidding input hinder a specific use-case you have?
>
> Joe
>
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joepercivall@yahoo.com
>
>
>
> On Friday, March 4, 2016 10:38 AM, Charlie Frasure <
> charliefrasure@gmail.com> wrote:
>
>
>
> I'm using the 0.5.1 build and having what I think is odd behavior for
> ListFile. The processor supports expression language in the Input
> Directory property, however I can't figure out how to configure an
> attribute as input.
>
> I tried using UpdateAttribute prior to ListFile, but ListFile doesn't seem
> to allow incoming connections. I also tried creating an attribute within
> ListFile, but it doesn't seem to be available for referencing within the
> same processor.
>
> Is this expected? If so, what is the intended purpose of allowing
> expression language in this attribute?
>
Re: List Files
Posted by Joe Percivall <jo...@yahoo.com>.
Hello,
ListFile is a source processor so this behavior is expected. It supports expression language so that it can be configured to utilize certains methods. For example, some people may want to get a list of files from a rotating directory that gets created every hour. To do that they would need to use the date functions included in Expression Language.
Does forbidding input hinder a specific use-case you have?
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Friday, March 4, 2016 10:38 AM, Charlie Frasure <ch...@gmail.com> wrote:
I'm using the 0.5.1 build and having what I think is odd behavior for ListFile. The processor supports expression language in the Input Directory property, however I can't figure out how to configure an attribute as input.
I tried using UpdateAttribute prior to ListFile, but ListFile doesn't seem to allow incoming connections. I also tried creating an attribute within ListFile, but it doesn't seem to be available for referencing within the same processor.
Is this expected? If so, what is the intended purpose of allowing expression language in this attribute?