You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Ron Sawyer <ro...@yahoo.com> on 2015/10/08 18:38:46 UTC
logging GetFile processes?
Hopefully this isn't asking too much, but I have several getFile processes running and would like to log the file names and times they process into a mongo collection to easily track what and when files have been handled, can someone tell how to do this or point me in a direction?
Thanks.
v/r,
Ron Sawyer
Re: logging GetFile processes?
Posted by Joe Witt <jo...@gmail.com>.
"what would be the flow to write the filename and date to a text file?"
You can use ReplaceText to write out the data in whatever text
oriented format you like (CSV, JSON, etc..).
Then it is a matter of getting that object to Mongo. I don't know
Mongo well enough myself to know the best options there.
Thanks
Joe
On Thu, Oct 8, 2015 at 1:45 PM, Ron Sawyer <ro...@yahoo.com> wrote:
> Joe,
> it appears that the version I'm using doesn't currently support putMongo. So until I can fix that, what would be the flow to write the filename and date to a text file? ExecuteScript?
>
> v/r,
> Ron Sawyer
>
> --------------------------------------------
> On Thu, 10/8/15, Joe Witt <jo...@gmail.com> wrote:
>
> Subject: Re: logging GetFile processes?
> To: users@nifi.apache.org
> Date: Thursday, October 8, 2015, 1:04 PM
>
> Ron,
>
> Yep makes sense. We'll try to put
> together a flow template for you to
> check
> out. Basic gist is
>
> <your flow> - > ReplaceText ->
> PutMongo
>
> In the ReplaceText
> processor you can use the expression language to
> make new content which would be a JSON document
> containing the
> filename and entry time and
> whatever other values you'd like. Then
> drive those new JSON documents into Mongo.
>
> Does that get you on the path
> or would the template help?
>
> Thanks
> Joe
>
> On
> Thu, Oct 8, 2015 at 12:59 PM, Ron Sawyer <ro...@yahoo.com>
> wrote:
> > Would prefer to log them in a
> mongodb collection, just the file name and date processed is
> all.
> >
> > I know this
> is being logged in the app log I think but need something
> that is publicly viewable so others can know files are in
> fact being picked up and processed.
> >
> >
> > v/r,
> > Ron Sawyer
> >
> >
> --------------------------------------------
> > On Thu, 10/8/15, David Wynne <dw...@hortonworks.com>
> wrote:
> >
> > Subject:
> Re: logging GetFile processes?
> > To:
> "Ron Sawyer" <ro...@yahoo.com>
> > Cc: "users@nifi.apache.org"
> <us...@nifi.apache.org>
> > Date: Thursday, October 8, 2015, 12:43
> PM
> >
> > Ron,
> > How
> > do you want
> to keep the file names? In a file on disk? In a
> > database?
> >
> >
> >
> >
> >
> David.
> >
> >
> >
> > On 10/8/15, 12:38
> PM, "Ron Sawyer"
> > <ro...@yahoo.com>
> > wrote:
> >
> > >Hopefully this
> > isn't asking too much, but I have
> several getFile
> > processes running and
> would like to log the file names and
> >
> times they process into a mongo collection to easily
> track
> > what and when files have been
> handled, can someone tell how
> > to do
> this or point me in a direction?
> >
> >
> > >Thanks.
> > >
> > >v/r,
> > >Ron Sawyer
> >
> >
> >
>
Re: logging GetFile processes?
Posted by Ron Sawyer <ro...@yahoo.com>.
Yep, that's it, thank you!!!
v/r,
Ron Sawyer
--------------------------------------------
On Thu, 10/8/15, Joe Witt <jo...@gmail.com> wrote:
Subject: Re: logging GetFile processes?
To: users@nifi.apache.org
Date: Thursday, October 8, 2015, 1:04 PM
Ron,
Yep makes sense. We'll try to put
together a flow template for you to
check
out. Basic gist is
<your flow> - > ReplaceText ->
PutMongo
In the ReplaceText
processor you can use the expression language to
make new content which would be a JSON document
containing the
filename and entry time and
whatever other values you'd like. Then
drive those new JSON documents into Mongo.
Does that get you on the path
or would the template help?
Thanks
Joe
On
Thu, Oct 8, 2015 at 12:59 PM, Ron Sawyer <ro...@yahoo.com>
wrote:
> Would prefer to log them in a
mongodb collection, just the file name and date processed is
all.
>
> I know this
is being logged in the app log I think but need something
that is publicly viewable so others can know files are in
fact being picked up and processed.
>
>
> v/r,
> Ron Sawyer
>
>
--------------------------------------------
> On Thu, 10/8/15, David Wynne <dw...@hortonworks.com>
wrote:
>
> Subject:
Re: logging GetFile processes?
> To:
"Ron Sawyer" <ro...@yahoo.com>
> Cc: "users@nifi.apache.org"
<us...@nifi.apache.org>
> Date: Thursday, October 8, 2015, 12:43
PM
>
> Ron,
> How
> do you want
to keep the file names? In a file on disk? In a
> database?
>
>
>
>
>
David.
>
>
>
> On 10/8/15, 12:38
PM, "Ron Sawyer"
> <ro...@yahoo.com>
> wrote:
>
> >Hopefully this
> isn't asking too much, but I have
several getFile
> processes running and
would like to log the file names and
>
times they process into a mongo collection to easily
track
> what and when files have been
handled, can someone tell how
> to do
this or point me in a direction?
>
>
> >Thanks.
> >
> >v/r,
> >Ron Sawyer
>
>
>
Re: logging GetFile processes?
Posted by Ron Sawyer <ro...@yahoo.com>.
Joe,
it appears that the version I'm using doesn't currently support putMongo. So until I can fix that, what would be the flow to write the filename and date to a text file? ExecuteScript?
v/r,
Ron Sawyer
--------------------------------------------
On Thu, 10/8/15, Joe Witt <jo...@gmail.com> wrote:
Subject: Re: logging GetFile processes?
To: users@nifi.apache.org
Date: Thursday, October 8, 2015, 1:04 PM
Ron,
Yep makes sense. We'll try to put
together a flow template for you to
check
out. Basic gist is
<your flow> - > ReplaceText ->
PutMongo
In the ReplaceText
processor you can use the expression language to
make new content which would be a JSON document
containing the
filename and entry time and
whatever other values you'd like. Then
drive those new JSON documents into Mongo.
Does that get you on the path
or would the template help?
Thanks
Joe
On
Thu, Oct 8, 2015 at 12:59 PM, Ron Sawyer <ro...@yahoo.com>
wrote:
> Would prefer to log them in a
mongodb collection, just the file name and date processed is
all.
>
> I know this
is being logged in the app log I think but need something
that is publicly viewable so others can know files are in
fact being picked up and processed.
>
>
> v/r,
> Ron Sawyer
>
>
--------------------------------------------
> On Thu, 10/8/15, David Wynne <dw...@hortonworks.com>
wrote:
>
> Subject:
Re: logging GetFile processes?
> To:
"Ron Sawyer" <ro...@yahoo.com>
> Cc: "users@nifi.apache.org"
<us...@nifi.apache.org>
> Date: Thursday, October 8, 2015, 12:43
PM
>
> Ron,
> How
> do you want
to keep the file names? In a file on disk? In a
> database?
>
>
>
>
>
David.
>
>
>
> On 10/8/15, 12:38
PM, "Ron Sawyer"
> <ro...@yahoo.com>
> wrote:
>
> >Hopefully this
> isn't asking too much, but I have
several getFile
> processes running and
would like to log the file names and
>
times they process into a mongo collection to easily
track
> what and when files have been
handled, can someone tell how
> to do
this or point me in a direction?
>
>
> >Thanks.
> >
> >v/r,
> >Ron Sawyer
>
>
>
Re: logging GetFile processes?
Posted by Joe Witt <jo...@gmail.com>.
Ron,
Yep makes sense. We'll try to put together a flow template for you to
check out. Basic gist is
<your flow> - > ReplaceText -> PutMongo
In the ReplaceText processor you can use the expression language to
make new content which would be a JSON document containing the
filename and entry time and whatever other values you'd like. Then
drive those new JSON documents into Mongo.
Does that get you on the path or would the template help?
Thanks
Joe
On Thu, Oct 8, 2015 at 12:59 PM, Ron Sawyer <ro...@yahoo.com> wrote:
> Would prefer to log them in a mongodb collection, just the file name and date processed is all.
>
> I know this is being logged in the app log I think but need something that is publicly viewable so others can know files are in fact being picked up and processed.
>
>
> v/r,
> Ron Sawyer
>
> --------------------------------------------
> On Thu, 10/8/15, David Wynne <dw...@hortonworks.com> wrote:
>
> Subject: Re: logging GetFile processes?
> To: "Ron Sawyer" <ro...@yahoo.com>
> Cc: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Thursday, October 8, 2015, 12:43 PM
>
> Ron,
> How
> do you want to keep the file names? In a file on disk? In a
> database?
>
>
>
>
> David.
>
>
>
> On 10/8/15, 12:38 PM, "Ron Sawyer"
> <ro...@yahoo.com>
> wrote:
>
> >Hopefully this
> isn't asking too much, but I have several getFile
> processes running and would like to log the file names and
> times they process into a mongo collection to easily track
> what and when files have been handled, can someone tell how
> to do this or point me in a direction?
> >
> >Thanks.
> >
> >v/r,
> >Ron Sawyer
> >
>
Re: logging GetFile processes?
Posted by Ron Sawyer <ro...@yahoo.com>.
Would prefer to log them in a mongodb collection, just the file name and date processed is all.
I know this is being logged in the app log I think but need something that is publicly viewable so others can know files are in fact being picked up and processed.
v/r,
Ron Sawyer
--------------------------------------------
On Thu, 10/8/15, David Wynne <dw...@hortonworks.com> wrote:
Subject: Re: logging GetFile processes?
To: "Ron Sawyer" <ro...@yahoo.com>
Cc: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, October 8, 2015, 12:43 PM
Ron,
How
do you want to keep the file names? In a file on disk? In a
database?
David.
On 10/8/15, 12:38 PM, "Ron Sawyer"
<ro...@yahoo.com>
wrote:
>Hopefully this
isn't asking too much, but I have several getFile
processes running and would like to log the file names and
times they process into a mongo collection to easily track
what and when files have been handled, can someone tell how
to do this or point me in a direction?
>
>Thanks.
>
>v/r,
>Ron Sawyer
>
Re: logging GetFile processes?
Posted by David Wynne <dw...@hortonworks.com>.
Ron,
How do you want to keep the file names? In a file on disk? In a database?
David.
On 10/8/15, 12:38 PM, "Ron Sawyer" <ro...@yahoo.com> wrote:
>Hopefully this isn't asking too much, but I have several getFile processes running and would like to log the file names and times they process into a mongo collection to easily track what and when files have been handled, can someone tell how to do this or point me in a direction?
>
>Thanks.
>
>v/r,
>Ron Sawyer
>