You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by sivam1 <sh...@gmail.com> on 2016/04/26 16:53:42 UTC
Sample to read file and insert into oracle based on the pattern in
the line
Hi,
I am new to Nifi, I have use case that is achieved using Mule Batch and
Spring Batch+Data. I would like to replicate using Nifi. Here is the
scenario.
1. Read a file from remote location ( *.txt)
2. Base on pattern matching ( lets say xxx, yyy as a pattern in the lines),
I should be able to insert xxx in to xxx_table, yyy into yyy_table.
Please share the sample code.
Regards
--Shan
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Sample to read file and insert into oracle based on the pattern
in the line
Posted by sivam1 <sh...@gmail.com>.
Thanks Bryan. Love to see one sample if you could help me with one.
Regards
Siva.
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9651.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Sample to read file and insert into oracle based on the pattern
in the line
Posted by Bryan Bende <bb...@gmail.com>.
Siva,
If I am understanding your scenario correctly, the processors to look at
would likely be the following...
GetFTP - to retrieve the remote file into NiFi
RouteText - to separate the lines of the file based on a pattern
SplitText - to split the output of RouteText into one line per flow file
ExtractText - to extract some pattern from the content into attributes
ReplaceText - to construct a SQL query
PutSQL - to perform an insert into your database
The above may not be the exact flow, but would likely be a starting point.
Thanks,
Bryan
On Wed, Apr 27, 2016 at 9:07 AM, sivam1 <sh...@gmail.com> wrote:
> Hi Pierre,
> Thanks for the reply. Remote location means, FTP or a remote Network File
> system using file://
> As I said, I executed all these in Mule ESB, Spring Batch. Wanting to see
> in the Nifi as I could see Nifi as great future in handling BigData.
>
> Regards
> Siva.
>
>
> On Wed, Apr 27, 2016 at 8:59 AM, Pierre Villard [via Apache NiFi Developer
> List] <ml...@n7.nabble.com> wrote:
>
> > Hi,
> >
> > Please detail "remote location". How the file can be accessed?
> >
> > Then you can use processors to extract your patterns and use expression
> > language to perform insertions into tables.
> > Depending of what is the technology behind your "tables" you have
> multiple
> > options regarding processors to use.
> >
> > HTH,
> > Pierre.
> >
> >
> > 2016-04-27 14:04 GMT+02:00 sivam1 <[hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=9646&i=0>>:
> >
> > > Any body has sample code for the above? I am new to NiFi and would like
> > to
> > > explore more?
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9645.html
> > > Sent from the Apache NiFi Developer List mailing list archive at
> > > Nabble.com.
> > >
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> >
> http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9646.html
> > To unsubscribe from Sample to read file and insert into oracle based on
> > the pattern in the line, click here
> > <
> http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=9620&code=c2hpdi5zaGl2c2hhbmthckBnbWFpbC5jb218OTYyMHwxMzEyMTU2MjIz
> >
> > .
> > NAML
> > <
> http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9647.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Re: Sample to read file and insert into oracle based on the pattern
in the line
Posted by sivam1 <sh...@gmail.com>.
Thanks Pierre,
Since I am a newbie to Nifi, struggling to get in to the groove, especially
while inserting into Oracle DB i.e mapping values to attributes.. Need some
light on this. Any working sample would make my life simple.
Regards
Siva.
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9652.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Sample to read file and insert into oracle based on the pattern
in the line
Posted by Pierre Villard <pi...@gmail.com>.
You can get your file using GetFtp processor in case of FTP or GetFile in
case the file is available locally.
Then, if the job is done on a line by line basis, you could use
SplitContent processor to have one flow file per line.
Once you have your line, depending of the format, you have multiple
solutions:
- option 1 : you extract the pattern you are looking for to route flow
files according to the pattern
- option 2 : you do all the job by scripting what you need in an
InvokeScripted processor
At this point it really depends of the format of your data, and the final
destination.
The best recommendation would be to install a local NiFi instance and to
try some of the processors by yourself to see what you can achieve. There a
lot of flexiblity provided by NiFi and the flow will change depending of
what kind of data you are handling.
HTH,
Pierre
2016-04-27 15:07 GMT+02:00 sivam1 <sh...@gmail.com>:
> Hi Pierre,
> Thanks for the reply. Remote location means, FTP or a remote Network File
> system using file://
> As I said, I executed all these in Mule ESB, Spring Batch. Wanting to see
> in the Nifi as I could see Nifi as great future in handling BigData.
>
> Regards
> Siva.
>
>
> On Wed, Apr 27, 2016 at 8:59 AM, Pierre Villard [via Apache NiFi Developer
> List] <ml...@n7.nabble.com> wrote:
>
> > Hi,
> >
> > Please detail "remote location". How the file can be accessed?
> >
> > Then you can use processors to extract your patterns and use expression
> > language to perform insertions into tables.
> > Depending of what is the technology behind your "tables" you have
> multiple
> > options regarding processors to use.
> >
> > HTH,
> > Pierre.
> >
> >
> > 2016-04-27 14:04 GMT+02:00 sivam1 <[hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=9646&i=0>>:
> >
> > > Any body has sample code for the above? I am new to NiFi and would like
> > to
> > > explore more?
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9645.html
> > > Sent from the Apache NiFi Developer List mailing list archive at
> > > Nabble.com.
> > >
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> >
> http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9646.html
> > To unsubscribe from Sample to read file and insert into oracle based on
> > the pattern in the line, click here
> > <
> http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=9620&code=c2hpdi5zaGl2c2hhbmthckBnbWFpbC5jb218OTYyMHwxMzEyMTU2MjIz
> >
> > .
> > NAML
> > <
> http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9647.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Re: Sample to read file and insert into oracle based on the pattern
in the line
Posted by sivam1 <sh...@gmail.com>.
Hi Pierre,
Thanks for the reply. Remote location means, FTP or a remote Network File
system using file://
As I said, I executed all these in Mule ESB, Spring Batch. Wanting to see
in the Nifi as I could see Nifi as great future in handling BigData.
Regards
Siva.
On Wed, Apr 27, 2016 at 8:59 AM, Pierre Villard [via Apache NiFi Developer
List] <ml...@n7.nabble.com> wrote:
> Hi,
>
> Please detail "remote location". How the file can be accessed?
>
> Then you can use processors to extract your patterns and use expression
> language to perform insertions into tables.
> Depending of what is the technology behind your "tables" you have multiple
> options regarding processors to use.
>
> HTH,
> Pierre.
>
>
> 2016-04-27 14:04 GMT+02:00 sivam1 <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9646&i=0>>:
>
> > Any body has sample code for the above? I am new to NiFi and would like
> to
> > explore more?
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9645.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> > Nabble.com.
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9646.html
> To unsubscribe from Sample to read file and insert into oracle based on
> the pattern in the line, click here
> <http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=9620&code=c2hpdi5zaGl2c2hhbmthckBnbWFpbC5jb218OTYyMHwxMzEyMTU2MjIz>
> .
> NAML
> <http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9647.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Sample to read file and insert into oracle based on the pattern
in the line
Posted by Pierre Villard <pi...@gmail.com>.
Hi,
Please detail "remote location". How the file can be accessed?
Then you can use processors to extract your patterns and use expression
language to perform insertions into tables.
Depending of what is the technology behind your "tables" you have multiple
options regarding processors to use.
HTH,
Pierre.
2016-04-27 14:04 GMT+02:00 sivam1 <sh...@gmail.com>:
> Any body has sample code for the above? I am new to NiFi and would like to
> explore more?
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9645.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Re: Sample to read file and insert into oracle based on the pattern
in the line
Posted by sivam1 <sh...@gmail.com>.
Any body has sample code for the above? I am new to NiFi and would like to
explore more?
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Sample-to-read-file-and-insert-into-oracle-based-on-the-pattern-in-the-line-tp9620p9645.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.