You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Karthik Ramakrishnan <ka...@gmail.com> on 2016/09/19 19:58:38 UTC

ETL processors for NiFi

Hello All,

I am currently developing an ETL pipeline and I am using Apache NiFi for
this task. I would like to know if there is any processor that help with
the following use case: I am consuming from Kafka and based on some
attributes in the message then I am updating the database with the data. I
have a GetKafka for getting the messages and a ExecuteSQL processor for
updating the database.  I am stuck in finding a processor that reads the
Kafka message and checks the attribute and then sends the data values to
the executeSQL processor. Can someone help on this?

Thanks!!

-- 
Thanks,
*Karthik Ramakrishnan*
*Data Services Intern*
*Copart Inc.*
*Dallas, TX*
*Contact : +1 (469) 951-8854*

Re: ETL processors for NiFi

Posted by Karthik Ramakrishnan <ka...@gmail.com>.
Hey Mark,

This would help me a lot in building the flow. Thanks a bunch!!

Warm Regards,
Karthik R.

On Mon, Sep 19, 2016 at 3:11 PM, Mark Payne <ma...@hotmail.com> wrote:

> Hi Karthik,
>
> I think what you want to be using here is PutSQL, rather than ExecuteSQL.
> ExecuteSQL
> is designed to perform a SELECT statement, whereas PutSQL would update a
> database.
>
> PutSQL expects the incoming FlowFile to contain the SQL to execute.
> So you could use ReplaceText with the 'Replacement Strategy' set to
> "Always Replace"
> and set the Replacement Value to whatever SQL you want to use. The
> ReplaceText processor
> supports the Expression Language, so you can reference whatever FlowFile
> attributes you would
> like to formulate the message.
>
> How to get information from the Kafka message into your FlowFile
> attributes depends on the type of
> data that you are pulling. If it is JSON, for instance, you could use
> EvaluateJsonPath to extract out
> particular pieces of information into FlowFile attributes.
>
> Does this help?
>
> Thanks
> -Mark
>
> On Sep 19, 2016, at 3:58 PM, Karthik Ramakrishnan <
> karthik.ramakrishnan20@gmail.com> wrote:
>
> Hello All,
>
> I am currently developing an ETL pipeline and I am using Apache NiFi for
> this task. I would like to know if there is any processor that help with
> the following use case: I am consuming from Kafka and based on some
> attributes in the message then I am updating the database with the data. I
> have a GetKafka for getting the messages and a ExecuteSQL processor for
> updating the database.  I am stuck in finding a processor that reads the
> Kafka message and checks the attribute and then sends the data values to
> the executeSQL processor. Can someone help on this?
>
> Thanks!!
>
> --
> Thanks,
> *Karthik Ramakrishnan*
> *Data Services Intern*
> *Copart Inc.*
> *Dallas, TX*
> *Contact : +1 (469) 951-8854 <%2B1%20%28469%29%20951-8854>*
>
>
>


-- 
Thanks,
*Karthik Ramakrishnan*
*Contact : +1 (469) 951-8854*

Re: ETL processors for NiFi

Posted by Mark Payne <ma...@hotmail.com>.
Hi Karthik,

I think what you want to be using here is PutSQL, rather than ExecuteSQL. ExecuteSQL
is designed to perform a SELECT statement, whereas PutSQL would update a database.

PutSQL expects the incoming FlowFile to contain the SQL to execute.
So you could use ReplaceText with the 'Replacement Strategy' set to "Always Replace"
and set the Replacement Value to whatever SQL you want to use. The ReplaceText processor
supports the Expression Language, so you can reference whatever FlowFile attributes you would
like to formulate the message.

How to get information from the Kafka message into your FlowFile attributes depends on the type of
data that you are pulling. If it is JSON, for instance, you could use EvaluateJsonPath to extract out
particular pieces of information into FlowFile attributes.

Does this help?

Thanks
-Mark

> On Sep 19, 2016, at 3:58 PM, Karthik Ramakrishnan <ka...@gmail.com> wrote:
> 
> Hello All,
> 
> I am currently developing an ETL pipeline and I am using Apache NiFi for this task. I would like to know if there is any processor that help with the following use case: I am consuming from Kafka and based on some attributes in the message then I am updating the database with the data. I have a GetKafka for getting the messages and a ExecuteSQL processor for updating the database.  I am stuck in finding a processor that reads the Kafka message and checks the attribute and then sends the data values to the executeSQL processor. Can someone help on this? 
> 
> Thanks!!
> 
> -- 
> Thanks,
> Karthik Ramakrishnan
> Data Services Intern
> Copart Inc.
> Dallas, TX
> Contact : +1 (469) 951-8854
>