You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nifi.apache.org by Gunjan Dave <gu...@gmail.com> on 2016/09/15 17:27:08 UTC

Custom Processor

Hi Nifi Team,
I want to create a custom processor in which I want to use the database
functionality.
Such as, i want to extract some information from database as part of
processor. I could Access the database controller service but how do I
reuse executesql processor in my custom processor?

Re: Custom Processor

Posted by Oleg Zhurakousky <oz...@hortonworks.com>.

Gunjan

If I understand you correctly you are trying to do something that works with JMS and SQL in atomic fashion. Possibly (and that is purely a guess) you are having some XA transactional concerns. . .
If so, that may be a bit counter-intuitive to micro-services nature of NiFi processors where each processor is an independent service with queues in between.
If that’s the case I would probably go with custom processor from scratch yet utilizing ControllerServices for JMS ConnectionFactory and SQL connection pool.

We can definitely help you with that, but I’ll stop right here just to see if you can clarify further

Cheers
Oleg

> On Sep 16, 2016, at 1:19 AM, Gunjan Dave <gu...@gmail.com> wrote:
> 
> Hi Oleg,
> Executesql is not the problem, but I am trying to achive something
> different here. Exevutesql was just an example.
> 
> Based on following article,
> https://dzone.com/articles/apache-nifi-not-from-scratch
> 
> i was trying to export classes of my existing application onto nifi custom
> processor. Some classes have multiple database calls and i was thinking of
> how I can do it. Those classes also have mq connection, so extending only
> executesql will not help me.
> 
> is there some other way I can use consumejms or publishjms and nifi
> sqlprocessors in my custom processor?
> 
> I understand that I can create these functionalities by using process
> groups and specific processors on nifi ui, but was just being adventurous
> on using NiFi differently by creating business specific custom processor
> 
> Anyone who has tried such a thing in past and give me some reference?
> 
> THANKS
> Gunjan
> 
> On Thu, Sep 15, 2016, 10:56 PM Gunjan Dave <gu...@gmail.com>
> wrote:
> 
>> Hi Nifi Team,
>> I want to create a custom processor in which I want to use the database
>> functionality.
>> Such as, i want to extract some information from database as part of
>> processor. I could Access the database controller service but how do I
>> reuse executesql processor in my custom processor?
>>

Re: Custom Processor

Posted by Gunjan Dave <gu...@gmail.com>.

Hi Oleg,
Executesql is not the problem, but I am trying to achive something
different here. Exevutesql was just an example.

Based on following article,
https://dzone.com/articles/apache-nifi-not-from-scratch

i was trying to export classes of my existing application onto nifi custom
processor. Some classes have multiple database calls and i was thinking of
how I can do it. Those classes also have mq connection, so extending only
executesql will not help me.

is there some other way I can use consumejms or publishjms and nifi
sqlprocessors in my custom processor?

I understand that I can create these functionalities by using process
groups and specific processors on nifi ui, but was just being adventurous
on using NiFi differently by creating business specific custom processor

Anyone who has tried such a thing in past and give me some reference?

THANKS
Gunjan

On Thu, Sep 15, 2016, 10:56 PM Gunjan Dave <gu...@gmail.com>
wrote:

> Hi Nifi Team,
> I want to create a custom processor in which I want to use the database
> functionality.
> Such as, i want to extract some information from database as part of
> processor. I could Access the database controller service but how do I
> reuse executesql processor in my custom processor?
>

Re: Custom Processor

Posted by Manish <ma...@gmail.com>.

Hi,

One enhancement I could think of would be to give an options to bring data
(in JSON) in FlowFile attribute instead of content. It will be up to the
user to ensure that not too much of data is pulled back causing heap
issues, and also not hit the database frequently.

Regards,
Manish


On Thu, Sep 15, 2016 at 2:55 PM, Oleg Zhurakousky [via Apache NiFi
Developer List] <ml...@n7.nabble.com> wrote:

> Hi
>
> You can obviously extends the executesql processor, but I am more
> interested in understanding what exactly are you trying to accomplish that
> executesql doesn’t do today. Don’t get me wrong as I am sure you may have a
> legitimate reason, but there may also be missing functionality that we may
> add to executesql that would satisfy your use case.
>
> Anyway, more details would definitely help.
> Cheers
> Oleg
>
> > On Sep 15, 2016, at 1:27 PM, Gunjan Dave <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=13373&i=0>> wrote:
> >
> > Hi Nifi Team,
> > I want to create a custom processor in which I want to use the database
> > functionality.
> > Such as, i want to extract some information from database as part of
> > processor. I could Access the database controller service but how do I
> > reuse executesql processor in my custom processor?
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Custom-Processor-
> tp13372p13373.html
> To start a new topic under Apache NiFi Developer List, email
> ml-node+s39713n1h73@n7.nabble.com
> To unsubscribe from Apache NiFi Developer List, click here
> <http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=bWFuaXNoLmdmb3JjZUBnbWFpbC5jb218MXwyMDc4ODk1NjEz>
> .
> NAML
> <http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Custom-Processor-tp13372p13374.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Custom Processor

Posted by Oleg Zhurakousky <oz...@hortonworks.com>.

Hi

You can obviously extends the executesql processor, but I am more interested in understanding what exactly are you trying to accomplish that executesql doesn’t do today. Don’t get me wrong as I am sure you may have a legitimate reason, but there may also be missing functionality that we may add to executesql that would satisfy your use case.

Anyway, more details would definitely help.
Cheers
Oleg

> On Sep 15, 2016, at 1:27 PM, Gunjan Dave <gu...@gmail.com> wrote:
> 
> Hi Nifi Team,
> I want to create a custom processor in which I want to use the database
> functionality.
> Such as, i want to extract some information from database as part of
> processor. I could Access the database controller service but how do I
> reuse executesql processor in my custom processor?