You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by cool nijandhan <ni...@gmail.com> on 2017/02/09 04:49:54 UTC
Custom Processor with Hive database connection pooling service not working
Hi
I am developing custom processor with hive database connection pooling
service. Added necessary dependency in pom file and added necessary folders
in custom processor folder. I have created class file and able to generate
nar file. Placed nar file in lib directory and restarted nifi server. Its
look like everything working fine. But services are not showing in database
connection dropdown box. Instead it showing connection id, but every time
services are creating when click "create" in controller section. For other
processors, it seems working fine. Please find the attached screenshots.
Any help appreciated.
Thanks
Re: Custom Processor with Hive database connection pooling service
not working
Posted by Nijandan D <ni...@gmail.com>.
Hi Bryan,
Thanks for your update. It's working fine. Thank you so much.
Regards,
Nijandan D
On Fri, Feb 10, 2017 at 7:21 PM, Bryan Bende <bb...@gmail.com> wrote:
> Hello,
>
> In general, to link a processor to a controller service, you need two
> dependencies as shown in the wiki page in the previous email:
>
> - The processor project needs a provided dependency on the JAR where
> the service interface is
> - The NAR project needs a NAR dependency on the NAR where the service
> interface is
>
> In the case of Hive, the interface and implementation are both in
> nifi-hive-processors which get packaged into nifi-hive-nar, so you
> need:
>
> In your processors pom.xml:
>
> <dependency>
> <groupId>org.apache.nifi</groupId>
> <artifactId>nifi-hive-processors</artifactId>
> <version>replace with the version of NiFi you are
> using</version>
> <scope>provided</scope>
> </dependency>
>
> In your NAR pom.xml:
>
> <dependency>
> <groupId>org.apache.nifi</groupId>
> <artifactId>nifi-hive-nar</artifactId>
> <version>replace with the version of NiFi you are
> using</version>
> <type>nar</type>
> </dependency>
>
>
> -Bryan
>
> On Fri, Feb 10, 2017 at 2:25 AM, Nijandan D <ni...@gmail.com> wrote:
> > Hi Bryan,
> >
> > I am new to Nifi, I'm tried to create custom processor with hive
> connection
> > pooling service as per your concern. But
> > same issue is occuring in custom processor. It shows only connection id.
> I
> > have no idea about pom file dependency.
> > Here I mentioned the steps what i have followed to create custom
> processor
> > with hive connection below. If it's wrong,
> > Please provide any link to get workflow for creating custom processor
> with
> > hive database connection pooling service.
> >
> > - Added below line in custom processor java class
> > public static final PropertyDescriptor HIVE_DBCP_SERVICE = new
> > PropertyDescriptor.Builder()
> > .name("Hive Database Connection Pooling Service")
> > .description("The Hive Controller Service that is used to
> obtain
> > connection(s) to the Hive database")
> > .required(true)
> > .identifiesControllerService(HiveDBCPService.class)
> > .build();
> >
> > - After imported HiveDBCPService in same java class
> > import org.apache.nifi.dbcp.hive.HiveDBCPService;
> >
> > - After searched dependency at maven repositories for HiveDBCPService and
> > added
> >
> > - In nifi-custom-nar pom file, added following lines,
> > <dependency>
> > <groupId>org.apache.nifi</groupId>
> > <artifactId>nifi-hive-processors</artifactId>
> > <version>1.0.0</version>
> > </dependency>
> > <dependency>
> > <groupId>org.apache.nifi</groupId>
> > <artifactId>nifi-hadoop-libraries-nar</artifactId>
> > <type>nar</type>
> > </dependency>
> >
> > - In nifi-custom-processor pom file, added following lines,
> > <dependency>
> > <groupId>commons-dbcp</groupId>
> > <artifactId>commons-dbcp</artifactId>
> > <version>1.5-SNAPSHOT</version>
> > <type>jar</type>
> > </dependency>
> > <dependency>
> > <groupId>org.apache.hadoop</groupId>
> > <artifactId>hadoop-core</artifactId>
> > <version>1.2.1</version>
> > <type>jar</type>
> > </dependency>
> > <dependency>
> > <groupId>org.apache.hadoop</groupId>
> > <artifactId>hadoop-common</artifactId>
> > <version>3.0.0-SNAPSHOT</version>
> > <type>jar</type>
> > </dependency>
> > <dependency>
> > <groupId>org.apache.hive</groupId>
> > <artifactId>hive-jdbc</artifactId>
> > <version>1.3.0-SNAPSHOT</version>
> > <type>jar</type>
> > </dependency>
> > <dependency>
> > <groupId>org.apache.nifi</groupId>
> > <artifactId>nifi-hadoop-utils</artifactId>
> > <version>1.0.0</version>
> > <type>jar</type>
> > </dependency>
> > <dependency>
> > <groupId>org.apache.nifi</groupId>
> > <artifactId>nifi-hive-processors</artifactId>
> > <version>1.0.0</version>
> > <type>jar</type>
> > </dependency>
> >
> > - After taking build and replaced Nar file in lib folder, restarted Nifi
> > source. Same error connection id only showing in
> > connection dropdown box.
> >
> > Thanks in advance.
> >
> > Regards,
> > Nijandan D
> >
> > On Thu, Feb 9, 2017 at 8:22 PM, Bryan Bende <bb...@gmail.com> wrote:
> >>
> >> Hello,
> >>
> >> This usually means your custom NAR is not correctly linked to the NAR
> >> where the service is.
> >>
> >> Take a look at these resources for examples:
> >>
> >>
> >> https://cwiki.apache.org/confluence/display/NIFI/Maven+
> Projects+for+Extensions#MavenProjectsforExtensions-
> LinkingProcessorsandControllerServices
> >>
> >>
> >> https://github.com/bbende/nifi-dependency-example/blob/
> master/nifi-example-processor-bundle/nifi-example-
> processors-nar/pom.xml#L40
> >>
> >> Thanks,
> >>
> >> Bryan
> >>
> >> On Wed, Feb 8, 2017 at 11:49 PM, cool nijandhan <ni...@gmail.com>
> >> wrote:
> >> > Hi
> >> >
> >> > I am developing custom processor with hive database connection pooling
> >> > service. Added necessary dependency in pom file and added necessary
> >> > folders
> >> > in custom processor folder. I have created class file and able to
> >> > generate
> >> > nar file. Placed nar file in lib directory and restarted nifi server.
> >> > Its
> >> > look like everything working fine. But services are not showing in
> >> > database
> >> > connection dropdown box. Instead it showing connection id, but every
> >> > time
> >> > services are creating when click "create" in controller section. For
> >> > other
> >> > processors, it seems working fine. Please find the attached
> screenshots.
> >> >
> >> > Any help appreciated.
> >> >
> >> > Thanks
> >
> >
>
Re: Custom Processor with Hive database connection pooling service
not working
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
In general, to link a processor to a controller service, you need two
dependencies as shown in the wiki page in the previous email:
- The processor project needs a provided dependency on the JAR where
the service interface is
- The NAR project needs a NAR dependency on the NAR where the service
interface is
In the case of Hive, the interface and implementation are both in
nifi-hive-processors which get packaged into nifi-hive-nar, so you
need:
In your processors pom.xml:
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-hive-processors</artifactId>
<version>replace with the version of NiFi you are using</version>
<scope>provided</scope>
</dependency>
In your NAR pom.xml:
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-hive-nar</artifactId>
<version>replace with the version of NiFi you are using</version>
<type>nar</type>
</dependency>
-Bryan
On Fri, Feb 10, 2017 at 2:25 AM, Nijandan D <ni...@gmail.com> wrote:
> Hi Bryan,
>
> I am new to Nifi, I'm tried to create custom processor with hive connection
> pooling service as per your concern. But
> same issue is occuring in custom processor. It shows only connection id. I
> have no idea about pom file dependency.
> Here I mentioned the steps what i have followed to create custom processor
> with hive connection below. If it's wrong,
> Please provide any link to get workflow for creating custom processor with
> hive database connection pooling service.
>
> - Added below line in custom processor java class
> public static final PropertyDescriptor HIVE_DBCP_SERVICE = new
> PropertyDescriptor.Builder()
> .name("Hive Database Connection Pooling Service")
> .description("The Hive Controller Service that is used to obtain
> connection(s) to the Hive database")
> .required(true)
> .identifiesControllerService(HiveDBCPService.class)
> .build();
>
> - After imported HiveDBCPService in same java class
> import org.apache.nifi.dbcp.hive.HiveDBCPService;
>
> - After searched dependency at maven repositories for HiveDBCPService and
> added
>
> - In nifi-custom-nar pom file, added following lines,
> <dependency>
> <groupId>org.apache.nifi</groupId>
> <artifactId>nifi-hive-processors</artifactId>
> <version>1.0.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.nifi</groupId>
> <artifactId>nifi-hadoop-libraries-nar</artifactId>
> <type>nar</type>
> </dependency>
>
> - In nifi-custom-processor pom file, added following lines,
> <dependency>
> <groupId>commons-dbcp</groupId>
> <artifactId>commons-dbcp</artifactId>
> <version>1.5-SNAPSHOT</version>
> <type>jar</type>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-core</artifactId>
> <version>1.2.1</version>
> <type>jar</type>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-common</artifactId>
> <version>3.0.0-SNAPSHOT</version>
> <type>jar</type>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-jdbc</artifactId>
> <version>1.3.0-SNAPSHOT</version>
> <type>jar</type>
> </dependency>
> <dependency>
> <groupId>org.apache.nifi</groupId>
> <artifactId>nifi-hadoop-utils</artifactId>
> <version>1.0.0</version>
> <type>jar</type>
> </dependency>
> <dependency>
> <groupId>org.apache.nifi</groupId>
> <artifactId>nifi-hive-processors</artifactId>
> <version>1.0.0</version>
> <type>jar</type>
> </dependency>
>
> - After taking build and replaced Nar file in lib folder, restarted Nifi
> source. Same error connection id only showing in
> connection dropdown box.
>
> Thanks in advance.
>
> Regards,
> Nijandan D
>
> On Thu, Feb 9, 2017 at 8:22 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> Hello,
>>
>> This usually means your custom NAR is not correctly linked to the NAR
>> where the service is.
>>
>> Take a look at these resources for examples:
>>
>>
>> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices
>>
>>
>> https://github.com/bbende/nifi-dependency-example/blob/master/nifi-example-processor-bundle/nifi-example-processors-nar/pom.xml#L40
>>
>> Thanks,
>>
>> Bryan
>>
>> On Wed, Feb 8, 2017 at 11:49 PM, cool nijandhan <ni...@gmail.com>
>> wrote:
>> > Hi
>> >
>> > I am developing custom processor with hive database connection pooling
>> > service. Added necessary dependency in pom file and added necessary
>> > folders
>> > in custom processor folder. I have created class file and able to
>> > generate
>> > nar file. Placed nar file in lib directory and restarted nifi server.
>> > Its
>> > look like everything working fine. But services are not showing in
>> > database
>> > connection dropdown box. Instead it showing connection id, but every
>> > time
>> > services are creating when click "create" in controller section. For
>> > other
>> > processors, it seems working fine. Please find the attached screenshots.
>> >
>> > Any help appreciated.
>> >
>> > Thanks
>
>
Re: Custom Processor with Hive database connection pooling service
not working
Posted by Nijandan D <ni...@gmail.com>.
Hi Bryan,
I am new to Nifi, I'm tried to create custom processor with hive connection
pooling service as per your concern. But
same issue is occuring in custom processor. It shows only connection id. I
have no idea about pom file dependency.
Here I mentioned the steps what i have followed to create custom processor
with hive connection below. If it's wrong,
Please provide any link to get workflow for creating custom processor with
hive database connection pooling service.
- Added below line in custom processor java class
* public static final PropertyDescriptor HIVE_DBCP_SERVICE = new
PropertyDescriptor.Builder()*
* .name("Hive Database Connection Pooling Service")*
* .description("The Hive Controller Service that is used to
obtain connection(s) to the Hive database")*
* .required(true)*
* .identifiesControllerService(HiveDBCPService.class)*
* .build();*
- After imported HiveDBCPService in same java class
* import org.apache.nifi.dbcp.hive.HiveDBCPService;*
*- *After searched dependency at maven repositories for HiveDBCPService and
added
- In nifi-custom-nar pom file, added following lines,
* <dependency>*
* <groupId>org.apache.nifi</groupId>*
* <artifactId>nifi-hive-processors</artifactId>*
* <version>1.0.0</version>*
* </dependency>*
* <dependency>*
* <groupId>org.apache.nifi</groupId>*
* <artifactId>nifi-hadoop-libraries-nar</artifactId>*
* <type>nar</type>*
* </dependency>*
- In nifi-custom-processor pom file, added following lines,
*<dependency>*
* <groupId>commons-dbcp</groupId>*
* <artifactId>commons-dbcp</artifactId>*
* <version>1.5-SNAPSHOT</version>*
* <type>jar</type>*
* </dependency>*
* <dependency>*
* <groupId>org.apache.hadoop</groupId>*
* <artifactId>hadoop-core</artifactId>*
* <version>1.2.1</version>*
* <type>jar</type>*
* </dependency>*
* <dependency>*
* <groupId>org.apache.hadoop</groupId>*
* <artifactId>hadoop-common</artifactId>*
* <version>3.0.0-SNAPSHOT</version>*
* <type>jar</type>*
* </dependency>*
* <dependency>*
* <groupId>org.apache.hive</groupId>*
* <artifactId>hive-jdbc</artifactId>*
* <version>1.3.0-SNAPSHOT</version>*
* <type>jar</type>*
* </dependency>*
* <dependency>*
* <groupId>org.apache.nifi</groupId>*
* <artifactId>nifi-hadoop-utils</artifactId>*
* <version>1.0.0</version>*
* <type>jar</type>*
* </dependency>*
* <dependency>*
* <groupId>org.apache.nifi</groupId>*
* <artifactId>nifi-hive-processors</artifactId>*
* <version>1.0.0</version>*
* <type>jar</type>*
* </dependency>*
*- *After taking build and replaced Nar file in lib folder, restarted Nifi
source. Same error connection id only showing in
connection dropdown box.
Thanks in advance.
Regards,
Nijandan D
On Thu, Feb 9, 2017 at 8:22 PM, Bryan Bende <bb...@gmail.com> wrote:
> Hello,
>
> This usually means your custom NAR is not correctly linked to the NAR
> where the service is.
>
> Take a look at these resources for examples:
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+
> Projects+for+Extensions#MavenProjectsforExtensions-
> LinkingProcessorsandControllerServices
>
> https://github.com/bbende/nifi-dependency-example/blob/
> master/nifi-example-processor-bundle/nifi-example-
> processors-nar/pom.xml#L40
>
> Thanks,
>
> Bryan
>
> On Wed, Feb 8, 2017 at 11:49 PM, cool nijandhan <ni...@gmail.com>
> wrote:
> > Hi
> >
> > I am developing custom processor with hive database connection pooling
> > service. Added necessary dependency in pom file and added necessary
> folders
> > in custom processor folder. I have created class file and able to
> generate
> > nar file. Placed nar file in lib directory and restarted nifi server. Its
> > look like everything working fine. But services are not showing in
> database
> > connection dropdown box. Instead it showing connection id, but every time
> > services are creating when click "create" in controller section. For
> other
> > processors, it seems working fine. Please find the attached screenshots.
> >
> > Any help appreciated.
> >
> > Thanks
>
Re: Custom Processor with Hive database connection pooling service
not working
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
This usually means your custom NAR is not correctly linked to the NAR
where the service is.
Take a look at these resources for examples:
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices
https://github.com/bbende/nifi-dependency-example/blob/master/nifi-example-processor-bundle/nifi-example-processors-nar/pom.xml#L40
Thanks,
Bryan
On Wed, Feb 8, 2017 at 11:49 PM, cool nijandhan <ni...@gmail.com> wrote:
> Hi
>
> I am developing custom processor with hive database connection pooling
> service. Added necessary dependency in pom file and added necessary folders
> in custom processor folder. I have created class file and able to generate
> nar file. Placed nar file in lib directory and restarted nifi server. Its
> look like everything working fine. But services are not showing in database
> connection dropdown box. Instead it showing connection id, but every time
> services are creating when click "create" in controller section. For other
> processors, it seems working fine. Please find the attached screenshots.
>
> Any help appreciated.
>
> Thanks