You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Peter Wicks (pwicks)" <pw...@micron.com> on 2016/11/23 18:02:16 UTC
Extra Pair of Eyes
I'm working on a ticket to add HIVE support to QueryDatabaseTable (NIFI-3093). I'm doing this by adding a HIVE DatabaseAdapter and everytime I find something that doesn't work in HIVE I add a "getXXXSupported" property to DatabaseAdapter.
Initially I had my HiveDatabaseAdapter in Standard Processors with the others and had no issues seeing it in the UI and testing my code, but then I decided it made sense to refactor so I moved the base DatabaseAdapter interface to DBCP API so it could be centrally referenced without including Standard Processors as a reference, and moved my new HiveDatabaseAdapter to the HIVE Processors NAR to show up on the list of adapters.
Now Generic and Oracle still show up on the list but my HIVE one is not showing up. I've pushed everything up to a branch, would appreciate an extra pair of eyes:
https://github.com/patricker/nifi/tree/NIFI-3093
Thanks!
Peter
RE: Extra Pair of Eyes
Posted by "Peter Wicks (pwicks)" <pw...@micron.com>.
Mark,
I have done this for ControllerServices in the past, but it didn't click that this applied for any cross-NAR item like DatabaseAdapter.
Thanks, I'll give it a shot.
-----Original Message-----
From: Mark Payne [mailto:markap14@hotmail.com]
Sent: Wednesday, November 23, 2016 11:08 AM
To: dev@nifi.apache.org
Subject: Re: Extra Pair of Eyes
Hey Peter,
It looks like the nifi-hive-nar does not have a dependency on the nifi-standard-services-api-nar.
In order for a component to reference a Controller Service, we need to ensure that both the Controller Service implementation and the component referencing it share the same common definition for the Controller Service interface. To do that, we have to ensure that the interface is loaded from the same ClassLoader. So we use NAR parents to do that.
So you'd need to add the following dependency to nifi-hive-nar:
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-services-api-nar</artifactId>
<type>nar</type>
</dependency>
This will will cause the parent of the Hive NAR to be the standard-services-api NAR, so that it can properly link the interface & implementation together.
Thanks
-Mark
> On Nov 23, 2016, at 1:02 PM, Peter Wicks (pwicks) <pw...@micron.com> wrote:
>
> I'm working on a ticket to add HIVE support to QueryDatabaseTable (NIFI-3093). I'm doing this by adding a HIVE DatabaseAdapter and everytime I find something that doesn't work in HIVE I add a "getXXXSupported" property to DatabaseAdapter.
>
> Initially I had my HiveDatabaseAdapter in Standard Processors with the others and had no issues seeing it in the UI and testing my code, but then I decided it made sense to refactor so I moved the base DatabaseAdapter interface to DBCP API so it could be centrally referenced without including Standard Processors as a reference, and moved my new HiveDatabaseAdapter to the HIVE Processors NAR to show up on the list of adapters.
>
> Now Generic and Oracle still show up on the list but my HIVE one is not showing up. I've pushed everything up to a branch, would appreciate an extra pair of eyes:
>
> https://github.com/patricker/nifi/tree/NIFI-3093
>
> Thanks!
> Peter
RE: Extra Pair of Eyes
Posted by "Peter Wicks (pwicks)" <pw...@micron.com>.
Mark,
I'm not implementing DBCPService. HiveDatabaseAdapter implements DatabaseAdapter and DatabaseAdapter implements nothing, it just lives in the same NAR as the DBCPService controller service, which is why your fix kind of made sense to me.
DatabaseAdapter is an existing interface in NiFI 1.0 used to help customize how SQL is generated (in 1.0 just for Oracle), but it's not a controller service.
Thanks,
Peter
-----Original Message-----
From: Mark Payne [mailto:markap14@hotmail.com]
Sent: Wednesday, November 23, 2016 1:25 PM
To: dev@nifi.apache.org
Subject: Re: Extra Pair of Eyes
Things look okay to me on that branch. The thing that is a bit odd though is that you've defined a HiveDBCPService, which extends DBCPService. And then your controller service implements this instead of just implementing DBCPService directly.
I've never tried this... you may need to have your service implement DBCPService directly instead of having that intermediate interface there. The HiveDBCPService doesn't really buy you anything, I don't think, anyway, since the idea is to use this from QueryDatabaseTable, which only knows about DBCPService.
> On Nov 23, 2016, at 3:15 PM, Peter Wicks (pwicks) <pw...@micron.com> wrote:
>
> Mark, I added the dependency but it appears to have had no effect after a rebuild and redeploy.
> I checked the change into the same branch.
>
> Thoughts?
>
>
> -----Original Message-----
> From: Peter Wicks (pwicks)
> Sent: Wednesday, November 23, 2016 11:27 AM
> To: dev@nifi.apache.org
> Subject: RE: Extra Pair of Eyes
>
> Mark,
>
> I have done this for ControllerServices in the past, but it didn't click that this applied for any cross-NAR item like DatabaseAdapter.
>
> Thanks, I'll give it a shot.
>
>
> -----Original Message-----
> From: Mark Payne [mailto:markap14@hotmail.com]
> Sent: Wednesday, November 23, 2016 11:08 AM
> To: dev@nifi.apache.org
> Subject: Re: Extra Pair of Eyes
>
> Hey Peter,
>
> It looks like the nifi-hive-nar does not have a dependency on the nifi-standard-services-api-nar.
> In order for a component to reference a Controller Service, we need to ensure that both the Controller Service implementation and the component referencing it share the same common definition for the Controller Service interface. To do that, we have to ensure that the interface is loaded from the same ClassLoader. So we use NAR parents to do that.
>
> So you'd need to add the following dependency to nifi-hive-nar:
>
> <dependency>
> <groupId>org.apache.nifi</groupId>
> <artifactId>nifi-standard-services-api-nar</artifactId>
> <type>nar</type>
> </dependency>
>
> This will will cause the parent of the Hive NAR to be the standard-services-api NAR, so that it can properly link the interface & implementation together.
>
> Thanks
> -Mark
>
>> On Nov 23, 2016, at 1:02 PM, Peter Wicks (pwicks) <pw...@micron.com> wrote:
>>
>> I'm working on a ticket to add HIVE support to QueryDatabaseTable (NIFI-3093). I'm doing this by adding a HIVE DatabaseAdapter and everytime I find something that doesn't work in HIVE I add a "getXXXSupported" property to DatabaseAdapter.
>>
>> Initially I had my HiveDatabaseAdapter in Standard Processors with the others and had no issues seeing it in the UI and testing my code, but then I decided it made sense to refactor so I moved the base DatabaseAdapter interface to DBCP API so it could be centrally referenced without including Standard Processors as a reference, and moved my new HiveDatabaseAdapter to the HIVE Processors NAR to show up on the list of adapters.
>>
>> Now Generic and Oracle still show up on the list but my HIVE one is not showing up. I've pushed everything up to a branch, would appreciate an extra pair of eyes:
>>
>> https://github.com/patricker/nifi/tree/NIFI-3093
>>
>> Thanks!
>> Peter
>
Re: Extra Pair of Eyes
Posted by Mark Payne <ma...@hotmail.com>.
Things look okay to me on that branch. The thing that is a bit odd though is that you've defined a HiveDBCPService, which
extends DBCPService. And then your controller service implements this instead of just implementing DBCPService directly.
I've never tried this... you may need to have your service implement DBCPService directly instead of having that intermediate
interface there. The HiveDBCPService doesn't really buy you anything, I don't think, anyway, since the idea is to use this from
QueryDatabaseTable, which only knows about DBCPService.
> On Nov 23, 2016, at 3:15 PM, Peter Wicks (pwicks) <pw...@micron.com> wrote:
>
> Mark, I added the dependency but it appears to have had no effect after a rebuild and redeploy.
> I checked the change into the same branch.
>
> Thoughts?
>
>
> -----Original Message-----
> From: Peter Wicks (pwicks)
> Sent: Wednesday, November 23, 2016 11:27 AM
> To: dev@nifi.apache.org
> Subject: RE: Extra Pair of Eyes
>
> Mark,
>
> I have done this for ControllerServices in the past, but it didn't click that this applied for any cross-NAR item like DatabaseAdapter.
>
> Thanks, I'll give it a shot.
>
>
> -----Original Message-----
> From: Mark Payne [mailto:markap14@hotmail.com]
> Sent: Wednesday, November 23, 2016 11:08 AM
> To: dev@nifi.apache.org
> Subject: Re: Extra Pair of Eyes
>
> Hey Peter,
>
> It looks like the nifi-hive-nar does not have a dependency on the nifi-standard-services-api-nar.
> In order for a component to reference a Controller Service, we need to ensure that both the Controller Service implementation and the component referencing it share the same common definition for the Controller Service interface. To do that, we have to ensure that the interface is loaded from the same ClassLoader. So we use NAR parents to do that.
>
> So you'd need to add the following dependency to nifi-hive-nar:
>
> <dependency>
> <groupId>org.apache.nifi</groupId>
> <artifactId>nifi-standard-services-api-nar</artifactId>
> <type>nar</type>
> </dependency>
>
> This will will cause the parent of the Hive NAR to be the standard-services-api NAR, so that it can properly link the interface & implementation together.
>
> Thanks
> -Mark
>
>> On Nov 23, 2016, at 1:02 PM, Peter Wicks (pwicks) <pw...@micron.com> wrote:
>>
>> I'm working on a ticket to add HIVE support to QueryDatabaseTable (NIFI-3093). I'm doing this by adding a HIVE DatabaseAdapter and everytime I find something that doesn't work in HIVE I add a "getXXXSupported" property to DatabaseAdapter.
>>
>> Initially I had my HiveDatabaseAdapter in Standard Processors with the others and had no issues seeing it in the UI and testing my code, but then I decided it made sense to refactor so I moved the base DatabaseAdapter interface to DBCP API so it could be centrally referenced without including Standard Processors as a reference, and moved my new HiveDatabaseAdapter to the HIVE Processors NAR to show up on the list of adapters.
>>
>> Now Generic and Oracle still show up on the list but my HIVE one is not showing up. I've pushed everything up to a branch, would appreciate an extra pair of eyes:
>>
>> https://github.com/patricker/nifi/tree/NIFI-3093
>>
>> Thanks!
>> Peter
>
RE: Extra Pair of Eyes
Posted by "Peter Wicks (pwicks)" <pw...@micron.com>.
Mark, I added the dependency but it appears to have had no effect after a rebuild and redeploy.
I checked the change into the same branch.
Thoughts?
-----Original Message-----
From: Peter Wicks (pwicks)
Sent: Wednesday, November 23, 2016 11:27 AM
To: dev@nifi.apache.org
Subject: RE: Extra Pair of Eyes
Mark,
I have done this for ControllerServices in the past, but it didn't click that this applied for any cross-NAR item like DatabaseAdapter.
Thanks, I'll give it a shot.
-----Original Message-----
From: Mark Payne [mailto:markap14@hotmail.com]
Sent: Wednesday, November 23, 2016 11:08 AM
To: dev@nifi.apache.org
Subject: Re: Extra Pair of Eyes
Hey Peter,
It looks like the nifi-hive-nar does not have a dependency on the nifi-standard-services-api-nar.
In order for a component to reference a Controller Service, we need to ensure that both the Controller Service implementation and the component referencing it share the same common definition for the Controller Service interface. To do that, we have to ensure that the interface is loaded from the same ClassLoader. So we use NAR parents to do that.
So you'd need to add the following dependency to nifi-hive-nar:
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-services-api-nar</artifactId>
<type>nar</type>
</dependency>
This will will cause the parent of the Hive NAR to be the standard-services-api NAR, so that it can properly link the interface & implementation together.
Thanks
-Mark
> On Nov 23, 2016, at 1:02 PM, Peter Wicks (pwicks) <pw...@micron.com> wrote:
>
> I'm working on a ticket to add HIVE support to QueryDatabaseTable (NIFI-3093). I'm doing this by adding a HIVE DatabaseAdapter and everytime I find something that doesn't work in HIVE I add a "getXXXSupported" property to DatabaseAdapter.
>
> Initially I had my HiveDatabaseAdapter in Standard Processors with the others and had no issues seeing it in the UI and testing my code, but then I decided it made sense to refactor so I moved the base DatabaseAdapter interface to DBCP API so it could be centrally referenced without including Standard Processors as a reference, and moved my new HiveDatabaseAdapter to the HIVE Processors NAR to show up on the list of adapters.
>
> Now Generic and Oracle still show up on the list but my HIVE one is not showing up. I've pushed everything up to a branch, would appreciate an extra pair of eyes:
>
> https://github.com/patricker/nifi/tree/NIFI-3093
>
> Thanks!
> Peter
Re: Extra Pair of Eyes
Posted by Mark Payne <ma...@hotmail.com>.
Hey Peter,
It looks like the nifi-hive-nar does not have a dependency on the nifi-standard-services-api-nar.
In order for a component to reference a Controller Service, we need to ensure that both the Controller Service
implementation and the component referencing it share the same common definition for the Controller Service
interface. To do that, we have to ensure that the interface is loaded from the same ClassLoader. So we use NAR
parents to do that.
So you'd need to add the following dependency to nifi-hive-nar:
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-services-api-nar</artifactId>
<type>nar</type>
</dependency>
This will will cause the parent of the Hive NAR to be the standard-services-api NAR, so that it can properly
link the interface & implementation together.
Thanks
-Mark
> On Nov 23, 2016, at 1:02 PM, Peter Wicks (pwicks) <pw...@micron.com> wrote:
>
> I'm working on a ticket to add HIVE support to QueryDatabaseTable (NIFI-3093). I'm doing this by adding a HIVE DatabaseAdapter and everytime I find something that doesn't work in HIVE I add a "getXXXSupported" property to DatabaseAdapter.
>
> Initially I had my HiveDatabaseAdapter in Standard Processors with the others and had no issues seeing it in the UI and testing my code, but then I decided it made sense to refactor so I moved the base DatabaseAdapter interface to DBCP API so it could be centrally referenced without including Standard Processors as a reference, and moved my new HiveDatabaseAdapter to the HIVE Processors NAR to show up on the list of adapters.
>
> Now Generic and Oracle still show up on the list but my HIVE one is not showing up. I've pushed everything up to a branch, would appreciate an extra pair of eyes:
>
> https://github.com/patricker/nifi/tree/NIFI-3093
>
> Thanks!
> Peter