You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Sivaprasanna <si...@gmail.com> on 2018/05/03 15:51:59 UTC
NIFI-5133: Guidance & help with tackling dependency version issues
Hi
I've started the initial works on implementing Google Cloud Pub/Sub
processors. The associated Jira ID is NIFI-5133
<https://issues.apache.org/jira/browse/NIFI-5133>. This will go to the
existing GCP bundle which currently has only the storage processors. Upon
some inspection, I noticed the following:
- As of now, the bundle uses google-cloud
<http://mvnrepository.com/artifact/com.google.cloud/google-cloud> as its
dependency which is like uber/fat jar that contains most of the Google
Cloud's client library SDKs including storage, bigquery, pubsub, etc. The
main point is it is using a very older version (0.8.0)
- I thought of using google-cloud-bom
<http://mvnrepository.com/artifact/com.google.cloud/google-cloud-bom> in
the bundle's POM
<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/pom.xml>
and then use the required artifacts in the processor's POM
<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
The benefit is, it will help us reduce the overall size of the NAR.
When I tried to do #2, I realized this is not a simple version change but a
change that brings backward compatibility issues. Ex: Some APIs used in the
older version i.e. 0.8.0 have now been entirely scrapped and moved to
different library. We can do either two things:
1. User the Pub/Sub APIs from the older version but the problem is it's
very old and the problem of upgrading would soon catchup with us.
2. Or we can continue to use the older version of google-cloud-storage
only for the storage processors and introduce the #2 mentioned above but I
don't think then the new processors can't properly extend the existing
AbstractGCPProcessor
<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.
A quick glance on the processor code and the POM would help you understand
my concern.
I'm stuck up here so any help & guidance in this regard is very much
appreciated. :)
Thanks,
Sivaprasanna
Re: NIFI-5133: Guidance & help with tackling dependency version issues
Posted by Sivaprasanna <si...@gmail.com>.
I have created a Jira - NIFI-5156
<https://issues.apache.org/jira/browse/NIFI-5156> and made the necessary
changes to the current nifi-gcp-bundle. I have tested the components - GCS
Processors & GCP Credentials Service. They all work as expected. Anyone
want to give it a look and review the changes? PR #2680
<https://github.com/apache/nifi/pull/2680>
-
Sivaprasanna
On Thu, May 3, 2018 at 9:48 PM, Sivaprasanna <si...@gmail.com>
wrote:
> I'm interested although it might consume some time since I don't know how
> big it is going to be. And I suppose it is better to capture it in a
> separate Jira. The summary could be upgrade and refactor GCP processor
> codebase or something like that. We could then make NIFI-5133 a dependent
> of the new Jira. Thoughts?
>
> Meanwhile, anyone who has any other recommendations, feel free to share. :)
>
> -
> Sivaprasanna
>
> On Thu, May 3, 2018 at 9:43 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Sivaprasanna
>>
>> Ok makes sense. Are you in a position of interest/expertise/time to
>> make those changes as well?
>>
>> Thanks
>>
>> On Thu, May 3, 2018 at 12:11 PM, Sivaprasanna <si...@gmail.com>
>> wrote:
>> > Hi. As I had mentioned, upgrading to the latest version of the library
>> is
>> > not as simple as I thought. Google Cloud team introduced many breaking
>> > changes. Many of the APIs (classes & methods) have been
>> > scrapped/replaced/modified/refactored/renamed.
>> >
>> > In short, a simple change of version may demand changes on the
>> processor's
>> > code, especially on the AbstractProcessors (AbstractGCS, AbstractGCP)
>> which
>> > may pose backward compatibility issues, I'm afraid.
>> >
>> > Thanks,
>> > Sivaprasanna
>> >
>> > On Thu, May 3, 2018 at 9:26 PM, Joe Witt <jo...@gmail.com> wrote:
>> >
>> >> Sivaprasanna
>> >>
>> >> I might not completely follow but is there a 3rd option to upgrade to
>> >> a more recent library and solve the use of the proper jars
>> >> problem(smaller nar)?
>> >>
>> >> Thanks
>> >>
>> >> On Thu, May 3, 2018 at 11:51 AM, Sivaprasanna <
>> sivaprasanna246@gmail.com>
>> >> wrote:
>> >> > Hi
>> >> >
>> >> > I've started the initial works on implementing Google Cloud Pub/Sub
>> >> > processors. The associated Jira ID is NIFI-5133
>> >> > <https://issues.apache.org/jira/browse/NIFI-5133>. This will go to
>> the
>> >> > existing GCP bundle which currently has only the storage processors.
>> Upon
>> >> > some inspection, I noticed the following:
>> >> >
>> >> > - As of now, the bundle uses google-cloud
>> >> > <http://mvnrepository.com/artifact/com.google.cloud/google-cloud>
>> as
>> >> its
>> >> > dependency which is like uber/fat jar that contains most of the
>> Google
>> >> > Cloud's client library SDKs including storage, bigquery, pubsub,
>> etc.
>> >> The
>> >> > main point is it is using a very older version (0.8.0)
>> >> > - I thought of using google-cloud-bom
>> >> > <http://mvnrepository.com/artifact/com.google.cloud/google-
>> cloud-bom>
>> >> in
>> >> > the bundle's POM
>> >> > <https://github.com/apache/nifi/blob/master/nifi-nar-
>> >> bundles/nifi-gcp-bundle/pom.xml>
>> >> > and then use the required artifacts in the processor's POM
>> >> > <https://github.com/apache/nifi/blob/master/nifi-nar-
>> >> bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
>> >> > The benefit is, it will help us reduce the overall size of the
>> NAR.
>> >> >
>> >> > When I tried to do #2, I realized this is not a simple version change
>> >> but a
>> >> > change that brings backward compatibility issues. Ex: Some APIs used
>> in
>> >> the
>> >> > older version i.e. 0.8.0 have now been entirely scrapped and moved to
>> >> > different library. We can do either two things:
>> >> >
>> >> > 1. User the Pub/Sub APIs from the older version but the problem is
>> >> it's
>> >> > very old and the problem of upgrading would soon catchup with us.
>> >> > 2. Or we can continue to use the older version of
>> google-cloud-storage
>> >> > only for the storage processors and introduce the #2 mentioned
>> above
>> >> but I
>> >> > don't think then the new processors can't properly extend the
>> existing
>> >> > AbstractGCPProcessor
>> >> > <https://github.com/apache/nifi/blob/master/nifi-nar-
>> >> bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/
>> >> org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.
>> >> >
>> >> >
>> >> > A quick glance on the processor code and the POM would help you
>> >> understand
>> >> > my concern.
>> >> >
>> >> > I'm stuck up here so any help & guidance in this regard is very much
>> >> > appreciated. :)
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Sivaprasanna
>> >>
>>
>
>
Re: NIFI-5133: Guidance & help with tackling dependency version issues
Posted by Sivaprasanna <si...@gmail.com>.
I'm interested although it might consume some time since I don't know how
big it is going to be. And I suppose it is better to capture it in a
separate Jira. The summary could be upgrade and refactor GCP processor
codebase or something like that. We could then make NIFI-5133 a dependent
of the new Jira. Thoughts?
Meanwhile, anyone who has any other recommendations, feel free to share. :)
-
Sivaprasanna
On Thu, May 3, 2018 at 9:43 PM, Joe Witt <jo...@gmail.com> wrote:
> Sivaprasanna
>
> Ok makes sense. Are you in a position of interest/expertise/time to
> make those changes as well?
>
> Thanks
>
> On Thu, May 3, 2018 at 12:11 PM, Sivaprasanna <si...@gmail.com>
> wrote:
> > Hi. As I had mentioned, upgrading to the latest version of the library is
> > not as simple as I thought. Google Cloud team introduced many breaking
> > changes. Many of the APIs (classes & methods) have been
> > scrapped/replaced/modified/refactored/renamed.
> >
> > In short, a simple change of version may demand changes on the
> processor's
> > code, especially on the AbstractProcessors (AbstractGCS, AbstractGCP)
> which
> > may pose backward compatibility issues, I'm afraid.
> >
> > Thanks,
> > Sivaprasanna
> >
> > On Thu, May 3, 2018 at 9:26 PM, Joe Witt <jo...@gmail.com> wrote:
> >
> >> Sivaprasanna
> >>
> >> I might not completely follow but is there a 3rd option to upgrade to
> >> a more recent library and solve the use of the proper jars
> >> problem(smaller nar)?
> >>
> >> Thanks
> >>
> >> On Thu, May 3, 2018 at 11:51 AM, Sivaprasanna <
> sivaprasanna246@gmail.com>
> >> wrote:
> >> > Hi
> >> >
> >> > I've started the initial works on implementing Google Cloud Pub/Sub
> >> > processors. The associated Jira ID is NIFI-5133
> >> > <https://issues.apache.org/jira/browse/NIFI-5133>. This will go to
> the
> >> > existing GCP bundle which currently has only the storage processors.
> Upon
> >> > some inspection, I noticed the following:
> >> >
> >> > - As of now, the bundle uses google-cloud
> >> > <http://mvnrepository.com/artifact/com.google.cloud/google-cloud>
> as
> >> its
> >> > dependency which is like uber/fat jar that contains most of the
> Google
> >> > Cloud's client library SDKs including storage, bigquery, pubsub,
> etc.
> >> The
> >> > main point is it is using a very older version (0.8.0)
> >> > - I thought of using google-cloud-bom
> >> > <http://mvnrepository.com/artifact/com.google.cloud/
> google-cloud-bom>
> >> in
> >> > the bundle's POM
> >> > <https://github.com/apache/nifi/blob/master/nifi-nar-
> >> bundles/nifi-gcp-bundle/pom.xml>
> >> > and then use the required artifacts in the processor's POM
> >> > <https://github.com/apache/nifi/blob/master/nifi-nar-
> >> bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
> >> > The benefit is, it will help us reduce the overall size of the NAR.
> >> >
> >> > When I tried to do #2, I realized this is not a simple version change
> >> but a
> >> > change that brings backward compatibility issues. Ex: Some APIs used
> in
> >> the
> >> > older version i.e. 0.8.0 have now been entirely scrapped and moved to
> >> > different library. We can do either two things:
> >> >
> >> > 1. User the Pub/Sub APIs from the older version but the problem is
> >> it's
> >> > very old and the problem of upgrading would soon catchup with us.
> >> > 2. Or we can continue to use the older version of
> google-cloud-storage
> >> > only for the storage processors and introduce the #2 mentioned
> above
> >> but I
> >> > don't think then the new processors can't properly extend the
> existing
> >> > AbstractGCPProcessor
> >> > <https://github.com/apache/nifi/blob/master/nifi-nar-
> >> bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/
> >> org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.
> >> >
> >> >
> >> > A quick glance on the processor code and the POM would help you
> >> understand
> >> > my concern.
> >> >
> >> > I'm stuck up here so any help & guidance in this regard is very much
> >> > appreciated. :)
> >> >
> >> > Thanks,
> >> >
> >> > Sivaprasanna
> >>
>
Re: NIFI-5133: Guidance & help with tackling dependency version issues
Posted by Joe Witt <jo...@gmail.com>.
Sivaprasanna
Ok makes sense. Are you in a position of interest/expertise/time to
make those changes as well?
Thanks
On Thu, May 3, 2018 at 12:11 PM, Sivaprasanna <si...@gmail.com> wrote:
> Hi. As I had mentioned, upgrading to the latest version of the library is
> not as simple as I thought. Google Cloud team introduced many breaking
> changes. Many of the APIs (classes & methods) have been
> scrapped/replaced/modified/refactored/renamed.
>
> In short, a simple change of version may demand changes on the processor's
> code, especially on the AbstractProcessors (AbstractGCS, AbstractGCP) which
> may pose backward compatibility issues, I'm afraid.
>
> Thanks,
> Sivaprasanna
>
> On Thu, May 3, 2018 at 9:26 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Sivaprasanna
>>
>> I might not completely follow but is there a 3rd option to upgrade to
>> a more recent library and solve the use of the proper jars
>> problem(smaller nar)?
>>
>> Thanks
>>
>> On Thu, May 3, 2018 at 11:51 AM, Sivaprasanna <si...@gmail.com>
>> wrote:
>> > Hi
>> >
>> > I've started the initial works on implementing Google Cloud Pub/Sub
>> > processors. The associated Jira ID is NIFI-5133
>> > <https://issues.apache.org/jira/browse/NIFI-5133>. This will go to the
>> > existing GCP bundle which currently has only the storage processors. Upon
>> > some inspection, I noticed the following:
>> >
>> > - As of now, the bundle uses google-cloud
>> > <http://mvnrepository.com/artifact/com.google.cloud/google-cloud> as
>> its
>> > dependency which is like uber/fat jar that contains most of the Google
>> > Cloud's client library SDKs including storage, bigquery, pubsub, etc.
>> The
>> > main point is it is using a very older version (0.8.0)
>> > - I thought of using google-cloud-bom
>> > <http://mvnrepository.com/artifact/com.google.cloud/google-cloud-bom>
>> in
>> > the bundle's POM
>> > <https://github.com/apache/nifi/blob/master/nifi-nar-
>> bundles/nifi-gcp-bundle/pom.xml>
>> > and then use the required artifacts in the processor's POM
>> > <https://github.com/apache/nifi/blob/master/nifi-nar-
>> bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
>> > The benefit is, it will help us reduce the overall size of the NAR.
>> >
>> > When I tried to do #2, I realized this is not a simple version change
>> but a
>> > change that brings backward compatibility issues. Ex: Some APIs used in
>> the
>> > older version i.e. 0.8.0 have now been entirely scrapped and moved to
>> > different library. We can do either two things:
>> >
>> > 1. User the Pub/Sub APIs from the older version but the problem is
>> it's
>> > very old and the problem of upgrading would soon catchup with us.
>> > 2. Or we can continue to use the older version of google-cloud-storage
>> > only for the storage processors and introduce the #2 mentioned above
>> but I
>> > don't think then the new processors can't properly extend the existing
>> > AbstractGCPProcessor
>> > <https://github.com/apache/nifi/blob/master/nifi-nar-
>> bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/
>> org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.
>> >
>> >
>> > A quick glance on the processor code and the POM would help you
>> understand
>> > my concern.
>> >
>> > I'm stuck up here so any help & guidance in this regard is very much
>> > appreciated. :)
>> >
>> > Thanks,
>> >
>> > Sivaprasanna
>>
Re: NIFI-5133: Guidance & help with tackling dependency version issues
Posted by Sivaprasanna <si...@gmail.com>.
Hi. As I had mentioned, upgrading to the latest version of the library is
not as simple as I thought. Google Cloud team introduced many breaking
changes. Many of the APIs (classes & methods) have been
scrapped/replaced/modified/refactored/renamed.
In short, a simple change of version may demand changes on the processor's
code, especially on the AbstractProcessors (AbstractGCS, AbstractGCP) which
may pose backward compatibility issues, I'm afraid.
Thanks,
Sivaprasanna
On Thu, May 3, 2018 at 9:26 PM, Joe Witt <jo...@gmail.com> wrote:
> Sivaprasanna
>
> I might not completely follow but is there a 3rd option to upgrade to
> a more recent library and solve the use of the proper jars
> problem(smaller nar)?
>
> Thanks
>
> On Thu, May 3, 2018 at 11:51 AM, Sivaprasanna <si...@gmail.com>
> wrote:
> > Hi
> >
> > I've started the initial works on implementing Google Cloud Pub/Sub
> > processors. The associated Jira ID is NIFI-5133
> > <https://issues.apache.org/jira/browse/NIFI-5133>. This will go to the
> > existing GCP bundle which currently has only the storage processors. Upon
> > some inspection, I noticed the following:
> >
> > - As of now, the bundle uses google-cloud
> > <http://mvnrepository.com/artifact/com.google.cloud/google-cloud> as
> its
> > dependency which is like uber/fat jar that contains most of the Google
> > Cloud's client library SDKs including storage, bigquery, pubsub, etc.
> The
> > main point is it is using a very older version (0.8.0)
> > - I thought of using google-cloud-bom
> > <http://mvnrepository.com/artifact/com.google.cloud/google-cloud-bom>
> in
> > the bundle's POM
> > <https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-gcp-bundle/pom.xml>
> > and then use the required artifacts in the processor's POM
> > <https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
> > The benefit is, it will help us reduce the overall size of the NAR.
> >
> > When I tried to do #2, I realized this is not a simple version change
> but a
> > change that brings backward compatibility issues. Ex: Some APIs used in
> the
> > older version i.e. 0.8.0 have now been entirely scrapped and moved to
> > different library. We can do either two things:
> >
> > 1. User the Pub/Sub APIs from the older version but the problem is
> it's
> > very old and the problem of upgrading would soon catchup with us.
> > 2. Or we can continue to use the older version of google-cloud-storage
> > only for the storage processors and introduce the #2 mentioned above
> but I
> > don't think then the new processors can't properly extend the existing
> > AbstractGCPProcessor
> > <https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/
> org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.
> >
> >
> > A quick glance on the processor code and the POM would help you
> understand
> > my concern.
> >
> > I'm stuck up here so any help & guidance in this regard is very much
> > appreciated. :)
> >
> > Thanks,
> >
> > Sivaprasanna
>
Re: NIFI-5133: Guidance & help with tackling dependency version issues
Posted by Joe Witt <jo...@gmail.com>.
Sivaprasanna
I might not completely follow but is there a 3rd option to upgrade to
a more recent library and solve the use of the proper jars
problem(smaller nar)?
Thanks
On Thu, May 3, 2018 at 11:51 AM, Sivaprasanna <si...@gmail.com> wrote:
> Hi
>
> I've started the initial works on implementing Google Cloud Pub/Sub
> processors. The associated Jira ID is NIFI-5133
> <https://issues.apache.org/jira/browse/NIFI-5133>. This will go to the
> existing GCP bundle which currently has only the storage processors. Upon
> some inspection, I noticed the following:
>
> - As of now, the bundle uses google-cloud
> <http://mvnrepository.com/artifact/com.google.cloud/google-cloud> as its
> dependency which is like uber/fat jar that contains most of the Google
> Cloud's client library SDKs including storage, bigquery, pubsub, etc. The
> main point is it is using a very older version (0.8.0)
> - I thought of using google-cloud-bom
> <http://mvnrepository.com/artifact/com.google.cloud/google-cloud-bom> in
> the bundle's POM
> <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/pom.xml>
> and then use the required artifacts in the processor's POM
> <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
> The benefit is, it will help us reduce the overall size of the NAR.
>
> When I tried to do #2, I realized this is not a simple version change but a
> change that brings backward compatibility issues. Ex: Some APIs used in the
> older version i.e. 0.8.0 have now been entirely scrapped and moved to
> different library. We can do either two things:
>
> 1. User the Pub/Sub APIs from the older version but the problem is it's
> very old and the problem of upgrading would soon catchup with us.
> 2. Or we can continue to use the older version of google-cloud-storage
> only for the storage processors and introduce the #2 mentioned above but I
> don't think then the new processors can't properly extend the existing
> AbstractGCPProcessor
> <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.
>
>
> A quick glance on the processor code and the POM would help you understand
> my concern.
>
> I'm stuck up here so any help & guidance in this regard is very much
> appreciated. :)
>
> Thanks,
>
> Sivaprasanna