You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Ralph Goers <ra...@dslextreme.com> on 2022/01/13 15:26:08 UTC

Morphlines-solr-sink

While I am not having any trouble building the morphline-solr-sink component, it is dependent on the abandoned kite-sdk, which makes its life very limited. 

In addition, the kite-sdk has a dependency on parquet-avro which, according to https://issues.apache.org/jira/browse/FLUME-3403, has vulnerabilities in every available release.

Due to these factors I am going to remove the morphline-solr-sink module from Flume for the 1.10.0 release.

Ralph

Re: Morphlines-solr-sink

Posted by Ralph Goers <ra...@dslextreme.com>.
OK. I’ll put it back.

The real question is if the bug in parquet-avro can be fixed. The reporter indicated that they had tried to fix it several times.

Ralph

> On Jan 14, 2022, at 3:33 PM, Tristan Stevens <tr...@apache.org> wrote:
> 
> -1 from me.
> 
> First wee can’t do that in a patch release, but that’s semantics.
> 
> Both the Morphlines interceptor and the  Morphlines-Solr-Sink are components that are widely used amongst the community. I did some analysis last year that I’ll dig out and share, but they are two of the  most used components after HDFS sink, Kafka and JMS.
> 
> Whilst I agree it’s sucky that Cloudera aren’t supporting Kite anymore, I wonder whether we can find a way to bring Morphlines into here, or otherwise get upstream and fix the bits that need fixing.
> 
> Tristan
> 
> 
> From: Ralph Goers <ra...@dslextreme.com> <ma...@dslextreme.com>
> Reply: dev@flume.apache.org <ma...@flume.apache.org> <de...@flume.apache.org> <ma...@flume.apache.org>
> Date: 13 January 2022 at 15:26:12
> To: dev@flume.apache.org <de...@flume.apache.org> <ma...@flume.apache.org>
> Subject:  Morphlines-solr-sink 
> 
>> While I am not having any trouble building the morphline-solr-sink component, it is dependent on the abandoned kite-sdk, which makes its life very limited.  
>> 
>> In addition, the kite-sdk has a dependency on parquet-avro which, according to https://issues.apache.org/jira/browse/FLUME-3403, has vulnerabilities in every available release. 
>> 
>> Due to these factors I am going to remove the morphline-solr-sink module from Flume for the 1.10.0 release. 
>> 
>> Ralph


Re: Morphlines-solr-sink

Posted by Ralph Goers <ra...@dslextreme.com>.
No, thanks. I got past the issue with the Mophline stuff for this release. We will have to discuss what to do about it long term.

I have been fighting Avro/Netty for the last week upgrading those dependencies. I just have one more test to figure out and then that too will be done.

Then I have to look at the hadoop family dependencies and determine what kinds of upgrades they may need.

Waiting years to do releases really sucks.

Ralph

> On Jan 23, 2022, at 10:57 AM, Tristan Stevens <tr...@apache.org> wrote:
> 
> Hi Ralph,
> Here is the data.
> 
> MorphlineSolrSink and MorphlineInterceptor appear 13-14% of the time in this sample set.
> 
> org.apache.flume.channel.kafka.KafkaChannel                64%  
> hdfs                                                       38%  
> org.apache.flume.source.kafka.KafkaSource                  33%  
> memory                                                     29%  
> file                                                       28%  
> spooldir                                                   26%  
> null                                                       25%  
> org.apache.flume.sink.kafka.KafkaSink                      25%  
> Custom JMSSource 18%  
> jms                                                        17%  
> static                                                     14%  
> org.apache.flume.sink.solr.morphline.MorphlineInterceptor  14%  
> ElasticSearchSink   13%  
> org.apache.flume.sink.solr.morphline.MorphlineSolrSink     13%  
> host                                                       13%  
> timestamp                                                  13%  
> avro                                                       11%  
> hbase                                                      11%
> 
> Let me know if you’d like to drill down further.
> 
> Tristan  
> 
> From: Ralph Goers <ra...@dslextreme.com>
> Reply: dev@flume.apache.org <de...@flume.apache.org>
> Date: 15 January 2022 at 06:48:43
> To: dev@flume.apache.org <de...@flume.apache.org>
> Subject:  Re: Morphlines-solr-sink  
> 
> I would like to see the data on the usage. I’m not sure how you would know since Cloudera doesn’t seem to include Flume in its products any more from what I can tell.  
> 
> The kite-morphines project consists of 18 sub-modules plug 4 aggregation modules. That is a heck of a lot of stuff to try to drag in. I would prefer to fork the parts of kite we would need to a new flume-kite repo.  
> 
> It seems that the CVE the reporter mentioned does have a fix. It is available in parquet-avro 1.11.2 and 1.12.2. I was able to swap the new version for the old one even though the groupId has changed. That said, the kite-sdk dependency that includes it is marked as optional, so parquet-avro would be optional as well. So I have no idea if it is even used.  
> 
> In any case, the unit tests all pass with the updated dependency.  
> 
> Ralph  
> 
> 
> 
>> On Jan 14, 2022, at 3:33 PM, Tristan Stevens <tr...@apache.org> wrote:  
>> 
>> -1 from me.  
>> 
>> First wee can’t do that in a patch release, but that’s semantics.  
>> 
>> Both the Morphlines interceptor and the Morphlines-Solr-Sink are components that are widely used amongst the community. I did some analysis last year that I’ll dig out and share, but they are two of the most used components after HDFS sink, Kafka and JMS.  
>> 
>> Whilst I agree it’s sucky that Cloudera aren’t supporting Kite anymore, I wonder whether we can find a way to bring Morphlines into here, or otherwise get upstream and fix the bits that need fixing.  
>> 
>> Tristan  
>> 
>> 
>> From: Ralph Goers <ra...@dslextreme.com> <ma...@dslextreme.com>  
>> Reply: dev@flume.apache.org <ma...@flume.apache.org> <de...@flume.apache.org> <ma...@flume.apache.org>  
>> Date: 13 January 2022 at 15:26:12  
>> To: dev@flume.apache.org <de...@flume.apache.org> <ma...@flume.apache.org>  
>> Subject: Morphlines-solr-sink  
>> 
>>> While I am not having any trouble building the morphline-solr-sink component, it is dependent on the abandoned kite-sdk, which makes its life very limited.  
>>> 
>>> In addition, the kite-sdk has a dependency on parquet-avro which, according to https://issues.apache.org/jira/browse/FLUME-3403, has vulnerabilities in every available release.  
>>> 
>>> Due to these factors I am going to remove the morphline-solr-sink module from Flume for the 1.10.0 release.  
>>> 
>>> Ralph  
> 


Re: Morphlines-solr-sink

Posted by Tristan Stevens <tr...@apache.org>.
Hi Ralph,
Here is the data.

MorphlineSolrSink and MorphlineInterceptor appear 13-14% of the time in this sample set.

org.apache.flume.channel.kafka.KafkaChannel                64%  
hdfs                                                       38%  
org.apache.flume.source.kafka.KafkaSource                  33%  
memory                                                     29%  
file                                                       28%  
spooldir                                                   26%  
null                                                       25%  
org.apache.flume.sink.kafka.KafkaSink                      25%  
Custom JMSSource 18%  
jms                                                        17%  
static                                                     14%  
org.apache.flume.sink.solr.morphline.MorphlineInterceptor  14%  
ElasticSearchSink   13%  
org.apache.flume.sink.solr.morphline.MorphlineSolrSink     13%  
host                                                       13%  
timestamp                                                  13%  
avro                                                       11%  
hbase                                                      11%

Let me know if you’d like to drill down further.

Tristan  

From: Ralph Goers <ra...@dslextreme.com>
Reply: dev@flume.apache.org <de...@flume.apache.org>
Date: 15 January 2022 at 06:48:43
To: dev@flume.apache.org <de...@flume.apache.org>
Subject:  Re: Morphlines-solr-sink  

I would like to see the data on the usage. I’m not sure how you would know since Cloudera doesn’t seem to include Flume in its products any more from what I can tell.  

The kite-morphines project consists of 18 sub-modules plug 4 aggregation modules. That is a heck of a lot of stuff to try to drag in. I would prefer to fork the parts of kite we would need to a new flume-kite repo.  

It seems that the CVE the reporter mentioned does have a fix. It is available in parquet-avro 1.11.2 and 1.12.2. I was able to swap the new version for the old one even though the groupId has changed. That said, the kite-sdk dependency that includes it is marked as optional, so parquet-avro would be optional as well. So I have no idea if it is even used.  

In any case, the unit tests all pass with the updated dependency.  

Ralph  



> On Jan 14, 2022, at 3:33 PM, Tristan Stevens <tr...@apache.org> wrote:  
>  
> -1 from me.  
>  
> First wee can’t do that in a patch release, but that’s semantics.  
>  
> Both the Morphlines interceptor and the Morphlines-Solr-Sink are components that are widely used amongst the community. I did some analysis last year that I’ll dig out and share, but they are two of the most used components after HDFS sink, Kafka and JMS.  
>  
> Whilst I agree it’s sucky that Cloudera aren’t supporting Kite anymore, I wonder whether we can find a way to bring Morphlines into here, or otherwise get upstream and fix the bits that need fixing.  
>  
> Tristan  
>  
>  
> From: Ralph Goers <ra...@dslextreme.com> <ma...@dslextreme.com>  
> Reply: dev@flume.apache.org <ma...@flume.apache.org> <de...@flume.apache.org> <ma...@flume.apache.org>  
> Date: 13 January 2022 at 15:26:12  
> To: dev@flume.apache.org <de...@flume.apache.org> <ma...@flume.apache.org>  
> Subject: Morphlines-solr-sink  
>  
>> While I am not having any trouble building the morphline-solr-sink component, it is dependent on the abandoned kite-sdk, which makes its life very limited.  
>>  
>> In addition, the kite-sdk has a dependency on parquet-avro which, according to https://issues.apache.org/jira/browse/FLUME-3403, has vulnerabilities in every available release.  
>>  
>> Due to these factors I am going to remove the morphline-solr-sink module from Flume for the 1.10.0 release.  
>>  
>> Ralph  


Re: Morphlines-solr-sink

Posted by Hari Shreedharan <hs...@apache.org>.
+1 to removing it. Since the underlying project is no longer maintained, I
think we should be ok removing it.

On Sat, Jan 15, 2022 at 12:18 PM Ralph Goers <ra...@dslextreme.com>
wrote:

> I would like to see the data on the usage. I’m not sure how you would know
> since Cloudera doesn’t seem to include Flume in its products any more from
> what I can tell.
>
> The kite-morphines project consists of 18 sub-modules plug 4 aggregation
> modules. That is a heck of a lot of stuff to try to drag in. I would prefer
> to fork the parts of kite we would need to a new flume-kite repo.
>
> It seems that the CVE the reporter mentioned does have a fix. It is
> available in parquet-avro 1.11.2 and 1.12.2.  I was able to swap the new
> version for the old one even though the groupId has changed. That said, the
> kite-sdk dependency that includes it is marked as optional, so parquet-avro
> would be optional as well. So I have no idea if it is even used.
>
> In any case, the unit tests all pass with the updated dependency.
>
> Ralph
>
>
>
> > On Jan 14, 2022, at 3:33 PM, Tristan Stevens <tr...@apache.org> wrote:
> >
> > -1 from me.
> >
> > First wee can’t do that in a patch release, but that’s semantics.
> >
> > Both the Morphlines interceptor and the  Morphlines-Solr-Sink are
> components that are widely used amongst the community. I did some analysis
> last year that I’ll dig out and share, but they are two of the  most used
> components after HDFS sink, Kafka and JMS.
> >
> > Whilst I agree it’s sucky that Cloudera aren’t supporting Kite anymore,
> I wonder whether we can find a way to bring Morphlines into here, or
> otherwise get upstream and fix the bits that need fixing.
> >
> > Tristan
> >
> >
> > From: Ralph Goers <ra...@dslextreme.com> <mailto:
> ralph.goers@dslextreme.com>
> > Reply: dev@flume.apache.org <ma...@flume.apache.org> <
> dev@flume.apache.org> <ma...@flume.apache.org>
> > Date: 13 January 2022 at 15:26:12
> > To: dev@flume.apache.org <de...@flume.apache.org> <mailto:
> dev@flume.apache.org>
> > Subject:  Morphlines-solr-sink
> >
> >> While I am not having any trouble building the morphline-solr-sink
> component, it is dependent on the abandoned kite-sdk, which makes its life
> very limited.
> >>
> >> In addition, the kite-sdk has a dependency on parquet-avro which,
> according to https://issues.apache.org/jira/browse/FLUME-3403, has
> vulnerabilities in every available release.
> >>
> >> Due to these factors I am going to remove the morphline-solr-sink
> module from Flume for the 1.10.0 release.
> >>
> >> Ralph
>
>

Re: Morphlines-solr-sink

Posted by Ralph Goers <ra...@dslextreme.com>.
I would like to see the data on the usage. I’m not sure how you would know since Cloudera doesn’t seem to include Flume in its products any more from what I can tell. 

The kite-morphines project consists of 18 sub-modules plug 4 aggregation modules. That is a heck of a lot of stuff to try to drag in. I would prefer to fork the parts of kite we would need to a new flume-kite repo.

It seems that the CVE the reporter mentioned does have a fix. It is available in parquet-avro 1.11.2 and 1.12.2.  I was able to swap the new version for the old one even though the groupId has changed. That said, the kite-sdk dependency that includes it is marked as optional, so parquet-avro would be optional as well. So I have no idea if it is even used.

In any case, the unit tests all pass with the updated dependency.

Ralph



> On Jan 14, 2022, at 3:33 PM, Tristan Stevens <tr...@apache.org> wrote:
> 
> -1 from me.
> 
> First wee can’t do that in a patch release, but that’s semantics.
> 
> Both the Morphlines interceptor and the  Morphlines-Solr-Sink are components that are widely used amongst the community. I did some analysis last year that I’ll dig out and share, but they are two of the  most used components after HDFS sink, Kafka and JMS.
> 
> Whilst I agree it’s sucky that Cloudera aren’t supporting Kite anymore, I wonder whether we can find a way to bring Morphlines into here, or otherwise get upstream and fix the bits that need fixing.
> 
> Tristan
> 
> 
> From: Ralph Goers <ra...@dslextreme.com> <ma...@dslextreme.com>
> Reply: dev@flume.apache.org <ma...@flume.apache.org> <de...@flume.apache.org> <ma...@flume.apache.org>
> Date: 13 January 2022 at 15:26:12
> To: dev@flume.apache.org <de...@flume.apache.org> <ma...@flume.apache.org>
> Subject:  Morphlines-solr-sink 
> 
>> While I am not having any trouble building the morphline-solr-sink component, it is dependent on the abandoned kite-sdk, which makes its life very limited.  
>> 
>> In addition, the kite-sdk has a dependency on parquet-avro which, according to https://issues.apache.org/jira/browse/FLUME-3403, has vulnerabilities in every available release. 
>> 
>> Due to these factors I am going to remove the morphline-solr-sink module from Flume for the 1.10.0 release. 
>> 
>> Ralph


Re: Morphlines-solr-sink

Posted by Tristan Stevens <tr...@apache.org>.
-1 from me.

First wee can’t do that in a patch release, but that’s semantics.

Both the Morphlines interceptor and the  Morphlines-Solr-Sink are components that are widely used amongst the community. I did some analysis last year that I’ll dig out and share, but they are two of the  most used components after HDFS sink, Kafka and JMS.

Whilst I agree it’s sucky that Cloudera aren’t supporting Kite anymore, I wonder whether we can find a way to bring Morphlines into here, or otherwise get upstream and fix the bits that need fixing.

Tristan


From: Ralph Goers <ra...@dslextreme.com>
Reply: dev@flume.apache.org <de...@flume.apache.org>
Date: 13 January 2022 at 15:26:12
To: dev@flume.apache.org <de...@flume.apache.org>
Subject:  Morphlines-solr-sink  

While I am not having any trouble building the morphline-solr-sink component, it is dependent on the abandoned kite-sdk, which makes its life very limited.  

In addition, the kite-sdk has a dependency on parquet-avro which, according to https://issues.apache.org/jira/browse/FLUME-3403, has vulnerabilities in every available release.  

Due to these factors I am going to remove the morphline-solr-sink module from Flume for the 1.10.0 release.  

Ralph