You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Jungtaek Lim <ka...@gmail.com> on 2016/07/22 00:37:47 UTC
[DISCUSSION] Policy of resolving dependencies for non storm-core modules
Hi devs,
AFAIK, we had been struggled to resolve dependency issues for storm-core.
As we all know, the strategy we have been using is shade & relocating.
Now State and Storm SQL requires that some of external modules need to be
included to extlib, which is the classpath workers refer.
http://issues.apache.org/jira/browse/STORM-1881
https://issues.apache.org/jira/browse/STORM-1435
There're two issues here:
- We don't make uber jar for external modules so users need to find and
copy dependencies jars to extlib manually.
- External modules also use Guava and Jackson and so on which are origin of
version conflict issues.
So we should apply the shade & relocating strategy for every external
modules (at least storm-redis, storm-kafka, storm-sql-core,
storm-sql-kafka), or introduce the way to add the dependency without adding
them to extlib. (like --packages and --jar for Spark)
Please express your opinions about this.
Thanks,
Jungtaek Lim (HeartSaVioR)
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core modules
Posted by Jungtaek Lim <ka...@gmail.com>.
Thanks Taylor!
Let me open a new thread for proposal and continue discussion from there.
Jungtaek Lim (HeartSaVioR)
2016년 8월 2일 (화) 오전 12:10, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> Jungtaek, I’ve given you permissions to the Storm wiki so you can post the
> document there.
>
> -Taylor
>
> > On Aug 1, 2016, at 12:46 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> >
> > Not yet. It would be good to continue discussion on this thread for now.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2016년 8월 1일 (월) 오후 1:38, Satish Duggana <sa...@gmail.com>님이 작성:
> >
> >> Hi,
> >> Did we find a way to comment on the proposal? How can we proceed with
> this
> >> discussion?
> >>
> >> Thanks,
> >> Satish.
> >>
> >>
> >> On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <ka...@gmail.com>
> wrote:
> >>
> >>> I've made a design document for adding jars and maven artifacts at
> >>> submission.
> >>> Since we don't have any formats for this, I just borrowed KIP format.
> >>> That's not we would like to adopt that format, it's just me.
> >>>
> >>>
> >>>
> >>
> https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing
> >>>
> >>> Btw, I guess ASF policy requires that discussion history should be
> logged
> >>> to Apache Infra.
> >>> (So I only grant view permission who opens the doc via link.)
> >>>
> >>> So what's recommended way to do? Would we want to file an issue for
> JIRA
> >>> with attaching design doc to PDF and discuss there?
> >>>
> >>> - Jungtaek Lim
> >>>
> >>> 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >>>
> >>>> Thought about it once more, and found that former approach still needs
> >> to
> >>>> add 'provided' scope libraries to extlib directory..
> >>>>
> >>>> Along with thinking former approach, I've experimented a bit of latter
> >>>> approach by creating POC project. Since I don't know about copy and
> use
> >>> ASF
> >>>> project codes for personal use, I'd just share main class to see if
> >> this
> >>>> POC and my theory makes sense for us.
> >>>> https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6
> >>>> 'zeppelin' package has some classes for transitive dependency resolver
> >>>> (copied from Zeppelin), and 'spark' package has some classes for
> >>>> classloader (yes also copied from Spark).
> >>>>
> >>>> Please share your experiences if you have knowledges regarding this
> >> area.
> >>>>
> >>>> - Jungtaek Lim
> >>>>
> >>>> 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <evans@yahoo-inc.com.invalid
> >님이
> >>> 작성:
> >>>>
> >>>>> You can do that with a combination of the distributed cache and
> >> setting
> >>>>> the classpath, just like with hadoop. It is not as clean as it could
> >>> be,
> >>>>> but it does work. - Bobby
> >>>>>
> >>>>> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <
> >>> arunm@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>
> >>>>> Shade and relocate the external modules sounds ok as a short term
> >>>>> solution.
> >>>>>
> >>>>> For the long term we should consider something like the second option
> >> to
> >>>>> add external modules without shipping uber jars.
> >>>>>
> >>>>> Thanks,
> >>>>> Arun
> >>>>>
> >>>>> On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >>>>>
> >>>>>> Hi devs,
> >>>>>>
> >>>>>> AFAIK, we had been struggled to resolve dependency issues for
> >>> storm-core.
> >>>>>> As we all know, the strategy we have been using is shade &
> >> relocating.
> >>>>>>
> >>>>>> Now State and Storm SQL requires that some of external modules need
> >> to
> >>> be
> >>>>>> included to extlib, which is the classpath workers refer.
> >>>>>>
> >>>>>> http://issues.apache.org/jira/browse/STORM-1881
> >>>>>> https://issues.apache.org/jira/browse/STORM-1435
> >>>>>>
> >>>>>> There're two issues here:
> >>>>>> - We don't make uber jar for external modules so users need to find
> >> and
> >>>>>> copy dependencies jars to extlib manually.
> >>>>>> - External modules also use Guava and Jackson and so on which are
> >>> origin
> >>>>> of
> >>>>>> version conflict issues.
> >>>>>>
> >>>>>> So we should apply the shade & relocating strategy for every
> external
> >>>>>> modules (at least storm-redis, storm-kafka, storm-sql-core,
> >>>>>> storm-sql-kafka), or introduce the way to add the dependency without
> >>>>> adding
> >>>>>> them to extlib. (like --packages and --jar for Spark)
> >>>>>>
> >>>>>> Please express your opinions about this.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core modules
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Jungtaek, I’ve given you permissions to the Storm wiki so you can post the document there.
-Taylor
> On Aug 1, 2016, at 12:46 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Not yet. It would be good to continue discussion on this thread for now.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 8월 1일 (월) 오후 1:38, Satish Duggana <sa...@gmail.com>님이 작성:
>
>> Hi,
>> Did we find a way to comment on the proposal? How can we proceed with this
>> discussion?
>>
>> Thanks,
>> Satish.
>>
>>
>> On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>>
>>> I've made a design document for adding jars and maven artifacts at
>>> submission.
>>> Since we don't have any formats for this, I just borrowed KIP format.
>>> That's not we would like to adopt that format, it's just me.
>>>
>>>
>>>
>> https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing
>>>
>>> Btw, I guess ASF policy requires that discussion history should be logged
>>> to Apache Infra.
>>> (So I only grant view permission who opens the doc via link.)
>>>
>>> So what's recommended way to do? Would we want to file an issue for JIRA
>>> with attaching design doc to PDF and discuss there?
>>>
>>> - Jungtaek Lim
>>>
>>> 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>
>>>> Thought about it once more, and found that former approach still needs
>> to
>>>> add 'provided' scope libraries to extlib directory..
>>>>
>>>> Along with thinking former approach, I've experimented a bit of latter
>>>> approach by creating POC project. Since I don't know about copy and use
>>> ASF
>>>> project codes for personal use, I'd just share main class to see if
>> this
>>>> POC and my theory makes sense for us.
>>>> https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6
>>>> 'zeppelin' package has some classes for transitive dependency resolver
>>>> (copied from Zeppelin), and 'spark' package has some classes for
>>>> classloader (yes also copied from Spark).
>>>>
>>>> Please share your experiences if you have knowledges regarding this
>> area.
>>>>
>>>> - Jungtaek Lim
>>>>
>>>> 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
>>> 작성:
>>>>
>>>>> You can do that with a combination of the distributed cache and
>> setting
>>>>> the classpath, just like with hadoop. It is not as clean as it could
>>> be,
>>>>> but it does work. - Bobby
>>>>>
>>>>> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <
>>> arunm@apache.org>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Shade and relocate the external modules sounds ok as a short term
>>>>> solution.
>>>>>
>>>>> For the long term we should consider something like the second option
>> to
>>>>> add external modules without shipping uber jars.
>>>>>
>>>>> Thanks,
>>>>> Arun
>>>>>
>>>>> On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>>>>
>>>>>> Hi devs,
>>>>>>
>>>>>> AFAIK, we had been struggled to resolve dependency issues for
>>> storm-core.
>>>>>> As we all know, the strategy we have been using is shade &
>> relocating.
>>>>>>
>>>>>> Now State and Storm SQL requires that some of external modules need
>> to
>>> be
>>>>>> included to extlib, which is the classpath workers refer.
>>>>>>
>>>>>> http://issues.apache.org/jira/browse/STORM-1881
>>>>>> https://issues.apache.org/jira/browse/STORM-1435
>>>>>>
>>>>>> There're two issues here:
>>>>>> - We don't make uber jar for external modules so users need to find
>> and
>>>>>> copy dependencies jars to extlib manually.
>>>>>> - External modules also use Guava and Jackson and so on which are
>>> origin
>>>>> of
>>>>>> version conflict issues.
>>>>>>
>>>>>> So we should apply the shade & relocating strategy for every external
>>>>>> modules (at least storm-redis, storm-kafka, storm-sql-core,
>>>>>> storm-sql-kafka), or introduce the way to add the dependency without
>>>>> adding
>>>>>> them to extlib. (like --packages and --jar for Spark)
>>>>>>
>>>>>> Please express your opinions about this.
>>>>>>
>>>>>> Thanks,
>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core modules
Posted by Jungtaek Lim <ka...@gmail.com>.
Not yet. It would be good to continue discussion on this thread for now.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 8월 1일 (월) 오후 1:38, Satish Duggana <sa...@gmail.com>님이 작성:
> Hi,
> Did we find a way to comment on the proposal? How can we proceed with this
> discussion?
>
> Thanks,
> Satish.
>
>
> On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> > I've made a design document for adding jars and maven artifacts at
> > submission.
> > Since we don't have any formats for this, I just borrowed KIP format.
> > That's not we would like to adopt that format, it's just me.
> >
> >
> >
> https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing
> >
> > Btw, I guess ASF policy requires that discussion history should be logged
> > to Apache Infra.
> > (So I only grant view permission who opens the doc via link.)
> >
> > So what's recommended way to do? Would we want to file an issue for JIRA
> > with attaching design doc to PDF and discuss there?
> >
> > - Jungtaek Lim
> >
> > 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >
> > > Thought about it once more, and found that former approach still needs
> to
> > > add 'provided' scope libraries to extlib directory..
> > >
> > > Along with thinking former approach, I've experimented a bit of latter
> > > approach by creating POC project. Since I don't know about copy and use
> > ASF
> > > project codes for personal use, I'd just share main class to see if
> this
> > > POC and my theory makes sense for us.
> > > https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6
> > > 'zeppelin' package has some classes for transitive dependency resolver
> > > (copied from Zeppelin), and 'spark' package has some classes for
> > > classloader (yes also copied from Spark).
> > >
> > > Please share your experiences if you have knowledges regarding this
> area.
> > >
> > > - Jungtaek Lim
> > >
> > > 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
> > 작성:
> > >
> > >> You can do that with a combination of the distributed cache and
> setting
> > >> the classpath, just like with hadoop. It is not as clean as it could
> > be,
> > >> but it does work. - Bobby
> > >>
> > >> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <
> > arunm@apache.org>
> > >> wrote:
> > >>
> > >>
> > >> Shade and relocate the external modules sounds ok as a short term
> > >> solution.
> > >>
> > >> For the long term we should consider something like the second option
> to
> > >> add external modules without shipping uber jars.
> > >>
> > >> Thanks,
> > >> Arun
> > >>
> > >> On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> > >>
> > >> >Hi devs,
> > >> >
> > >> >AFAIK, we had been struggled to resolve dependency issues for
> > storm-core.
> > >> >As we all know, the strategy we have been using is shade &
> relocating.
> > >> >
> > >> >Now State and Storm SQL requires that some of external modules need
> to
> > be
> > >> >included to extlib, which is the classpath workers refer.
> > >> >
> > >> >http://issues.apache.org/jira/browse/STORM-1881
> > >> >https://issues.apache.org/jira/browse/STORM-1435
> > >> >
> > >> >There're two issues here:
> > >> >- We don't make uber jar for external modules so users need to find
> and
> > >> >copy dependencies jars to extlib manually.
> > >> >- External modules also use Guava and Jackson and so on which are
> > origin
> > >> of
> > >> >version conflict issues.
> > >> >
> > >> >So we should apply the shade & relocating strategy for every external
> > >> >modules (at least storm-redis, storm-kafka, storm-sql-core,
> > >> >storm-sql-kafka), or introduce the way to add the dependency without
> > >> adding
> > >> >them to extlib. (like --packages and --jar for Spark)
> > >> >
> > >> >Please express your opinions about this.
> > >> >
> > >> >Thanks,
> > >> >Jungtaek Lim (HeartSaVioR)
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> >
>
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core modules
Posted by Satish Duggana <sa...@gmail.com>.
Hi,
Did we find a way to comment on the proposal? How can we proceed with this
discussion?
Thanks,
Satish.
On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> I've made a design document for adding jars and maven artifacts at
> submission.
> Since we don't have any formats for this, I just borrowed KIP format.
> That's not we would like to adopt that format, it's just me.
>
>
> https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing
>
> Btw, I guess ASF policy requires that discussion history should be logged
> to Apache Infra.
> (So I only grant view permission who opens the doc via link.)
>
> So what's recommended way to do? Would we want to file an issue for JIRA
> with attaching design doc to PDF and discuss there?
>
> - Jungtaek Lim
>
> 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
> > Thought about it once more, and found that former approach still needs to
> > add 'provided' scope libraries to extlib directory..
> >
> > Along with thinking former approach, I've experimented a bit of latter
> > approach by creating POC project. Since I don't know about copy and use
> ASF
> > project codes for personal use, I'd just share main class to see if this
> > POC and my theory makes sense for us.
> > https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6
> > 'zeppelin' package has some classes for transitive dependency resolver
> > (copied from Zeppelin), and 'spark' package has some classes for
> > classloader (yes also copied from Spark).
> >
> > Please share your experiences if you have knowledges regarding this area.
> >
> > - Jungtaek Lim
> >
> > 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
> 작성:
> >
> >> You can do that with a combination of the distributed cache and setting
> >> the classpath, just like with hadoop. It is not as clean as it could
> be,
> >> but it does work. - Bobby
> >>
> >> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <
> arunm@apache.org>
> >> wrote:
> >>
> >>
> >> Shade and relocate the external modules sounds ok as a short term
> >> solution.
> >>
> >> For the long term we should consider something like the second option to
> >> add external modules without shipping uber jars.
> >>
> >> Thanks,
> >> Arun
> >>
> >> On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >>
> >> >Hi devs,
> >> >
> >> >AFAIK, we had been struggled to resolve dependency issues for
> storm-core.
> >> >As we all know, the strategy we have been using is shade & relocating.
> >> >
> >> >Now State and Storm SQL requires that some of external modules need to
> be
> >> >included to extlib, which is the classpath workers refer.
> >> >
> >> >http://issues.apache.org/jira/browse/STORM-1881
> >> >https://issues.apache.org/jira/browse/STORM-1435
> >> >
> >> >There're two issues here:
> >> >- We don't make uber jar for external modules so users need to find and
> >> >copy dependencies jars to extlib manually.
> >> >- External modules also use Guava and Jackson and so on which are
> origin
> >> of
> >> >version conflict issues.
> >> >
> >> >So we should apply the shade & relocating strategy for every external
> >> >modules (at least storm-redis, storm-kafka, storm-sql-core,
> >> >storm-sql-kafka), or introduce the way to add the dependency without
> >> adding
> >> >them to extlib. (like --packages and --jar for Spark)
> >> >
> >> >Please express your opinions about this.
> >> >
> >> >Thanks,
> >> >Jungtaek Lim (HeartSaVioR)
> >>
> >>
> >>
> >>
> >>
> >
> >
>
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core modules
Posted by Jungtaek Lim <ka...@gmail.com>.
I've made a design document for adding jars and maven artifacts at
submission.
Since we don't have any formats for this, I just borrowed KIP format.
That's not we would like to adopt that format, it's just me.
https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing
Btw, I guess ASF policy requires that discussion history should be logged
to Apache Infra.
(So I only grant view permission who opens the doc via link.)
So what's recommended way to do? Would we want to file an issue for JIRA
with attaching design doc to PDF and discuss there?
- Jungtaek Lim
2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <ka...@gmail.com>님이 작성:
> Thought about it once more, and found that former approach still needs to
> add 'provided' scope libraries to extlib directory..
>
> Along with thinking former approach, I've experimented a bit of latter
> approach by creating POC project. Since I don't know about copy and use ASF
> project codes for personal use, I'd just share main class to see if this
> POC and my theory makes sense for us.
> https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6
> 'zeppelin' package has some classes for transitive dependency resolver
> (copied from Zeppelin), and 'spark' package has some classes for
> classloader (yes also copied from Spark).
>
> Please share your experiences if you have knowledges regarding this area.
>
> - Jungtaek Lim
>
> 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:
>
>> You can do that with a combination of the distributed cache and setting
>> the classpath, just like with hadoop. It is not as clean as it could be,
>> but it does work. - Bobby
>>
>> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <ar...@apache.org>
>> wrote:
>>
>>
>> Shade and relocate the external modules sounds ok as a short term
>> solution.
>>
>> For the long term we should consider something like the second option to
>> add external modules without shipping uber jars.
>>
>> Thanks,
>> Arun
>>
>> On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>
>> >Hi devs,
>> >
>> >AFAIK, we had been struggled to resolve dependency issues for storm-core.
>> >As we all know, the strategy we have been using is shade & relocating.
>> >
>> >Now State and Storm SQL requires that some of external modules need to be
>> >included to extlib, which is the classpath workers refer.
>> >
>> >http://issues.apache.org/jira/browse/STORM-1881
>> >https://issues.apache.org/jira/browse/STORM-1435
>> >
>> >There're two issues here:
>> >- We don't make uber jar for external modules so users need to find and
>> >copy dependencies jars to extlib manually.
>> >- External modules also use Guava and Jackson and so on which are origin
>> of
>> >version conflict issues.
>> >
>> >So we should apply the shade & relocating strategy for every external
>> >modules (at least storm-redis, storm-kafka, storm-sql-core,
>> >storm-sql-kafka), or introduce the way to add the dependency without
>> adding
>> >them to extlib. (like --packages and --jar for Spark)
>> >
>> >Please express your opinions about this.
>> >
>> >Thanks,
>> >Jungtaek Lim (HeartSaVioR)
>>
>>
>>
>>
>>
>
>
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core modules
Posted by Jungtaek Lim <ka...@gmail.com>.
Thought about it once more, and found that former approach still needs to
add 'provided' scope libraries to extlib directory..
Along with thinking former approach, I've experimented a bit of latter
approach by creating POC project. Since I don't know about copy and use ASF
project codes for personal use, I'd just share main class to see if this
POC and my theory makes sense for us.
https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6
'zeppelin' package has some classes for transitive dependency resolver
(copied from Zeppelin), and 'spark' package has some classes for
classloader (yes also copied from Spark).
Please share your experiences if you have knowledges regarding this area.
- Jungtaek Lim
2016년 7월 22일 (금) 오후 10:59, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:
> You can do that with a combination of the distributed cache and setting
> the classpath, just like with hadoop. It is not as clean as it could be,
> but it does work. - Bobby
>
> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <ar...@apache.org>
> wrote:
>
>
> Shade and relocate the external modules sounds ok as a short term
> solution.
>
> For the long term we should consider something like the second option to
> add external modules without shipping uber jars.
>
> Thanks,
> Arun
>
> On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>
> >Hi devs,
> >
> >AFAIK, we had been struggled to resolve dependency issues for storm-core.
> >As we all know, the strategy we have been using is shade & relocating.
> >
> >Now State and Storm SQL requires that some of external modules need to be
> >included to extlib, which is the classpath workers refer.
> >
> >http://issues.apache.org/jira/browse/STORM-1881
> >https://issues.apache.org/jira/browse/STORM-1435
> >
> >There're two issues here:
> >- We don't make uber jar for external modules so users need to find and
> >copy dependencies jars to extlib manually.
> >- External modules also use Guava and Jackson and so on which are origin
> of
> >version conflict issues.
> >
> >So we should apply the shade & relocating strategy for every external
> >modules (at least storm-redis, storm-kafka, storm-sql-core,
> >storm-sql-kafka), or introduce the way to add the dependency without
> adding
> >them to extlib. (like --packages and --jar for Spark)
> >
> >Please express your opinions about this.
> >
> >Thanks,
> >Jungtaek Lim (HeartSaVioR)
>
>
>
>
>
Re: [DISCUSSION] Policy of resolving dependencies for non
storm-core modules
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
You can do that with a combination of the distributed cache and setting the classpath, just like with hadoop. It is not as clean as it could be, but it does work. - Bobby
On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <ar...@apache.org> wrote:
Shade and relocate the external modules sounds ok as a short term solution.
For the long term we should consider something like the second option to add external modules without shipping uber jars.
Thanks,
Arun
On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>Hi devs,
>
>AFAIK, we had been struggled to resolve dependency issues for storm-core.
>As we all know, the strategy we have been using is shade & relocating.
>
>Now State and Storm SQL requires that some of external modules need to be
>included to extlib, which is the classpath workers refer.
>
>http://issues.apache.org/jira/browse/STORM-1881
>https://issues.apache.org/jira/browse/STORM-1435
>
>There're two issues here:
>- We don't make uber jar for external modules so users need to find and
>copy dependencies jars to extlib manually.
>- External modules also use Guava and Jackson and so on which are origin of
>version conflict issues.
>
>So we should apply the shade & relocating strategy for every external
>modules (at least storm-redis, storm-kafka, storm-sql-core,
>storm-sql-kafka), or introduce the way to add the dependency without adding
>them to extlib. (like --packages and --jar for Spark)
>
>Please express your opinions about this.
>
>Thanks,
>Jungtaek Lim (HeartSaVioR)
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core
modules
Posted by Arun Mahadevan <ar...@apache.org>.
Shade and relocate the external modules sounds ok as a short term solution.
For the long term we should consider something like the second option to add external modules without shipping uber jars.
Thanks,
Arun
On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>Hi devs,
>
>AFAIK, we had been struggled to resolve dependency issues for storm-core.
>As we all know, the strategy we have been using is shade & relocating.
>
>Now State and Storm SQL requires that some of external modules need to be
>included to extlib, which is the classpath workers refer.
>
>http://issues.apache.org/jira/browse/STORM-1881
>https://issues.apache.org/jira/browse/STORM-1435
>
>There're two issues here:
>- We don't make uber jar for external modules so users need to find and
>copy dependencies jars to extlib manually.
>- External modules also use Guava and Jackson and so on which are origin of
>version conflict issues.
>
>So we should apply the shade & relocating strategy for every external
>modules (at least storm-redis, storm-kafka, storm-sql-core,
>storm-sql-kafka), or introduce the way to add the dependency without adding
>them to extlib. (like --packages and --jar for Spark)
>
>Please express your opinions about this.
>
>Thanks,
>Jungtaek Lim (HeartSaVioR)