You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Satish Duggana <sa...@gmail.com> on 2016/08/01 04:38:25 UTC
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core modules
Hi,
Did we find a way to comment on the proposal? How can we proceed with this
discussion?
Thanks,
Satish.
On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> I've made a design document for adding jars and maven artifacts at
> submission.
> Since we don't have any formats for this, I just borrowed KIP format.
> That's not we would like to adopt that format, it's just me.
>
>
> https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing
>
> Btw, I guess ASF policy requires that discussion history should be logged
> to Apache Infra.
> (So I only grant view permission who opens the doc via link.)
>
> So what's recommended way to do? Would we want to file an issue for JIRA
> with attaching design doc to PDF and discuss there?
>
> - Jungtaek Lim
>
> 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
> > Thought about it once more, and found that former approach still needs to
> > add 'provided' scope libraries to extlib directory..
> >
> > Along with thinking former approach, I've experimented a bit of latter
> > approach by creating POC project. Since I don't know about copy and use
> ASF
> > project codes for personal use, I'd just share main class to see if this
> > POC and my theory makes sense for us.
> > https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6
> > 'zeppelin' package has some classes for transitive dependency resolver
> > (copied from Zeppelin), and 'spark' package has some classes for
> > classloader (yes also copied from Spark).
> >
> > Please share your experiences if you have knowledges regarding this area.
> >
> > - Jungtaek Lim
> >
> > 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
> 작성:
> >
> >> You can do that with a combination of the distributed cache and setting
> >> the classpath, just like with hadoop. It is not as clean as it could
> be,
> >> but it does work. - Bobby
> >>
> >> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <
> arunm@apache.org>
> >> wrote:
> >>
> >>
> >> Shade and relocate the external modules sounds ok as a short term
> >> solution.
> >>
> >> For the long term we should consider something like the second option to
> >> add external modules without shipping uber jars.
> >>
> >> Thanks,
> >> Arun
> >>
> >> On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >>
> >> >Hi devs,
> >> >
> >> >AFAIK, we had been struggled to resolve dependency issues for
> storm-core.
> >> >As we all know, the strategy we have been using is shade & relocating.
> >> >
> >> >Now State and Storm SQL requires that some of external modules need to
> be
> >> >included to extlib, which is the classpath workers refer.
> >> >
> >> >http://issues.apache.org/jira/browse/STORM-1881
> >> >https://issues.apache.org/jira/browse/STORM-1435
> >> >
> >> >There're two issues here:
> >> >- We don't make uber jar for external modules so users need to find and
> >> >copy dependencies jars to extlib manually.
> >> >- External modules also use Guava and Jackson and so on which are
> origin
> >> of
> >> >version conflict issues.
> >> >
> >> >So we should apply the shade & relocating strategy for every external
> >> >modules (at least storm-redis, storm-kafka, storm-sql-core,
> >> >storm-sql-kafka), or introduce the way to add the dependency without
> >> adding
> >> >them to extlib. (like --packages and --jar for Spark)
> >> >
> >> >Please express your opinions about this.
> >> >
> >> >Thanks,
> >> >Jungtaek Lim (HeartSaVioR)
> >>
> >>
> >>
> >>
> >>
> >
> >
>
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core modules
Posted by Jungtaek Lim <ka...@gmail.com>.
Thanks Taylor!
Let me open a new thread for proposal and continue discussion from there.
Jungtaek Lim (HeartSaVioR)
2016년 8월 2일 (화) 오전 12:10, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> Jungtaek, I’ve given you permissions to the Storm wiki so you can post the
> document there.
>
> -Taylor
>
> > On Aug 1, 2016, at 12:46 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> >
> > Not yet. It would be good to continue discussion on this thread for now.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2016년 8월 1일 (월) 오후 1:38, Satish Duggana <sa...@gmail.com>님이 작성:
> >
> >> Hi,
> >> Did we find a way to comment on the proposal? How can we proceed with
> this
> >> discussion?
> >>
> >> Thanks,
> >> Satish.
> >>
> >>
> >> On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <ka...@gmail.com>
> wrote:
> >>
> >>> I've made a design document for adding jars and maven artifacts at
> >>> submission.
> >>> Since we don't have any formats for this, I just borrowed KIP format.
> >>> That's not we would like to adopt that format, it's just me.
> >>>
> >>>
> >>>
> >>
> https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing
> >>>
> >>> Btw, I guess ASF policy requires that discussion history should be
> logged
> >>> to Apache Infra.
> >>> (So I only grant view permission who opens the doc via link.)
> >>>
> >>> So what's recommended way to do? Would we want to file an issue for
> JIRA
> >>> with attaching design doc to PDF and discuss there?
> >>>
> >>> - Jungtaek Lim
> >>>
> >>> 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >>>
> >>>> Thought about it once more, and found that former approach still needs
> >> to
> >>>> add 'provided' scope libraries to extlib directory..
> >>>>
> >>>> Along with thinking former approach, I've experimented a bit of latter
> >>>> approach by creating POC project. Since I don't know about copy and
> use
> >>> ASF
> >>>> project codes for personal use, I'd just share main class to see if
> >> this
> >>>> POC and my theory makes sense for us.
> >>>> https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6
> >>>> 'zeppelin' package has some classes for transitive dependency resolver
> >>>> (copied from Zeppelin), and 'spark' package has some classes for
> >>>> classloader (yes also copied from Spark).
> >>>>
> >>>> Please share your experiences if you have knowledges regarding this
> >> area.
> >>>>
> >>>> - Jungtaek Lim
> >>>>
> >>>> 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <evans@yahoo-inc.com.invalid
> >님이
> >>> 작성:
> >>>>
> >>>>> You can do that with a combination of the distributed cache and
> >> setting
> >>>>> the classpath, just like with hadoop. It is not as clean as it could
> >>> be,
> >>>>> but it does work. - Bobby
> >>>>>
> >>>>> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <
> >>> arunm@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>
> >>>>> Shade and relocate the external modules sounds ok as a short term
> >>>>> solution.
> >>>>>
> >>>>> For the long term we should consider something like the second option
> >> to
> >>>>> add external modules without shipping uber jars.
> >>>>>
> >>>>> Thanks,
> >>>>> Arun
> >>>>>
> >>>>> On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >>>>>
> >>>>>> Hi devs,
> >>>>>>
> >>>>>> AFAIK, we had been struggled to resolve dependency issues for
> >>> storm-core.
> >>>>>> As we all know, the strategy we have been using is shade &
> >> relocating.
> >>>>>>
> >>>>>> Now State and Storm SQL requires that some of external modules need
> >> to
> >>> be
> >>>>>> included to extlib, which is the classpath workers refer.
> >>>>>>
> >>>>>> http://issues.apache.org/jira/browse/STORM-1881
> >>>>>> https://issues.apache.org/jira/browse/STORM-1435
> >>>>>>
> >>>>>> There're two issues here:
> >>>>>> - We don't make uber jar for external modules so users need to find
> >> and
> >>>>>> copy dependencies jars to extlib manually.
> >>>>>> - External modules also use Guava and Jackson and so on which are
> >>> origin
> >>>>> of
> >>>>>> version conflict issues.
> >>>>>>
> >>>>>> So we should apply the shade & relocating strategy for every
> external
> >>>>>> modules (at least storm-redis, storm-kafka, storm-sql-core,
> >>>>>> storm-sql-kafka), or introduce the way to add the dependency without
> >>>>> adding
> >>>>>> them to extlib. (like --packages and --jar for Spark)
> >>>>>>
> >>>>>> Please express your opinions about this.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core modules
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Jungtaek, I’ve given you permissions to the Storm wiki so you can post the document there.
-Taylor
> On Aug 1, 2016, at 12:46 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Not yet. It would be good to continue discussion on this thread for now.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 8월 1일 (월) 오후 1:38, Satish Duggana <sa...@gmail.com>님이 작성:
>
>> Hi,
>> Did we find a way to comment on the proposal? How can we proceed with this
>> discussion?
>>
>> Thanks,
>> Satish.
>>
>>
>> On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>>
>>> I've made a design document for adding jars and maven artifacts at
>>> submission.
>>> Since we don't have any formats for this, I just borrowed KIP format.
>>> That's not we would like to adopt that format, it's just me.
>>>
>>>
>>>
>> https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing
>>>
>>> Btw, I guess ASF policy requires that discussion history should be logged
>>> to Apache Infra.
>>> (So I only grant view permission who opens the doc via link.)
>>>
>>> So what's recommended way to do? Would we want to file an issue for JIRA
>>> with attaching design doc to PDF and discuss there?
>>>
>>> - Jungtaek Lim
>>>
>>> 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>
>>>> Thought about it once more, and found that former approach still needs
>> to
>>>> add 'provided' scope libraries to extlib directory..
>>>>
>>>> Along with thinking former approach, I've experimented a bit of latter
>>>> approach by creating POC project. Since I don't know about copy and use
>>> ASF
>>>> project codes for personal use, I'd just share main class to see if
>> this
>>>> POC and my theory makes sense for us.
>>>> https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6
>>>> 'zeppelin' package has some classes for transitive dependency resolver
>>>> (copied from Zeppelin), and 'spark' package has some classes for
>>>> classloader (yes also copied from Spark).
>>>>
>>>> Please share your experiences if you have knowledges regarding this
>> area.
>>>>
>>>> - Jungtaek Lim
>>>>
>>>> 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
>>> 작성:
>>>>
>>>>> You can do that with a combination of the distributed cache and
>> setting
>>>>> the classpath, just like with hadoop. It is not as clean as it could
>>> be,
>>>>> but it does work. - Bobby
>>>>>
>>>>> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <
>>> arunm@apache.org>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Shade and relocate the external modules sounds ok as a short term
>>>>> solution.
>>>>>
>>>>> For the long term we should consider something like the second option
>> to
>>>>> add external modules without shipping uber jars.
>>>>>
>>>>> Thanks,
>>>>> Arun
>>>>>
>>>>> On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>>>>
>>>>>> Hi devs,
>>>>>>
>>>>>> AFAIK, we had been struggled to resolve dependency issues for
>>> storm-core.
>>>>>> As we all know, the strategy we have been using is shade &
>> relocating.
>>>>>>
>>>>>> Now State and Storm SQL requires that some of external modules need
>> to
>>> be
>>>>>> included to extlib, which is the classpath workers refer.
>>>>>>
>>>>>> http://issues.apache.org/jira/browse/STORM-1881
>>>>>> https://issues.apache.org/jira/browse/STORM-1435
>>>>>>
>>>>>> There're two issues here:
>>>>>> - We don't make uber jar for external modules so users need to find
>> and
>>>>>> copy dependencies jars to extlib manually.
>>>>>> - External modules also use Guava and Jackson and so on which are
>>> origin
>>>>> of
>>>>>> version conflict issues.
>>>>>>
>>>>>> So we should apply the shade & relocating strategy for every external
>>>>>> modules (at least storm-redis, storm-kafka, storm-sql-core,
>>>>>> storm-sql-kafka), or introduce the way to add the dependency without
>>>>> adding
>>>>>> them to extlib. (like --packages and --jar for Spark)
>>>>>>
>>>>>> Please express your opinions about this.
>>>>>>
>>>>>> Thanks,
>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
Re: [DISCUSSION] Policy of resolving dependencies for non storm-core modules
Posted by Jungtaek Lim <ka...@gmail.com>.
Not yet. It would be good to continue discussion on this thread for now.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 8월 1일 (월) 오후 1:38, Satish Duggana <sa...@gmail.com>님이 작성:
> Hi,
> Did we find a way to comment on the proposal? How can we proceed with this
> discussion?
>
> Thanks,
> Satish.
>
>
> On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> > I've made a design document for adding jars and maven artifacts at
> > submission.
> > Since we don't have any formats for this, I just borrowed KIP format.
> > That's not we would like to adopt that format, it's just me.
> >
> >
> >
> https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing
> >
> > Btw, I guess ASF policy requires that discussion history should be logged
> > to Apache Infra.
> > (So I only grant view permission who opens the doc via link.)
> >
> > So what's recommended way to do? Would we want to file an issue for JIRA
> > with attaching design doc to PDF and discuss there?
> >
> > - Jungtaek Lim
> >
> > 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >
> > > Thought about it once more, and found that former approach still needs
> to
> > > add 'provided' scope libraries to extlib directory..
> > >
> > > Along with thinking former approach, I've experimented a bit of latter
> > > approach by creating POC project. Since I don't know about copy and use
> > ASF
> > > project codes for personal use, I'd just share main class to see if
> this
> > > POC and my theory makes sense for us.
> > > https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6
> > > 'zeppelin' package has some classes for transitive dependency resolver
> > > (copied from Zeppelin), and 'spark' package has some classes for
> > > classloader (yes also copied from Spark).
> > >
> > > Please share your experiences if you have knowledges regarding this
> area.
> > >
> > > - Jungtaek Lim
> > >
> > > 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
> > 작성:
> > >
> > >> You can do that with a combination of the distributed cache and
> setting
> > >> the classpath, just like with hadoop. It is not as clean as it could
> > be,
> > >> but it does work. - Bobby
> > >>
> > >> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <
> > arunm@apache.org>
> > >> wrote:
> > >>
> > >>
> > >> Shade and relocate the external modules sounds ok as a short term
> > >> solution.
> > >>
> > >> For the long term we should consider something like the second option
> to
> > >> add external modules without shipping uber jars.
> > >>
> > >> Thanks,
> > >> Arun
> > >>
> > >> On 7/22/16, 6:07 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> > >>
> > >> >Hi devs,
> > >> >
> > >> >AFAIK, we had been struggled to resolve dependency issues for
> > storm-core.
> > >> >As we all know, the strategy we have been using is shade &
> relocating.
> > >> >
> > >> >Now State and Storm SQL requires that some of external modules need
> to
> > be
> > >> >included to extlib, which is the classpath workers refer.
> > >> >
> > >> >http://issues.apache.org/jira/browse/STORM-1881
> > >> >https://issues.apache.org/jira/browse/STORM-1435
> > >> >
> > >> >There're two issues here:
> > >> >- We don't make uber jar for external modules so users need to find
> and
> > >> >copy dependencies jars to extlib manually.
> > >> >- External modules also use Guava and Jackson and so on which are
> > origin
> > >> of
> > >> >version conflict issues.
> > >> >
> > >> >So we should apply the shade & relocating strategy for every external
> > >> >modules (at least storm-redis, storm-kafka, storm-sql-core,
> > >> >storm-sql-kafka), or introduce the way to add the dependency without
> > >> adding
> > >> >them to extlib. (like --packages and --jar for Spark)
> > >> >
> > >> >Please express your opinions about this.
> > >> >
> > >> >Thanks,
> > >> >Jungtaek Lim (HeartSaVioR)
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> >
>