You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@myriad.apache.org by Patrick Wong <pw...@maprtech.com> on 2015/07/17 22:50:43 UTC
Question about build
Hello Myriad Team,
I noticed that the application plugin is applied to all subprojects. Does
this mean that you are supposed to be able to run "gradle installDist" to
get a working local installation for each subproject, and "gradle run" to
run them?
--
Patrick Wong
510.386.7205
mapr.com
Re: Question about build
Posted by Yuliya Feldman <yu...@yahoo.com.INVALID>.
Yes,
If you rely just on fat jar. For dev it is great tool, but for production may not be so much.
On Jul 19, 2015, at 8:46 PM, Ken Sipe <ke...@mesosphere.io> wrote:
> the point of a fat jar is to not have jars “outside”… it would make sense that would cause challenges. We have a fat jar now because 1) that is the way it was original dev and 2) the ability to distribute all things the needed by the executor in a runtime jar. Fat jars work great when they are all inclusive, all stand alone. if in a shared class path space then bad things are sure to happen:)
>
> ken
>> On Jul 19, 2015, at 10:08 PM, Yuliya Feldman <yu...@yahoo.com.INVALID> wrote:
>>
>> I am wondering about usage of fat jar in principal - fat jar unless it has the same versions of jars inside fat one and outside can create issues. slf jars , mesos jars
>>
>> What is your experience? Ours so far was quite negative
>>
>>
>>
>> On Jul 19, 2015, at 6:46 PM, Ken Sipe <ke...@mesosphere.io> wrote:
>>
>>>
>>>> On Jul 17, 2015, at 5:51 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
>>>>
>>>> Ken, Swapnil is looking at turning the executor into a YARN NM AuxService
>>>> anyway, so it might end up without a main too. Then we wouldn't need the
>>>> application plugin at all?
>>>>
>>> actually I don’t think we need the application plugin right now. I think the capsule uses the application property for the main for configuration. We can go without it I believe. When we make the executor a NM AuxService, we may need to look at new packaging. I believe the capsule is a “runnable” fat jar and we will need just a standard fat jar at that point. I’ll look into it.
>>>
>>> ken
>>>
>>>> On Fri, Jul 17, 2015 at 3:31 PM, Ken Sipe <ken@mesosphere.io <ma...@mesosphere.io>> wrote:
>>>>
>>>>> Hey Patrick,
>>>>>
>>>>> I was the one that refactored myriad in to a multi-project structure. I
>>>>> took what was originally there and tried to maintain the semantics into the
>>>>> new structure (meaning it was my intention to not change anything other
>>>>> than the structure). At first glance this appears to be a miss on my
>>>>> part. There is no main class in the scheduler. The scheduler is a YARN
>>>>> resource manager.
>>>>>
>>>>> Looking it over in more detail, I’m not sure we even need the application
>>>>> plugin for the executor. It is required for the building of the capsule,
>>>>> but that could be declared in the capsule instead of using the
>>>>> configuration of the application. It is worth more investigation.
>>>>>
>>>>> As for long, there are no other projects other than the executor that need
>>>>> the application plugin. I have refactored this and have submitted it as
>>>>> PR-120: https://github.com/mesos/myriad/pull/120 <
>>>>> https://github.com/mesos/myriad/pull/120 <https://github.com/mesos/myriad/pull/120>>
>>>>>
>>>>> Thanks for helping to identify this.
>>>>>
>>>>> Ken
>>>>>
>>>>>> On Jul 17, 2015, at 3:50 PM, Patrick Wong <pw...@maprtech.com> wrote:
>>>>>>
>>>>>> Hello Myriad Team,
>>>>>>
>>>>>> I noticed that the application plugin is applied to all subprojects. Does
>>>>>> this mean that you are supposed to be able to run "gradle installDist" to
>>>>>> get a working local installation for each subproject, and "gradle run" to
>>>>>> run them?
>>>>>>
>>>>>> --
>>>>>> Patrick Wong
>>>>>> 510.386.7205
>>>>>>
>>>>>> mapr.com
>>>
>
Re: Question about build
Posted by Ken Sipe <ke...@mesosphere.io>.
the point of a fat jar is to not have jars “outside”… it would make sense that would cause challenges. We have a fat jar now because 1) that is the way it was original dev and 2) the ability to distribute all things the needed by the executor in a runtime jar. Fat jars work great when they are all inclusive, all stand alone. if in a shared class path space then bad things are sure to happen:)
ken
> On Jul 19, 2015, at 10:08 PM, Yuliya Feldman <yu...@yahoo.com.INVALID> wrote:
>
> I am wondering about usage of fat jar in principal - fat jar unless it has the same versions of jars inside fat one and outside can create issues. slf jars , mesos jars
>
> What is your experience? Ours so far was quite negative
>
>
>
> On Jul 19, 2015, at 6:46 PM, Ken Sipe <ke...@mesosphere.io> wrote:
>
>>
>>> On Jul 17, 2015, at 5:51 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
>>>
>>> Ken, Swapnil is looking at turning the executor into a YARN NM AuxService
>>> anyway, so it might end up without a main too. Then we wouldn't need the
>>> application plugin at all?
>>>
>> actually I don’t think we need the application plugin right now. I think the capsule uses the application property for the main for configuration. We can go without it I believe. When we make the executor a NM AuxService, we may need to look at new packaging. I believe the capsule is a “runnable” fat jar and we will need just a standard fat jar at that point. I’ll look into it.
>>
>> ken
>>
>>> On Fri, Jul 17, 2015 at 3:31 PM, Ken Sipe <ken@mesosphere.io <ma...@mesosphere.io>> wrote:
>>>
>>>> Hey Patrick,
>>>>
>>>> I was the one that refactored myriad in to a multi-project structure. I
>>>> took what was originally there and tried to maintain the semantics into the
>>>> new structure (meaning it was my intention to not change anything other
>>>> than the structure). At first glance this appears to be a miss on my
>>>> part. There is no main class in the scheduler. The scheduler is a YARN
>>>> resource manager.
>>>>
>>>> Looking it over in more detail, I’m not sure we even need the application
>>>> plugin for the executor. It is required for the building of the capsule,
>>>> but that could be declared in the capsule instead of using the
>>>> configuration of the application. It is worth more investigation.
>>>>
>>>> As for long, there are no other projects other than the executor that need
>>>> the application plugin. I have refactored this and have submitted it as
>>>> PR-120: https://github.com/mesos/myriad/pull/120 <
>>>> https://github.com/mesos/myriad/pull/120 <https://github.com/mesos/myriad/pull/120>>
>>>>
>>>> Thanks for helping to identify this.
>>>>
>>>> Ken
>>>>
>>>>> On Jul 17, 2015, at 3:50 PM, Patrick Wong <pw...@maprtech.com> wrote:
>>>>>
>>>>> Hello Myriad Team,
>>>>>
>>>>> I noticed that the application plugin is applied to all subprojects. Does
>>>>> this mean that you are supposed to be able to run "gradle installDist" to
>>>>> get a working local installation for each subproject, and "gradle run" to
>>>>> run them?
>>>>>
>>>>> --
>>>>> Patrick Wong
>>>>> 510.386.7205
>>>>>
>>>>> mapr.com
>>
Re: Question about build
Posted by Yuliya Feldman <yu...@yahoo.com.INVALID>.
I am wondering about usage of fat jar in principal - fat jar unless it has the same versions of jars inside fat one and outside can create issues. slf jars , mesos jars
What is your experience? Ours so far was quite negative
On Jul 19, 2015, at 6:46 PM, Ken Sipe <ke...@mesosphere.io> wrote:
>
>> On Jul 17, 2015, at 5:51 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
>>
>> Ken, Swapnil is looking at turning the executor into a YARN NM AuxService
>> anyway, so it might end up without a main too. Then we wouldn't need the
>> application plugin at all?
>>
> actually I don’t think we need the application plugin right now. I think the capsule uses the application property for the main for configuration. We can go without it I believe. When we make the executor a NM AuxService, we may need to look at new packaging. I believe the capsule is a “runnable” fat jar and we will need just a standard fat jar at that point. I’ll look into it.
>
> ken
>
>> On Fri, Jul 17, 2015 at 3:31 PM, Ken Sipe <ken@mesosphere.io <ma...@mesosphere.io>> wrote:
>>
>>> Hey Patrick,
>>>
>>> I was the one that refactored myriad in to a multi-project structure. I
>>> took what was originally there and tried to maintain the semantics into the
>>> new structure (meaning it was my intention to not change anything other
>>> than the structure). At first glance this appears to be a miss on my
>>> part. There is no main class in the scheduler. The scheduler is a YARN
>>> resource manager.
>>>
>>> Looking it over in more detail, I’m not sure we even need the application
>>> plugin for the executor. It is required for the building of the capsule,
>>> but that could be declared in the capsule instead of using the
>>> configuration of the application. It is worth more investigation.
>>>
>>> As for long, there are no other projects other than the executor that need
>>> the application plugin. I have refactored this and have submitted it as
>>> PR-120: https://github.com/mesos/myriad/pull/120 <
>>> https://github.com/mesos/myriad/pull/120 <https://github.com/mesos/myriad/pull/120>>
>>>
>>> Thanks for helping to identify this.
>>>
>>> Ken
>>>
>>>> On Jul 17, 2015, at 3:50 PM, Patrick Wong <pw...@maprtech.com> wrote:
>>>>
>>>> Hello Myriad Team,
>>>>
>>>> I noticed that the application plugin is applied to all subprojects. Does
>>>> this mean that you are supposed to be able to run "gradle installDist" to
>>>> get a working local installation for each subproject, and "gradle run" to
>>>> run them?
>>>>
>>>> --
>>>> Patrick Wong
>>>> 510.386.7205
>>>>
>>>> mapr.com
>
Re: Question about build
Posted by Ken Sipe <ke...@mesosphere.io>.
> On Jul 17, 2015, at 5:51 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
>
> Ken, Swapnil is looking at turning the executor into a YARN NM AuxService
> anyway, so it might end up without a main too. Then we wouldn't need the
> application plugin at all?
>
actually I don’t think we need the application plugin right now. I think the capsule uses the application property for the main for configuration. We can go without it I believe. When we make the executor a NM AuxService, we may need to look at new packaging. I believe the capsule is a “runnable” fat jar and we will need just a standard fat jar at that point. I’ll look into it.
ken
> On Fri, Jul 17, 2015 at 3:31 PM, Ken Sipe <ken@mesosphere.io <ma...@mesosphere.io>> wrote:
>
>> Hey Patrick,
>>
>> I was the one that refactored myriad in to a multi-project structure. I
>> took what was originally there and tried to maintain the semantics into the
>> new structure (meaning it was my intention to not change anything other
>> than the structure). At first glance this appears to be a miss on my
>> part. There is no main class in the scheduler. The scheduler is a YARN
>> resource manager.
>>
>> Looking it over in more detail, I’m not sure we even need the application
>> plugin for the executor. It is required for the building of the capsule,
>> but that could be declared in the capsule instead of using the
>> configuration of the application. It is worth more investigation.
>>
>> As for long, there are no other projects other than the executor that need
>> the application plugin. I have refactored this and have submitted it as
>> PR-120: https://github.com/mesos/myriad/pull/120 <
>> https://github.com/mesos/myriad/pull/120 <https://github.com/mesos/myriad/pull/120>>
>>
>> Thanks for helping to identify this.
>>
>> Ken
>>
>>> On Jul 17, 2015, at 3:50 PM, Patrick Wong <pw...@maprtech.com> wrote:
>>>
>>> Hello Myriad Team,
>>>
>>> I noticed that the application plugin is applied to all subprojects. Does
>>> this mean that you are supposed to be able to run "gradle installDist" to
>>> get a working local installation for each subproject, and "gradle run" to
>>> run them?
>>>
>>> --
>>> Patrick Wong
>>> 510.386.7205
>>>
>>> mapr.com
Re: Question about build
Posted by Jim Klucar <kl...@gmail.com>.
Then I guess we're just a we service then ;)
On Friday, July 17, 2015, Adam Bordelon <ad...@mesosphere.io> wrote:
> Ken, Swapnil is looking at turning the executor into a YARN NM AuxService
> anyway, so it might end up without a main too. Then we wouldn't need the
> application plugin at all?
>
> On Fri, Jul 17, 2015 at 3:31 PM, Ken Sipe <ken@mesosphere.io
> <javascript:;>> wrote:
>
> > Hey Patrick,
> >
> > I was the one that refactored myriad in to a multi-project structure. I
> > took what was originally there and tried to maintain the semantics into
> the
> > new structure (meaning it was my intention to not change anything other
> > than the structure). At first glance this appears to be a miss on my
> > part. There is no main class in the scheduler. The scheduler is a YARN
> > resource manager.
> >
> > Looking it over in more detail, I’m not sure we even need the application
> > plugin for the executor. It is required for the building of the capsule,
> > but that could be declared in the capsule instead of using the
> > configuration of the application. It is worth more investigation.
> >
> > As for long, there are no other projects other than the executor that
> need
> > the application plugin. I have refactored this and have submitted it as
> > PR-120: https://github.com/mesos/myriad/pull/120 <
> > https://github.com/mesos/myriad/pull/120>
> >
> > Thanks for helping to identify this.
> >
> > Ken
> >
> > > On Jul 17, 2015, at 3:50 PM, Patrick Wong <pwong@maprtech.com
> <javascript:;>> wrote:
> > >
> > > Hello Myriad Team,
> > >
> > > I noticed that the application plugin is applied to all subprojects.
> Does
> > > this mean that you are supposed to be able to run "gradle installDist"
> to
> > > get a working local installation for each subproject, and "gradle run"
> to
> > > run them?
> > >
> > > --
> > > Patrick Wong
> > > 510.386.7205
> > >
> > > mapr.com
> >
> >
>
Re: Question about build
Posted by Adam Bordelon <ad...@mesosphere.io>.
Ken, Swapnil is looking at turning the executor into a YARN NM AuxService
anyway, so it might end up without a main too. Then we wouldn't need the
application plugin at all?
On Fri, Jul 17, 2015 at 3:31 PM, Ken Sipe <ke...@mesosphere.io> wrote:
> Hey Patrick,
>
> I was the one that refactored myriad in to a multi-project structure. I
> took what was originally there and tried to maintain the semantics into the
> new structure (meaning it was my intention to not change anything other
> than the structure). At first glance this appears to be a miss on my
> part. There is no main class in the scheduler. The scheduler is a YARN
> resource manager.
>
> Looking it over in more detail, I’m not sure we even need the application
> plugin for the executor. It is required for the building of the capsule,
> but that could be declared in the capsule instead of using the
> configuration of the application. It is worth more investigation.
>
> As for long, there are no other projects other than the executor that need
> the application plugin. I have refactored this and have submitted it as
> PR-120: https://github.com/mesos/myriad/pull/120 <
> https://github.com/mesos/myriad/pull/120>
>
> Thanks for helping to identify this.
>
> Ken
>
> > On Jul 17, 2015, at 3:50 PM, Patrick Wong <pw...@maprtech.com> wrote:
> >
> > Hello Myriad Team,
> >
> > I noticed that the application plugin is applied to all subprojects. Does
> > this mean that you are supposed to be able to run "gradle installDist" to
> > get a working local installation for each subproject, and "gradle run" to
> > run them?
> >
> > --
> > Patrick Wong
> > 510.386.7205
> >
> > mapr.com
>
>
Re: Question about build
Posted by Ken Sipe <ke...@mesosphere.io>.
Hey Patrick,
I was the one that refactored myriad in to a multi-project structure. I took what was originally there and tried to maintain the semantics into the new structure (meaning it was my intention to not change anything other than the structure). At first glance this appears to be a miss on my part. There is no main class in the scheduler. The scheduler is a YARN resource manager.
Looking it over in more detail, I’m not sure we even need the application plugin for the executor. It is required for the building of the capsule, but that could be declared in the capsule instead of using the configuration of the application. It is worth more investigation.
As for long, there are no other projects other than the executor that need the application plugin. I have refactored this and have submitted it as PR-120: https://github.com/mesos/myriad/pull/120 <https://github.com/mesos/myriad/pull/120>
Thanks for helping to identify this.
Ken
> On Jul 17, 2015, at 3:50 PM, Patrick Wong <pw...@maprtech.com> wrote:
>
> Hello Myriad Team,
>
> I noticed that the application plugin is applied to all subprojects. Does
> this mean that you are supposed to be able to run "gradle installDist" to
> get a working local installation for each subproject, and "gradle run" to
> run them?
>
> --
> Patrick Wong
> 510.386.7205
>
> mapr.com