You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Chris Mattmann <ch...@gmail.com> on 2014/11/05 19:03:06 UTC
Re: Extra Compiler Tools
Hi Mike,
Great discussion. It would be nice if there was
a protoc Maven plugin:
http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
Looks like there is. My suggestion:
1. use a Profile, something like -Pwith-mesos and
then when activated;
2. call the above plugin if -Pwith-mesos is activated
in the resource manager
Sound good?
Cheers,
Chris
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: <de...@oodt.apache.org>
Date: Wednesday, November 5, 2014 at 11:46 AM
To: <de...@oodt.apache.org>
Subject: Extra Compiler Tools
>All,
>
>I am trying to integrate apache-mesos with our resource manager. However,
>mesos uses a technology called "protobuff" from Google for
>marshaling/unmarshaling data.
>
>This requires running a tool called "protoc" to generate a source file in
>java. What is the best way to integrate this step into our build process?
>
>Options I can conceive of:
> -Check in generated java file
> -Require "protoc" installation to build resource manager
> -Separate extra resource package into new module
>
>None of these ideas are very clean.
>
>Any other ideas? I tried setting up a profile to only compile these
>sources when selected, but that turned out not to work.
>
>-Michael Starch
Re: Extra Compiler Tools
Posted by Michael Starch <st...@umich.edu>.
I tried this approach. The plugin requires a path to the "protoc" tool and
thus a working installation. This is what prompted the discussion.
Running the plugin under a profile works. However, not running the plugin
causes compile errors in dependant code. Excluding this code except within
the profile doesn't seem to work, and is considered by some to be bad form
because there is nothing inside the jar file that notes which profiles were
used to compile.
Any ideas on how to continue?
Michael
On Nov 5, 2014 11:04 AM, "Chris Mattmann" <ch...@gmail.com> wrote:
> Hi Mike,
>
> Great discussion. It would be nice if there was
> a protoc Maven plugin:
>
> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>
>
> Looks like there is. My suggestion:
>
> 1. use a Profile, something like -Pwith-mesos and
> then when activated;
> 2. call the above plugin if -Pwith-mesos is activated
> in the resource manager
>
> Sound good?
>
> Cheers,
> Chris
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Michael Starch <st...@umich.edu>
> Reply-To: <de...@oodt.apache.org>
> Date: Wednesday, November 5, 2014 at 11:46 AM
> To: <de...@oodt.apache.org>
> Subject: Extra Compiler Tools
>
> >All,
> >
> >I am trying to integrate apache-mesos with our resource manager. However,
> >mesos uses a technology called "protobuff" from Google for
> >marshaling/unmarshaling data.
> >
> >This requires running a tool called "protoc" to generate a source file in
> >java. What is the best way to integrate this step into our build process?
> >
> >Options I can conceive of:
> > -Check in generated java file
> > -Require "protoc" installation to build resource manager
> > -Separate extra resource package into new module
> >
> >None of these ideas are very clean.
> >
> >Any other ideas? I tried setting up a profile to only compile these
> >sources when selected, but that turned out not to work.
> >
> >-Michael Starch
>
>
>
Re: Extra Compiler Tools
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Ship it :) It’s ugly, but will keep the core stable at least through
another revision.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Date: Friday, November 7, 2014 at 10:39 PM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Re: Extra Compiler Tools
>Chris, Et Al.
>
>The solution write-up can be found here. Turns out it isn't too
>ugly....just has a lot of duplication with the classes it is wrapping.
>
>https://reviews.apache.org/r/27773/
>
>Enjoy,
>
>-Michael
>
>On Fri, Nov 7, 2014 at 9:46 AM, Michael Starch <st...@umich.edu> wrote:
>
>> Sound fine, expect to see this on review board when I get it done and
>> passing my test.
>>
>> Michael
>> On Nov 7, 2014 9:44 AM, "Chris Mattmann" <ch...@gmail.com>
>>wrote:
>>
>>> Yep I’d like to see the non changing
>>> core structs version, in a patch to
>>> evaluate it. You said you know how, so let’s see it
>>> and then go from there. We can reserve 0.9 to
>>> consider changing structs if the code becomes
>>> too unwieldily to maintain.
>>>
>>> Good?
>>>
>>> Cheers,
>>> Chris
>>>
>>> ------------------------
>>> Chris Mattmann
>>> chris.mattmann@gmail.com
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Michael Starch <st...@umich.edu>
>>> Reply-To: <de...@oodt.apache.org>
>>> Date: Friday, November 7, 2014 at 9:21 AM
>>> To: <de...@oodt.apache.org>
>>> Subject: Re: Extra Compiler Tools
>>>
>>> >Chris,
>>> >
>>> >Did you ever come to a conclusion on this?
>>> >
>>> >-Michael
>>> >
>>> >On Wed, Nov 5, 2014 at 7:14 PM, Chris Mattmann
>>><chris.mattmann@gmail.com
>>> >
>>> >wrote:
>>> >
>>> >> OK, let me think about this tonight.
>>> >> Maybe we can figure this out tomorrow,
>>> >> I won’t hold this up longer than that.
>>> >>
>>> >> ------------------------
>>> >> Chris Mattmann
>>> >> chris.mattmann@gmail.com
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: Michael Starch <st...@umich.edu>
>>> >> Reply-To: <de...@oodt.apache.org>
>>> >> Date: Wednesday, November 5, 2014 at 8:23 PM
>>> >> To: <de...@oodt.apache.org>
>>> >> Subject: Re: Extra Compiler Tools
>>> >>
>>> >> >According to the specs, any subclass that implements Serializable
>>>must
>>> >> >manually implement the serialization of the parents' members. I
>>> tested
>>> >> >this and it fails exactly as expected. The parent's members aren't
>>> >> >serialized.
>>> >> >
>>> >> >Also, JobInput is an interface.... so I would have no way of
>>>catching
>>> >>all
>>> >> >of the possible implementations that could come at me.
>>> >> >
>>> >> >Michael
>>> >> >On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
>>> >> >chris.a.mattmann@jpl.nasa.gov> wrote:
>>> >> >
>>> >> >> Got it, Mike.
>>> >> >>
>>> >> >> Hmm, how about simply creating SerializableJobSpec and
>>> >> >> SerializableJob and SerializableJobInput and then making
>>> >> >> them sub-class their parents and implement Serializable.
>>> >> >> Then, use these classes in your Mesos implementation.
>>> >> >> That seems self-contained, doesn’t change core classes,
>>> >> >> and pretty easy, right?
>>> >> >>
>>> >> >> Cheers,
>>> >> >> Chris
>>> >> >>
>>> >> >>
>>> >> >>
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> >> Chris Mattmann, Ph.D.
>>> >> >> Chief Architect
>>> >> >> Instrument Software and Science Data Systems Section (398)
>>> >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> >> Office: 168-519, Mailstop: 168-527
>>> >> >> Email: chris.a.mattmann@nasa.gov
>>> >> >> WWW: http://sunset.usc.edu/~mattmann/
>>> >> >>
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> >> Adjunct Associate Professor, Computer Science Department
>>> >> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> >>
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> -----Original Message-----
>>> >> >> From: Michael Starch <st...@umich.edu>
>>> >> >> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>> >> >> Date: Wednesday, November 5, 2014 at 8:00 PM
>>> >> >> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>> >> >> Subject: Re: Extra Compiler Tools
>>> >> >>
>>> >> >> >I need to serialize a JobSpec and children (Job and JobInput)
>>>to a
>>> >> >>byte[].
>>> >> >> >Java can do this automatically by marking all three as
>>> Serializable.
>>> >> >> >
>>> >> >> >The work around is to manually serialize to a private inner
>>>struct
>>> >>and
>>> >> >> >back
>>> >> >> >out again. The inner class will have members for each member in
>>> the
>>> >> >> >JobSpec and children. Java can the auto-serialize that without
>>> >> >>changing
>>> >> >> >the other three.
>>> >> >> >
>>> >> >> >It is ugly, and essentially a reimplementation of those three
>>> >> >> >classes....but it is entirely self-contained.
>>> >> >> >
>>> >> >> >Michael
>>> >> >> >On Nov 5, 2014 6:45 PM, "Chris Mattmann"
>>><chris.mattmann@gmail.com
>>> >
>>> >> >> wrote:
>>> >> >> >
>>> >> >> >> Hey Mike,
>>> >> >> >>
>>> >> >> >> Hmm, what’s the work around just so I know
>>> >> >> >> what we’re trading against?
>>> >> >> >>
>>> >> >> >> Cheers,
>>> >> >> >> Chris
>>> >> >> >>
>>> >> >> >> ------------------------
>>> >> >> >> Chris Mattmann
>>> >> >> >> chris.mattmann@gmail.com
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> -----Original Message-----
>>> >> >> >> From: Michael Starch <st...@umich.edu>
>>> >> >> >> Reply-To: <de...@oodt.apache.org>
>>> >> >> >> Date: Wednesday, November 5, 2014 at 6:31 PM
>>> >> >> >> To: <de...@oodt.apache.org>
>>> >> >> >> Subject: Re: Extra Compiler Tools
>>> >> >> >>
>>> >> >> >> >That is basically what I did. Regardless, protobuff proves
>>>to be
>>> >> >> >>overkill.
>>> >> >> >> >
>>> >> >> >> >If I mark those classes as serializable, the correct
>>>solution is
>>> >>2
>>> >> >> >>lines
>>> >> >> >> >of
>>> >> >> >> >code. (protobuff was like 20). Wrote a test case, and it
>>>works
>>> >> >> >> >perfectly.
>>> >> >> >> >
>>> >> >> >> >If I cannot make JobSpec Job and JonInput implement
>>>Serializable
>>> >> >>then
>>> >> >> >>the
>>> >> >> >> >work around is simple too.
>>> >> >> >> >
>>> >> >> >> >What do you think? Should I mark them as Serializable, or
>>>use a
>>> >> >> >> >work-around. Either is a better solution than protobuff.
>>> >> >> >> >
>>> >> >> >> >Michael
>>> >> >> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann"
>>> >><ch...@gmail.com>
>>> >> >> >> wrote:
>>> >> >> >> >
>>> >> >> >> >> Mike, have you looked at this yet?
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >>
>>> >> >>
>>> >>
>>> >>
>>>
>>>http://techtraits.com/build%20management/maven/2011/09/09/compiling-prot
>>>o
>>> >> >> >> >>co
>>> >> >> >> >> l-buffers-from-maven/
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> I’m going to play with it tonight and see if
>>> >> >> >> >> I can help here. Do you have some files I can test
>>> >> >> >> >> with? Can you attach them to JIRA or dropbox them to me
>>> >> >> >> >> so I can scope?
>>> >> >> >> >>
>>> >> >> >> >> Cheers,
>>> >> >> >> >> Chris
>>> >> >> >> >>
>>> >> >> >> >> ------------------------
>>> >> >> >> >> Chris Mattmann
>>> >> >> >> >> chris.mattmann@gmail.com
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> -----Original Message-----
>>> >> >> >> >> From: Michael Starch <st...@umich.edu>
>>> >> >> >> >> Reply-To: <de...@oodt.apache.org>
>>> >> >> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM
>>> >> >> >> >> To: <de...@oodt.apache.org>
>>> >> >> >> >> Subject: Re: Extra Compiler Tools
>>> >> >> >> >>
>>> >> >> >> >> >Ok....time for an audible. Protoc needs to be built from
>>> >> >>source, no
>>> >> >> >> >> >binary
>>> >> >> >> >> >distributions available. Thus I am going to purge
>>> >>proto-buffers
>>> >> >> >>from
>>> >> >> >> >>the
>>> >> >> >> >> >new code and be done with it.
>>> >> >> >> >> >
>>> >> >> >> >> >Any problem making the following classes/interfaces
>>>implement
>>> >> >> >> >> >java.io.Serializable:
>>> >> >> >> >> >
>>> >> >> >> >> >JobSpec
>>> >> >> >> >> >Job
>>> >> >> >> >> >JobInput
>>> >> >> >> >> >
>>> >> >> >> >> >Doing so would allow apache and native java serialization
>>>and
>>> >> >>thus
>>> >> >> >>we
>>> >> >> >> >> >wouldn't need something like proto-buffers.
>>> >> >> >> >> >
>>> >> >> >> >> >-Michael
>>> >> >> >> >> >Thanks Mike +1
>>> >> >> >> >> >
>>> >> >> >> >> >------------------------
>>> >> >> >> >> >Chris Mattmann
>>> >> >> >> >> >chris.mattmann@gmail.com
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >-----Original Message-----
>>> >> >> >> >> >From: Michael Starch <st...@umich.edu>
>>> >> >> >> >> >Reply-To: <de...@oodt.apache.org>
>>> >> >> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
>>> >> >> >> >> >To: <de...@oodt.apache.org>
>>> >> >> >> >> >Subject: Re: Extra Compiler Tools
>>> >> >> >> >> >
>>> >> >> >> >> >>Looks like you followed the same reasoning chain that I
>>>did.
>>> >> >>Yes,
>>> >> >> >>I
>>> >> >> >> >>came
>>> >> >> >> >> >>to the same conclusion that ant-build was best.
>>> >> >> >> >> >>
>>> >> >> >> >> >>I wasn't sure how to download protoc, but you just
>>>answered
>>> >> >> >> >>that....so I
>>> >> >> >> >> >>think this is a great solution!
>>> >> >> >> >> >>
>>> >> >> >> >> >>Thanks,
>>> >> >> >> >> >>
>>> >> >> >> >> >>Michael
>>> >> >> >> >> >>
>>> >> >> >> >> >>
>>> >> >> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
>>> >> >> >> >> >><ch...@gmail.com>
>>> >> >> >> >> >>wrote:
>>> >> >> >> >> >>
>>> >> >> >> >> >>> Hi Mike,
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Thanks for flushing this out.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> My thoughts on the below:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> -----Original Message-----
>>> >> >> >> >> >>> From: Michael Starch <st...@umich.edu>
>>> >> >> >> >> >>> Reply-To: <de...@oodt.apache.org>
>>> >> >> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
>>> >> >> >> >> >>> To: <de...@oodt.apache.org>
>>> >> >> >> >> >>> Subject: Re: Extra Compiler Tools
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> >I tried this approach. The plugin requires a path to
>>>the
>>> >> >> >>"protoc"
>>> >> >> >> >>tool
>>> >> >> >> >> >>>and
>>> >> >> >> >> >>> >thus a working installation. This is what prompted
>>>the
>>> >> >> >>discussion.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Ah - no worries, what you could do is:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled;
>>>and
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> >
>>> >> >> >> >> >>> >Running the plugin under a profile works.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Yep.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> > However, not running the plugin
>>> >> >> >> >> >>> >causes compile errors in dependant code. Excluding
>>>this
>>> >>code
>>> >> >> >> >>except
>>> >> >> >> >> >>> >within
>>> >> >> >> >> >>> >the profile doesn't seem to work, and is considered by
>>> >>some
>>> >> >>to
>>> >> >> >>be
>>> >> >> >> >>bad
>>> >> >> >> >> >>>form
>>> >> >> >> >> >>> >because there is nothing inside the jar file that
>>>notes
>>> >>which
>>> >> >> >> >>profiles
>>> >> >> >> >> >>> >were
>>> >> >> >> >> >>> >used to compile.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Got it. Suggestion here would be:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 2. create a new module, cas-resource-mesos, and inside
>>>of
>>> >>that
>>> >> >> >> >>module,
>>> >> >> >> >> >>> take one of the following approaches, assuming the
>>>module
>>> >>is
>>> >> >> >> >>activated
>>> >> >> >> >> >>> when -Pwith-mesos is enabled:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 2a. Maven Antrun like so (in this old example):
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >>
>>> >> >> >>
>>> >> >>
>>> >>
>>>
>>>http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
>>> >> >> >> >> >>>t
>>> >> >> >> >> >>>o-
>>> >> >> >> >> >>> maven2-build
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> (pro: more flexibility in case protoc isn¹t there; to
>>>fail
>>> >>on
>>> >> >> >> >>error; to
>>> >> >> >> >> >>> only compile if
>>> >> >> >> >> >>> protoc is available
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 2b. Maven protobuf plugin
>>> >> >> >> >> >>>
>>> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Here¹s how to enable a module with a profile:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >>
>>> >> >> >>
>>> >> >>
>>> >>
>>>
>>>http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
>>> >> >> >> >> >>>a
>>> >> >> >> >> >>>ct
>>> >> >> >> >> >>> ive-modules-by-profiles-in-maven/
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> It seems like that is a bad idea though, based on that
>>> >> >> >>discussion.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> So, here¹s another option:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 1. Inside of cas-resource (no special new module or
>>> >>anything
>>> >> >> >>else)
>>> >> >> >> >> >>> 2. include some custom Ant magic via a build.xml file
>>>and
>>> >>the
>>> >> >> >>Maven
>>> >> >> >> >> >>> AntRun plugin:
>>> >> >> >> >> >>> 2a. test if protoc is on the system path, and if not,
>>> >> >>download
>>> >> >> >>it,
>>> >> >> >> >> >>>e.g.,
>>> >> >> >> >> >>> into the target directory (gets deleted on clean)
>>> >> >> >> >> >>> 2b. call protoc and compile after 2a
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> I would suggest this solution as I think it¹s the most
>>> >>robust
>>> >> >>and
>>> >> >> >> >> >>>ensures
>>> >> >> >> >> >>> we always have a cas-resource that includes mesos and
>>> >>compiled
>>> >> >> >> >> >>>correctly.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Cheers,
>>> >> >> >> >> >>> Chris
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> >
>>> >> >> >> >> >>> >Any ideas on how to continue?
>>> >> >> >> >> >>> >
>>> >> >> >> >> >>> >Michael
>>> >> >> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
>>> >> >> >> >><ch...@gmail.com>
>>> >> >> >> >> >>> >wrote:
>>> >> >> >> >> >>> >
>>> >> >> >> >> >>> >> Hi Mike,
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> Great discussion. It would be nice if there was
>>> >> >> >> >> >>> >> a protoc Maven plugin:
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> Looks like there is. My suggestion:
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos and
>>> >> >> >> >> >>> >> then when activated;
>>> >> >> >> >> >>> >> 2. call the above plugin if -Pwith-mesos is
>>>activated
>>> >> >> >> >> >>> >> in the resource manager
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> Sound good?
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> Cheers,
>>> >> >> >> >> >>> >> Chris
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> ------------------------
>>> >> >> >> >> >>> >> Chris Mattmann
>>> >> >> >> >> >>> >> chris.mattmann@gmail.com
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> -----Original Message-----
>>> >> >> >> >> >>> >> From: Michael Starch <st...@umich.edu>
>>> >> >> >> >> >>> >> Reply-To: <de...@oodt.apache.org>
>>> >> >> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
>>> >> >> >> >> >>> >> To: <de...@oodt.apache.org>
>>> >> >> >> >> >>> >> Subject: Extra Compiler Tools
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >> >All,
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >I am trying to integrate apache-mesos with our
>>> resource
>>> >> >> >>manager.
>>> >> >> >> >> >>> >>However,
>>> >> >> >> >> >>> >> >mesos uses a technology called "protobuff" from
>>>Google
>>> >>for
>>> >> >> >> >> >>> >> >marshaling/unmarshaling data.
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >This requires running a tool called "protoc" to
>>> >>generate a
>>> >> >> >> >>source
>>> >> >> >> >> >>>file
>>> >> >> >> >> >>> >>in
>>> >> >> >> >> >>> >> >java. What is the best way to integrate this step
>>> into
>>> >> >>our
>>> >> >> >> >>build
>>> >> >> >> >> >>> >>process?
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >Options I can conceive of:
>>> >> >> >> >> >>> >> > -Check in generated java file
>>> >> >> >> >> >>> >> > -Require "protoc" installation to build resource
>>> >> >>manager
>>> >> >> >> >> >>> >> > -Separate extra resource package into new module
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >None of these ideas are very clean.
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >Any other ideas? I tried setting up a profile to
>>>only
>>> >> >> >>compile
>>> >> >> >> >> >>>these
>>> >> >> >> >> >>> >> >sources when selected, but that turned out not to
>>> work.
>>> >> >> >> >> >>> >> >
>>> >> >> >> >> >>> >> >-Michael Starch
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>> >>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >>
>>> >> >>
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
Re: Extra Compiler Tools
Posted by Michael Starch <st...@umich.edu>.
Chris, Et Al.
The solution write-up can be found here. Turns out it isn't too
ugly....just has a lot of duplication with the classes it is wrapping.
https://reviews.apache.org/r/27773/
Enjoy,
-Michael
On Fri, Nov 7, 2014 at 9:46 AM, Michael Starch <st...@umich.edu> wrote:
> Sound fine, expect to see this on review board when I get it done and
> passing my test.
>
> Michael
> On Nov 7, 2014 9:44 AM, "Chris Mattmann" <ch...@gmail.com> wrote:
>
>> Yep I’d like to see the non changing
>> core structs version, in a patch to
>> evaluate it. You said you know how, so let’s see it
>> and then go from there. We can reserve 0.9 to
>> consider changing structs if the code becomes
>> too unwieldily to maintain.
>>
>> Good?
>>
>> Cheers,
>> Chris
>>
>> ------------------------
>> Chris Mattmann
>> chris.mattmann@gmail.com
>>
>>
>>
>>
>> -----Original Message-----
>> From: Michael Starch <st...@umich.edu>
>> Reply-To: <de...@oodt.apache.org>
>> Date: Friday, November 7, 2014 at 9:21 AM
>> To: <de...@oodt.apache.org>
>> Subject: Re: Extra Compiler Tools
>>
>> >Chris,
>> >
>> >Did you ever come to a conclusion on this?
>> >
>> >-Michael
>> >
>> >On Wed, Nov 5, 2014 at 7:14 PM, Chris Mattmann <chris.mattmann@gmail.com
>> >
>> >wrote:
>> >
>> >> OK, let me think about this tonight.
>> >> Maybe we can figure this out tomorrow,
>> >> I won’t hold this up longer than that.
>> >>
>> >> ------------------------
>> >> Chris Mattmann
>> >> chris.mattmann@gmail.com
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Michael Starch <st...@umich.edu>
>> >> Reply-To: <de...@oodt.apache.org>
>> >> Date: Wednesday, November 5, 2014 at 8:23 PM
>> >> To: <de...@oodt.apache.org>
>> >> Subject: Re: Extra Compiler Tools
>> >>
>> >> >According to the specs, any subclass that implements Serializable must
>> >> >manually implement the serialization of the parents' members. I
>> tested
>> >> >this and it fails exactly as expected. The parent's members aren't
>> >> >serialized.
>> >> >
>> >> >Also, JobInput is an interface.... so I would have no way of catching
>> >>all
>> >> >of the possible implementations that could come at me.
>> >> >
>> >> >Michael
>> >> >On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
>> >> >chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> >
>> >> >> Got it, Mike.
>> >> >>
>> >> >> Hmm, how about simply creating SerializableJobSpec and
>> >> >> SerializableJob and SerializableJobInput and then making
>> >> >> them sub-class their parents and implement Serializable.
>> >> >> Then, use these classes in your Mesos implementation.
>> >> >> That seems self-contained, doesn’t change core classes,
>> >> >> and pretty easy, right?
>> >> >>
>> >> >> Cheers,
>> >> >> Chris
>> >> >>
>> >> >>
>> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >> Chris Mattmann, Ph.D.
>> >> >> Chief Architect
>> >> >> Instrument Software and Science Data Systems Section (398)
>> >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> >> Office: 168-519, Mailstop: 168-527
>> >> >> Email: chris.a.mattmann@nasa.gov
>> >> >> WWW: http://sunset.usc.edu/~mattmann/
>> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >> Adjunct Associate Professor, Computer Science Department
>> >> >> University of Southern California, Los Angeles, CA 90089 USA
>> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Michael Starch <st...@umich.edu>
>> >> >> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> >> >> Date: Wednesday, November 5, 2014 at 8:00 PM
>> >> >> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> >> >> Subject: Re: Extra Compiler Tools
>> >> >>
>> >> >> >I need to serialize a JobSpec and children (Job and JobInput) to a
>> >> >>byte[].
>> >> >> >Java can do this automatically by marking all three as
>> Serializable.
>> >> >> >
>> >> >> >The work around is to manually serialize to a private inner struct
>> >>and
>> >> >> >back
>> >> >> >out again. The inner class will have members for each member in
>> the
>> >> >> >JobSpec and children. Java can the auto-serialize that without
>> >> >>changing
>> >> >> >the other three.
>> >> >> >
>> >> >> >It is ugly, and essentially a reimplementation of those three
>> >> >> >classes....but it is entirely self-contained.
>> >> >> >
>> >> >> >Michael
>> >> >> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" <chris.mattmann@gmail.com
>> >
>> >> >> wrote:
>> >> >> >
>> >> >> >> Hey Mike,
>> >> >> >>
>> >> >> >> Hmm, what’s the work around just so I know
>> >> >> >> what we’re trading against?
>> >> >> >>
>> >> >> >> Cheers,
>> >> >> >> Chris
>> >> >> >>
>> >> >> >> ------------------------
>> >> >> >> Chris Mattmann
>> >> >> >> chris.mattmann@gmail.com
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> -----Original Message-----
>> >> >> >> From: Michael Starch <st...@umich.edu>
>> >> >> >> Reply-To: <de...@oodt.apache.org>
>> >> >> >> Date: Wednesday, November 5, 2014 at 6:31 PM
>> >> >> >> To: <de...@oodt.apache.org>
>> >> >> >> Subject: Re: Extra Compiler Tools
>> >> >> >>
>> >> >> >> >That is basically what I did. Regardless, protobuff proves to be
>> >> >> >>overkill.
>> >> >> >> >
>> >> >> >> >If I mark those classes as serializable, the correct solution is
>> >>2
>> >> >> >>lines
>> >> >> >> >of
>> >> >> >> >code. (protobuff was like 20). Wrote a test case, and it works
>> >> >> >> >perfectly.
>> >> >> >> >
>> >> >> >> >If I cannot make JobSpec Job and JonInput implement Serializable
>> >> >>then
>> >> >> >>the
>> >> >> >> >work around is simple too.
>> >> >> >> >
>> >> >> >> >What do you think? Should I mark them as Serializable, or use a
>> >> >> >> >work-around. Either is a better solution than protobuff.
>> >> >> >> >
>> >> >> >> >Michael
>> >> >> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann"
>> >><ch...@gmail.com>
>> >> >> >> wrote:
>> >> >> >> >
>> >> >> >> >> Mike, have you looked at this yet?
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
>> >> >> >> >>co
>> >> >> >> >> l-buffers-from-maven/
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> I’m going to play with it tonight and see if
>> >> >> >> >> I can help here. Do you have some files I can test
>> >> >> >> >> with? Can you attach them to JIRA or dropbox them to me
>> >> >> >> >> so I can scope?
>> >> >> >> >>
>> >> >> >> >> Cheers,
>> >> >> >> >> Chris
>> >> >> >> >>
>> >> >> >> >> ------------------------
>> >> >> >> >> Chris Mattmann
>> >> >> >> >> chris.mattmann@gmail.com
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> -----Original Message-----
>> >> >> >> >> From: Michael Starch <st...@umich.edu>
>> >> >> >> >> Reply-To: <de...@oodt.apache.org>
>> >> >> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM
>> >> >> >> >> To: <de...@oodt.apache.org>
>> >> >> >> >> Subject: Re: Extra Compiler Tools
>> >> >> >> >>
>> >> >> >> >> >Ok....time for an audible. Protoc needs to be built from
>> >> >>source, no
>> >> >> >> >> >binary
>> >> >> >> >> >distributions available. Thus I am going to purge
>> >>proto-buffers
>> >> >> >>from
>> >> >> >> >>the
>> >> >> >> >> >new code and be done with it.
>> >> >> >> >> >
>> >> >> >> >> >Any problem making the following classes/interfaces implement
>> >> >> >> >> >java.io.Serializable:
>> >> >> >> >> >
>> >> >> >> >> >JobSpec
>> >> >> >> >> >Job
>> >> >> >> >> >JobInput
>> >> >> >> >> >
>> >> >> >> >> >Doing so would allow apache and native java serialization and
>> >> >>thus
>> >> >> >>we
>> >> >> >> >> >wouldn't need something like proto-buffers.
>> >> >> >> >> >
>> >> >> >> >> >-Michael
>> >> >> >> >> >Thanks Mike +1
>> >> >> >> >> >
>> >> >> >> >> >------------------------
>> >> >> >> >> >Chris Mattmann
>> >> >> >> >> >chris.mattmann@gmail.com
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >-----Original Message-----
>> >> >> >> >> >From: Michael Starch <st...@umich.edu>
>> >> >> >> >> >Reply-To: <de...@oodt.apache.org>
>> >> >> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
>> >> >> >> >> >To: <de...@oodt.apache.org>
>> >> >> >> >> >Subject: Re: Extra Compiler Tools
>> >> >> >> >> >
>> >> >> >> >> >>Looks like you followed the same reasoning chain that I did.
>> >> >>Yes,
>> >> >> >>I
>> >> >> >> >>came
>> >> >> >> >> >>to the same conclusion that ant-build was best.
>> >> >> >> >> >>
>> >> >> >> >> >>I wasn't sure how to download protoc, but you just answered
>> >> >> >> >>that....so I
>> >> >> >> >> >>think this is a great solution!
>> >> >> >> >> >>
>> >> >> >> >> >>Thanks,
>> >> >> >> >> >>
>> >> >> >> >> >>Michael
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
>> >> >> >> >> >><ch...@gmail.com>
>> >> >> >> >> >>wrote:
>> >> >> >> >> >>
>> >> >> >> >> >>> Hi Mike,
>> >> >> >> >> >>>
>> >> >> >> >> >>> Thanks for flushing this out.
>> >> >> >> >> >>>
>> >> >> >> >> >>> My thoughts on the below:
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> -----Original Message-----
>> >> >> >> >> >>> From: Michael Starch <st...@umich.edu>
>> >> >> >> >> >>> Reply-To: <de...@oodt.apache.org>
>> >> >> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
>> >> >> >> >> >>> To: <de...@oodt.apache.org>
>> >> >> >> >> >>> Subject: Re: Extra Compiler Tools
>> >> >> >> >> >>>
>> >> >> >> >> >>> >I tried this approach. The plugin requires a path to the
>> >> >> >>"protoc"
>> >> >> >> >>tool
>> >> >> >> >> >>>and
>> >> >> >> >> >>> >thus a working installation. This is what prompted the
>> >> >> >>discussion.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Ah - no worries, what you could do is:
>> >> >> >> >> >>>
>> >> >> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
>> >> >> >> >> >>>
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >Running the plugin under a profile works.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Yep.
>> >> >> >> >> >>>
>> >> >> >> >> >>> > However, not running the plugin
>> >> >> >> >> >>> >causes compile errors in dependant code. Excluding this
>> >>code
>> >> >> >> >>except
>> >> >> >> >> >>> >within
>> >> >> >> >> >>> >the profile doesn't seem to work, and is considered by
>> >>some
>> >> >>to
>> >> >> >>be
>> >> >> >> >>bad
>> >> >> >> >> >>>form
>> >> >> >> >> >>> >because there is nothing inside the jar file that notes
>> >>which
>> >> >> >> >>profiles
>> >> >> >> >> >>> >were
>> >> >> >> >> >>> >used to compile.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Got it. Suggestion here would be:
>> >> >> >> >> >>>
>> >> >> >> >> >>> 2. create a new module, cas-resource-mesos, and inside of
>> >>that
>> >> >> >> >>module,
>> >> >> >> >> >>> take one of the following approaches, assuming the module
>> >>is
>> >> >> >> >>activated
>> >> >> >> >> >>> when -Pwith-mesos is enabled:
>> >> >> >> >> >>>
>> >> >> >> >> >>> 2a. Maven Antrun like so (in this old example):
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
>> >> >> >> >> >>>t
>> >> >> >> >> >>>o-
>> >> >> >> >> >>> maven2-build
>> >> >> >> >> >>>
>> >> >> >> >> >>> (pro: more flexibility in case protoc isn¹t there; to fail
>> >>on
>> >> >> >> >>error; to
>> >> >> >> >> >>> only compile if
>> >> >> >> >> >>> protoc is available
>> >> >> >> >> >>>
>> >> >> >> >> >>> 2b. Maven protobuf plugin
>> >> >> >> >> >>>
>> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >> >> >> >>>
>> >> >> >> >> >>> Here¹s how to enable a module with a profile:
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
>> >> >> >> >> >>>a
>> >> >> >> >> >>>ct
>> >> >> >> >> >>> ive-modules-by-profiles-in-maven/
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> It seems like that is a bad idea though, based on that
>> >> >> >>discussion.
>> >> >> >> >> >>>
>> >> >> >> >> >>> So, here¹s another option:
>> >> >> >> >> >>>
>> >> >> >> >> >>> 1. Inside of cas-resource (no special new module or
>> >>anything
>> >> >> >>else)
>> >> >> >> >> >>> 2. include some custom Ant magic via a build.xml file and
>> >>the
>> >> >> >>Maven
>> >> >> >> >> >>> AntRun plugin:
>> >> >> >> >> >>> 2a. test if protoc is on the system path, and if not,
>> >> >>download
>> >> >> >>it,
>> >> >> >> >> >>>e.g.,
>> >> >> >> >> >>> into the target directory (gets deleted on clean)
>> >> >> >> >> >>> 2b. call protoc and compile after 2a
>> >> >> >> >> >>>
>> >> >> >> >> >>> I would suggest this solution as I think it¹s the most
>> >>robust
>> >> >>and
>> >> >> >> >> >>>ensures
>> >> >> >> >> >>> we always have a cas-resource that includes mesos and
>> >>compiled
>> >> >> >> >> >>>correctly.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Cheers,
>> >> >> >> >> >>> Chris
>> >> >> >> >> >>>
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >Any ideas on how to continue?
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >Michael
>> >> >> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
>> >> >> >> >><ch...@gmail.com>
>> >> >> >> >> >>> >wrote:
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >> Hi Mike,
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Great discussion. It would be nice if there was
>> >> >> >> >> >>> >> a protoc Maven plugin:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Looks like there is. My suggestion:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos and
>> >> >> >> >> >>> >> then when activated;
>> >> >> >> >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
>> >> >> >> >> >>> >> in the resource manager
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Sound good?
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Cheers,
>> >> >> >> >> >>> >> Chris
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> ------------------------
>> >> >> >> >> >>> >> Chris Mattmann
>> >> >> >> >> >>> >> chris.mattmann@gmail.com
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> -----Original Message-----
>> >> >> >> >> >>> >> From: Michael Starch <st...@umich.edu>
>> >> >> >> >> >>> >> Reply-To: <de...@oodt.apache.org>
>> >> >> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
>> >> >> >> >> >>> >> To: <de...@oodt.apache.org>
>> >> >> >> >> >>> >> Subject: Extra Compiler Tools
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> >All,
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >I am trying to integrate apache-mesos with our
>> resource
>> >> >> >>manager.
>> >> >> >> >> >>> >>However,
>> >> >> >> >> >>> >> >mesos uses a technology called "protobuff" from Google
>> >>for
>> >> >> >> >> >>> >> >marshaling/unmarshaling data.
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >This requires running a tool called "protoc" to
>> >>generate a
>> >> >> >> >>source
>> >> >> >> >> >>>file
>> >> >> >> >> >>> >>in
>> >> >> >> >> >>> >> >java. What is the best way to integrate this step
>> into
>> >> >>our
>> >> >> >> >>build
>> >> >> >> >> >>> >>process?
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >Options I can conceive of:
>> >> >> >> >> >>> >> > -Check in generated java file
>> >> >> >> >> >>> >> > -Require "protoc" installation to build resource
>> >> >>manager
>> >> >> >> >> >>> >> > -Separate extra resource package into new module
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >None of these ideas are very clean.
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >Any other ideas? I tried setting up a profile to only
>> >> >> >>compile
>> >> >> >> >> >>>these
>> >> >> >> >> >>> >> >sources when selected, but that turned out not to
>> work.
>> >> >> >> >> >>> >> >
>> >> >> >> >> >>> >> >-Michael Starch
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> >>
>>
>>
>>
Re: Extra Compiler Tools
Posted by Michael Starch <st...@umich.edu>.
Sound fine, expect to see this on review board when I get it done and
passing my test.
Michael
On Nov 7, 2014 9:44 AM, "Chris Mattmann" <ch...@gmail.com> wrote:
> Yep I’d like to see the non changing
> core structs version, in a patch to
> evaluate it. You said you know how, so let’s see it
> and then go from there. We can reserve 0.9 to
> consider changing structs if the code becomes
> too unwieldily to maintain.
>
> Good?
>
> Cheers,
> Chris
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Michael Starch <st...@umich.edu>
> Reply-To: <de...@oodt.apache.org>
> Date: Friday, November 7, 2014 at 9:21 AM
> To: <de...@oodt.apache.org>
> Subject: Re: Extra Compiler Tools
>
> >Chris,
> >
> >Did you ever come to a conclusion on this?
> >
> >-Michael
> >
> >On Wed, Nov 5, 2014 at 7:14 PM, Chris Mattmann <ch...@gmail.com>
> >wrote:
> >
> >> OK, let me think about this tonight.
> >> Maybe we can figure this out tomorrow,
> >> I won’t hold this up longer than that.
> >>
> >> ------------------------
> >> Chris Mattmann
> >> chris.mattmann@gmail.com
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Michael Starch <st...@umich.edu>
> >> Reply-To: <de...@oodt.apache.org>
> >> Date: Wednesday, November 5, 2014 at 8:23 PM
> >> To: <de...@oodt.apache.org>
> >> Subject: Re: Extra Compiler Tools
> >>
> >> >According to the specs, any subclass that implements Serializable must
> >> >manually implement the serialization of the parents' members. I tested
> >> >this and it fails exactly as expected. The parent's members aren't
> >> >serialized.
> >> >
> >> >Also, JobInput is an interface.... so I would have no way of catching
> >>all
> >> >of the possible implementations that could come at me.
> >> >
> >> >Michael
> >> >On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
> >> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >> >
> >> >> Got it, Mike.
> >> >>
> >> >> Hmm, how about simply creating SerializableJobSpec and
> >> >> SerializableJob and SerializableJobInput and then making
> >> >> them sub-class their parents and implement Serializable.
> >> >> Then, use these classes in your Mesos implementation.
> >> >> That seems self-contained, doesn’t change core classes,
> >> >> and pretty easy, right?
> >> >>
> >> >> Cheers,
> >> >> Chris
> >> >>
> >> >>
> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >> Chris Mattmann, Ph.D.
> >> >> Chief Architect
> >> >> Instrument Software and Science Data Systems Section (398)
> >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> >> Office: 168-519, Mailstop: 168-527
> >> >> Email: chris.a.mattmann@nasa.gov
> >> >> WWW: http://sunset.usc.edu/~mattmann/
> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >> Adjunct Associate Professor, Computer Science Department
> >> >> University of Southern California, Los Angeles, CA 90089 USA
> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> -----Original Message-----
> >> >> From: Michael Starch <st...@umich.edu>
> >> >> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> >> >> Date: Wednesday, November 5, 2014 at 8:00 PM
> >> >> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> >> >> Subject: Re: Extra Compiler Tools
> >> >>
> >> >> >I need to serialize a JobSpec and children (Job and JobInput) to a
> >> >>byte[].
> >> >> >Java can do this automatically by marking all three as Serializable.
> >> >> >
> >> >> >The work around is to manually serialize to a private inner struct
> >>and
> >> >> >back
> >> >> >out again. The inner class will have members for each member in the
> >> >> >JobSpec and children. Java can the auto-serialize that without
> >> >>changing
> >> >> >the other three.
> >> >> >
> >> >> >It is ugly, and essentially a reimplementation of those three
> >> >> >classes....but it is entirely self-contained.
> >> >> >
> >> >> >Michael
> >> >> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" <ch...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> Hey Mike,
> >> >> >>
> >> >> >> Hmm, what’s the work around just so I know
> >> >> >> what we’re trading against?
> >> >> >>
> >> >> >> Cheers,
> >> >> >> Chris
> >> >> >>
> >> >> >> ------------------------
> >> >> >> Chris Mattmann
> >> >> >> chris.mattmann@gmail.com
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> -----Original Message-----
> >> >> >> From: Michael Starch <st...@umich.edu>
> >> >> >> Reply-To: <de...@oodt.apache.org>
> >> >> >> Date: Wednesday, November 5, 2014 at 6:31 PM
> >> >> >> To: <de...@oodt.apache.org>
> >> >> >> Subject: Re: Extra Compiler Tools
> >> >> >>
> >> >> >> >That is basically what I did. Regardless, protobuff proves to be
> >> >> >>overkill.
> >> >> >> >
> >> >> >> >If I mark those classes as serializable, the correct solution is
> >>2
> >> >> >>lines
> >> >> >> >of
> >> >> >> >code. (protobuff was like 20). Wrote a test case, and it works
> >> >> >> >perfectly.
> >> >> >> >
> >> >> >> >If I cannot make JobSpec Job and JonInput implement Serializable
> >> >>then
> >> >> >>the
> >> >> >> >work around is simple too.
> >> >> >> >
> >> >> >> >What do you think? Should I mark them as Serializable, or use a
> >> >> >> >work-around. Either is a better solution than protobuff.
> >> >> >> >
> >> >> >> >Michael
> >> >> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann"
> >><ch...@gmail.com>
> >> >> >> wrote:
> >> >> >> >
> >> >> >> >> Mike, have you looked at this yet?
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >>
> >>
> http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
> >> >> >> >>co
> >> >> >> >> l-buffers-from-maven/
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> I’m going to play with it tonight and see if
> >> >> >> >> I can help here. Do you have some files I can test
> >> >> >> >> with? Can you attach them to JIRA or dropbox them to me
> >> >> >> >> so I can scope?
> >> >> >> >>
> >> >> >> >> Cheers,
> >> >> >> >> Chris
> >> >> >> >>
> >> >> >> >> ------------------------
> >> >> >> >> Chris Mattmann
> >> >> >> >> chris.mattmann@gmail.com
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> -----Original Message-----
> >> >> >> >> From: Michael Starch <st...@umich.edu>
> >> >> >> >> Reply-To: <de...@oodt.apache.org>
> >> >> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM
> >> >> >> >> To: <de...@oodt.apache.org>
> >> >> >> >> Subject: Re: Extra Compiler Tools
> >> >> >> >>
> >> >> >> >> >Ok....time for an audible. Protoc needs to be built from
> >> >>source, no
> >> >> >> >> >binary
> >> >> >> >> >distributions available. Thus I am going to purge
> >>proto-buffers
> >> >> >>from
> >> >> >> >>the
> >> >> >> >> >new code and be done with it.
> >> >> >> >> >
> >> >> >> >> >Any problem making the following classes/interfaces implement
> >> >> >> >> >java.io.Serializable:
> >> >> >> >> >
> >> >> >> >> >JobSpec
> >> >> >> >> >Job
> >> >> >> >> >JobInput
> >> >> >> >> >
> >> >> >> >> >Doing so would allow apache and native java serialization and
> >> >>thus
> >> >> >>we
> >> >> >> >> >wouldn't need something like proto-buffers.
> >> >> >> >> >
> >> >> >> >> >-Michael
> >> >> >> >> >Thanks Mike +1
> >> >> >> >> >
> >> >> >> >> >------------------------
> >> >> >> >> >Chris Mattmann
> >> >> >> >> >chris.mattmann@gmail.com
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >-----Original Message-----
> >> >> >> >> >From: Michael Starch <st...@umich.edu>
> >> >> >> >> >Reply-To: <de...@oodt.apache.org>
> >> >> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
> >> >> >> >> >To: <de...@oodt.apache.org>
> >> >> >> >> >Subject: Re: Extra Compiler Tools
> >> >> >> >> >
> >> >> >> >> >>Looks like you followed the same reasoning chain that I did.
> >> >>Yes,
> >> >> >>I
> >> >> >> >>came
> >> >> >> >> >>to the same conclusion that ant-build was best.
> >> >> >> >> >>
> >> >> >> >> >>I wasn't sure how to download protoc, but you just answered
> >> >> >> >>that....so I
> >> >> >> >> >>think this is a great solution!
> >> >> >> >> >>
> >> >> >> >> >>Thanks,
> >> >> >> >> >>
> >> >> >> >> >>Michael
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
> >> >> >> >> >><ch...@gmail.com>
> >> >> >> >> >>wrote:
> >> >> >> >> >>
> >> >> >> >> >>> Hi Mike,
> >> >> >> >> >>>
> >> >> >> >> >>> Thanks for flushing this out.
> >> >> >> >> >>>
> >> >> >> >> >>> My thoughts on the below:
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>> -----Original Message-----
> >> >> >> >> >>> From: Michael Starch <st...@umich.edu>
> >> >> >> >> >>> Reply-To: <de...@oodt.apache.org>
> >> >> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
> >> >> >> >> >>> To: <de...@oodt.apache.org>
> >> >> >> >> >>> Subject: Re: Extra Compiler Tools
> >> >> >> >> >>>
> >> >> >> >> >>> >I tried this approach. The plugin requires a path to the
> >> >> >>"protoc"
> >> >> >> >>tool
> >> >> >> >> >>>and
> >> >> >> >> >>> >thus a working installation. This is what prompted the
> >> >> >>discussion.
> >> >> >> >> >>>
> >> >> >> >> >>> Ah - no worries, what you could do is:
> >> >> >> >> >>>
> >> >> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
> >> >> >> >> >>>
> >> >> >> >> >>> >
> >> >> >> >> >>> >Running the plugin under a profile works.
> >> >> >> >> >>>
> >> >> >> >> >>> Yep.
> >> >> >> >> >>>
> >> >> >> >> >>> > However, not running the plugin
> >> >> >> >> >>> >causes compile errors in dependant code. Excluding this
> >>code
> >> >> >> >>except
> >> >> >> >> >>> >within
> >> >> >> >> >>> >the profile doesn't seem to work, and is considered by
> >>some
> >> >>to
> >> >> >>be
> >> >> >> >>bad
> >> >> >> >> >>>form
> >> >> >> >> >>> >because there is nothing inside the jar file that notes
> >>which
> >> >> >> >>profiles
> >> >> >> >> >>> >were
> >> >> >> >> >>> >used to compile.
> >> >> >> >> >>>
> >> >> >> >> >>> Got it. Suggestion here would be:
> >> >> >> >> >>>
> >> >> >> >> >>> 2. create a new module, cas-resource-mesos, and inside of
> >>that
> >> >> >> >>module,
> >> >> >> >> >>> take one of the following approaches, assuming the module
> >>is
> >> >> >> >>activated
> >> >> >> >> >>> when -Pwith-mesos is enabled:
> >> >> >> >> >>>
> >> >> >> >> >>> 2a. Maven Antrun like so (in this old example):
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
> >> >> >> >> >>>t
> >> >> >> >> >>>o-
> >> >> >> >> >>> maven2-build
> >> >> >> >> >>>
> >> >> >> >> >>> (pro: more flexibility in case protoc isn¹t there; to fail
> >>on
> >> >> >> >>error; to
> >> >> >> >> >>> only compile if
> >> >> >> >> >>> protoc is available
> >> >> >> >> >>>
> >> >> >> >> >>> 2b. Maven protobuf plugin
> >> >> >> >> >>>
> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >> >> >> >> >>>
> >> >> >> >> >>> Here¹s how to enable a module with a profile:
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
> >> >> >> >> >>>a
> >> >> >> >> >>>ct
> >> >> >> >> >>> ive-modules-by-profiles-in-maven/
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>> It seems like that is a bad idea though, based on that
> >> >> >>discussion.
> >> >> >> >> >>>
> >> >> >> >> >>> So, here¹s another option:
> >> >> >> >> >>>
> >> >> >> >> >>> 1. Inside of cas-resource (no special new module or
> >>anything
> >> >> >>else)
> >> >> >> >> >>> 2. include some custom Ant magic via a build.xml file and
> >>the
> >> >> >>Maven
> >> >> >> >> >>> AntRun plugin:
> >> >> >> >> >>> 2a. test if protoc is on the system path, and if not,
> >> >>download
> >> >> >>it,
> >> >> >> >> >>>e.g.,
> >> >> >> >> >>> into the target directory (gets deleted on clean)
> >> >> >> >> >>> 2b. call protoc and compile after 2a
> >> >> >> >> >>>
> >> >> >> >> >>> I would suggest this solution as I think it¹s the most
> >>robust
> >> >>and
> >> >> >> >> >>>ensures
> >> >> >> >> >>> we always have a cas-resource that includes mesos and
> >>compiled
> >> >> >> >> >>>correctly.
> >> >> >> >> >>>
> >> >> >> >> >>> Cheers,
> >> >> >> >> >>> Chris
> >> >> >> >> >>>
> >> >> >> >> >>> >
> >> >> >> >> >>> >Any ideas on how to continue?
> >> >> >> >> >>> >
> >> >> >> >> >>> >Michael
> >> >> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
> >> >> >> >><ch...@gmail.com>
> >> >> >> >> >>> >wrote:
> >> >> >> >> >>> >
> >> >> >> >> >>> >> Hi Mike,
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> Great discussion. It would be nice if there was
> >> >> >> >> >>> >> a protoc Maven plugin:
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> Looks like there is. My suggestion:
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos and
> >> >> >> >> >>> >> then when activated;
> >> >> >> >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
> >> >> >> >> >>> >> in the resource manager
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> Sound good?
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> Cheers,
> >> >> >> >> >>> >> Chris
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> ------------------------
> >> >> >> >> >>> >> Chris Mattmann
> >> >> >> >> >>> >> chris.mattmann@gmail.com
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> -----Original Message-----
> >> >> >> >> >>> >> From: Michael Starch <st...@umich.edu>
> >> >> >> >> >>> >> Reply-To: <de...@oodt.apache.org>
> >> >> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
> >> >> >> >> >>> >> To: <de...@oodt.apache.org>
> >> >> >> >> >>> >> Subject: Extra Compiler Tools
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> >All,
> >> >> >> >> >>> >> >
> >> >> >> >> >>> >> >I am trying to integrate apache-mesos with our resource
> >> >> >>manager.
> >> >> >> >> >>> >>However,
> >> >> >> >> >>> >> >mesos uses a technology called "protobuff" from Google
> >>for
> >> >> >> >> >>> >> >marshaling/unmarshaling data.
> >> >> >> >> >>> >> >
> >> >> >> >> >>> >> >This requires running a tool called "protoc" to
> >>generate a
> >> >> >> >>source
> >> >> >> >> >>>file
> >> >> >> >> >>> >>in
> >> >> >> >> >>> >> >java. What is the best way to integrate this step into
> >> >>our
> >> >> >> >>build
> >> >> >> >> >>> >>process?
> >> >> >> >> >>> >> >
> >> >> >> >> >>> >> >Options I can conceive of:
> >> >> >> >> >>> >> > -Check in generated java file
> >> >> >> >> >>> >> > -Require "protoc" installation to build resource
> >> >>manager
> >> >> >> >> >>> >> > -Separate extra resource package into new module
> >> >> >> >> >>> >> >
> >> >> >> >> >>> >> >None of these ideas are very clean.
> >> >> >> >> >>> >> >
> >> >> >> >> >>> >> >Any other ideas? I tried setting up a profile to only
> >> >> >>compile
> >> >> >> >> >>>these
> >> >> >> >> >>> >> >sources when selected, but that turned out not to work.
> >> >> >> >> >>> >> >
> >> >> >> >> >>> >> >-Michael Starch
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
>
>
>
Re: Extra Compiler Tools
Posted by Chris Mattmann <ch...@gmail.com>.
Yep I’d like to see the non changing
core structs version, in a patch to
evaluate it. You said you know how, so let’s see it
and then go from there. We can reserve 0.9 to
consider changing structs if the code becomes
too unwieldily to maintain.
Good?
Cheers,
Chris
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: <de...@oodt.apache.org>
Date: Friday, November 7, 2014 at 9:21 AM
To: <de...@oodt.apache.org>
Subject: Re: Extra Compiler Tools
>Chris,
>
>Did you ever come to a conclusion on this?
>
>-Michael
>
>On Wed, Nov 5, 2014 at 7:14 PM, Chris Mattmann <ch...@gmail.com>
>wrote:
>
>> OK, let me think about this tonight.
>> Maybe we can figure this out tomorrow,
>> I won’t hold this up longer than that.
>>
>> ------------------------
>> Chris Mattmann
>> chris.mattmann@gmail.com
>>
>>
>>
>>
>> -----Original Message-----
>> From: Michael Starch <st...@umich.edu>
>> Reply-To: <de...@oodt.apache.org>
>> Date: Wednesday, November 5, 2014 at 8:23 PM
>> To: <de...@oodt.apache.org>
>> Subject: Re: Extra Compiler Tools
>>
>> >According to the specs, any subclass that implements Serializable must
>> >manually implement the serialization of the parents' members. I tested
>> >this and it fails exactly as expected. The parent's members aren't
>> >serialized.
>> >
>> >Also, JobInput is an interface.... so I would have no way of catching
>>all
>> >of the possible implementations that could come at me.
>> >
>> >Michael
>> >On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
>> >chris.a.mattmann@jpl.nasa.gov> wrote:
>> >
>> >> Got it, Mike.
>> >>
>> >> Hmm, how about simply creating SerializableJobSpec and
>> >> SerializableJob and SerializableJobInput and then making
>> >> them sub-class their parents and implement Serializable.
>> >> Then, use these classes in your Mesos implementation.
>> >> That seems self-contained, doesn’t change core classes,
>> >> and pretty easy, right?
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >>
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Chris Mattmann, Ph.D.
>> >> Chief Architect
>> >> Instrument Software and Science Data Systems Section (398)
>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> Office: 168-519, Mailstop: 168-527
>> >> Email: chris.a.mattmann@nasa.gov
>> >> WWW: http://sunset.usc.edu/~mattmann/
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Adjunct Associate Professor, Computer Science Department
>> >> University of Southern California, Los Angeles, CA 90089 USA
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Michael Starch <st...@umich.edu>
>> >> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> >> Date: Wednesday, November 5, 2014 at 8:00 PM
>> >> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> >> Subject: Re: Extra Compiler Tools
>> >>
>> >> >I need to serialize a JobSpec and children (Job and JobInput) to a
>> >>byte[].
>> >> >Java can do this automatically by marking all three as Serializable.
>> >> >
>> >> >The work around is to manually serialize to a private inner struct
>>and
>> >> >back
>> >> >out again. The inner class will have members for each member in the
>> >> >JobSpec and children. Java can the auto-serialize that without
>> >>changing
>> >> >the other three.
>> >> >
>> >> >It is ugly, and essentially a reimplementation of those three
>> >> >classes....but it is entirely self-contained.
>> >> >
>> >> >Michael
>> >> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" <ch...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Hey Mike,
>> >> >>
>> >> >> Hmm, what’s the work around just so I know
>> >> >> what we’re trading against?
>> >> >>
>> >> >> Cheers,
>> >> >> Chris
>> >> >>
>> >> >> ------------------------
>> >> >> Chris Mattmann
>> >> >> chris.mattmann@gmail.com
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Michael Starch <st...@umich.edu>
>> >> >> Reply-To: <de...@oodt.apache.org>
>> >> >> Date: Wednesday, November 5, 2014 at 6:31 PM
>> >> >> To: <de...@oodt.apache.org>
>> >> >> Subject: Re: Extra Compiler Tools
>> >> >>
>> >> >> >That is basically what I did. Regardless, protobuff proves to be
>> >> >>overkill.
>> >> >> >
>> >> >> >If I mark those classes as serializable, the correct solution is
>>2
>> >> >>lines
>> >> >> >of
>> >> >> >code. (protobuff was like 20). Wrote a test case, and it works
>> >> >> >perfectly.
>> >> >> >
>> >> >> >If I cannot make JobSpec Job and JonInput implement Serializable
>> >>then
>> >> >>the
>> >> >> >work around is simple too.
>> >> >> >
>> >> >> >What do you think? Should I mark them as Serializable, or use a
>> >> >> >work-around. Either is a better solution than protobuff.
>> >> >> >
>> >> >> >Michael
>> >> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann"
>><ch...@gmail.com>
>> >> >> wrote:
>> >> >> >
>> >> >> >> Mike, have you looked at this yet?
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
>> >> >> >>co
>> >> >> >> l-buffers-from-maven/
>> >> >> >>
>> >> >> >>
>> >> >> >> I’m going to play with it tonight and see if
>> >> >> >> I can help here. Do you have some files I can test
>> >> >> >> with? Can you attach them to JIRA or dropbox them to me
>> >> >> >> so I can scope?
>> >> >> >>
>> >> >> >> Cheers,
>> >> >> >> Chris
>> >> >> >>
>> >> >> >> ------------------------
>> >> >> >> Chris Mattmann
>> >> >> >> chris.mattmann@gmail.com
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> -----Original Message-----
>> >> >> >> From: Michael Starch <st...@umich.edu>
>> >> >> >> Reply-To: <de...@oodt.apache.org>
>> >> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM
>> >> >> >> To: <de...@oodt.apache.org>
>> >> >> >> Subject: Re: Extra Compiler Tools
>> >> >> >>
>> >> >> >> >Ok....time for an audible. Protoc needs to be built from
>> >>source, no
>> >> >> >> >binary
>> >> >> >> >distributions available. Thus I am going to purge
>>proto-buffers
>> >> >>from
>> >> >> >>the
>> >> >> >> >new code and be done with it.
>> >> >> >> >
>> >> >> >> >Any problem making the following classes/interfaces implement
>> >> >> >> >java.io.Serializable:
>> >> >> >> >
>> >> >> >> >JobSpec
>> >> >> >> >Job
>> >> >> >> >JobInput
>> >> >> >> >
>> >> >> >> >Doing so would allow apache and native java serialization and
>> >>thus
>> >> >>we
>> >> >> >> >wouldn't need something like proto-buffers.
>> >> >> >> >
>> >> >> >> >-Michael
>> >> >> >> >Thanks Mike +1
>> >> >> >> >
>> >> >> >> >------------------------
>> >> >> >> >Chris Mattmann
>> >> >> >> >chris.mattmann@gmail.com
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >-----Original Message-----
>> >> >> >> >From: Michael Starch <st...@umich.edu>
>> >> >> >> >Reply-To: <de...@oodt.apache.org>
>> >> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
>> >> >> >> >To: <de...@oodt.apache.org>
>> >> >> >> >Subject: Re: Extra Compiler Tools
>> >> >> >> >
>> >> >> >> >>Looks like you followed the same reasoning chain that I did.
>> >>Yes,
>> >> >>I
>> >> >> >>came
>> >> >> >> >>to the same conclusion that ant-build was best.
>> >> >> >> >>
>> >> >> >> >>I wasn't sure how to download protoc, but you just answered
>> >> >> >>that....so I
>> >> >> >> >>think this is a great solution!
>> >> >> >> >>
>> >> >> >> >>Thanks,
>> >> >> >> >>
>> >> >> >> >>Michael
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
>> >> >> >> >><ch...@gmail.com>
>> >> >> >> >>wrote:
>> >> >> >> >>
>> >> >> >> >>> Hi Mike,
>> >> >> >> >>>
>> >> >> >> >>> Thanks for flushing this out.
>> >> >> >> >>>
>> >> >> >> >>> My thoughts on the below:
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>> -----Original Message-----
>> >> >> >> >>> From: Michael Starch <st...@umich.edu>
>> >> >> >> >>> Reply-To: <de...@oodt.apache.org>
>> >> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
>> >> >> >> >>> To: <de...@oodt.apache.org>
>> >> >> >> >>> Subject: Re: Extra Compiler Tools
>> >> >> >> >>>
>> >> >> >> >>> >I tried this approach. The plugin requires a path to the
>> >> >>"protoc"
>> >> >> >>tool
>> >> >> >> >>>and
>> >> >> >> >>> >thus a working installation. This is what prompted the
>> >> >>discussion.
>> >> >> >> >>>
>> >> >> >> >>> Ah - no worries, what you could do is:
>> >> >> >> >>>
>> >> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
>> >> >> >> >>>
>> >> >> >> >>> >
>> >> >> >> >>> >Running the plugin under a profile works.
>> >> >> >> >>>
>> >> >> >> >>> Yep.
>> >> >> >> >>>
>> >> >> >> >>> > However, not running the plugin
>> >> >> >> >>> >causes compile errors in dependant code. Excluding this
>>code
>> >> >> >>except
>> >> >> >> >>> >within
>> >> >> >> >>> >the profile doesn't seem to work, and is considered by
>>some
>> >>to
>> >> >>be
>> >> >> >>bad
>> >> >> >> >>>form
>> >> >> >> >>> >because there is nothing inside the jar file that notes
>>which
>> >> >> >>profiles
>> >> >> >> >>> >were
>> >> >> >> >>> >used to compile.
>> >> >> >> >>>
>> >> >> >> >>> Got it. Suggestion here would be:
>> >> >> >> >>>
>> >> >> >> >>> 2. create a new module, cas-resource-mesos, and inside of
>>that
>> >> >> >>module,
>> >> >> >> >>> take one of the following approaches, assuming the module
>>is
>> >> >> >>activated
>> >> >> >> >>> when -Pwith-mesos is enabled:
>> >> >> >> >>>
>> >> >> >> >>> 2a. Maven Antrun like so (in this old example):
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >>
>> >> >>
>> >>
>> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
>> >> >> >> >>>t
>> >> >> >> >>>o-
>> >> >> >> >>> maven2-build
>> >> >> >> >>>
>> >> >> >> >>> (pro: more flexibility in case protoc isn¹t there; to fail
>>on
>> >> >> >>error; to
>> >> >> >> >>> only compile if
>> >> >> >> >>> protoc is available
>> >> >> >> >>>
>> >> >> >> >>> 2b. Maven protobuf plugin
>> >> >> >> >>>
>>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >> >> >>>
>> >> >> >> >>> Here¹s how to enable a module with a profile:
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >>
>> >> >>
>> >>
>> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
>> >> >> >> >>>a
>> >> >> >> >>>ct
>> >> >> >> >>> ive-modules-by-profiles-in-maven/
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>> It seems like that is a bad idea though, based on that
>> >> >>discussion.
>> >> >> >> >>>
>> >> >> >> >>> So, here¹s another option:
>> >> >> >> >>>
>> >> >> >> >>> 1. Inside of cas-resource (no special new module or
>>anything
>> >> >>else)
>> >> >> >> >>> 2. include some custom Ant magic via a build.xml file and
>>the
>> >> >>Maven
>> >> >> >> >>> AntRun plugin:
>> >> >> >> >>> 2a. test if protoc is on the system path, and if not,
>> >>download
>> >> >>it,
>> >> >> >> >>>e.g.,
>> >> >> >> >>> into the target directory (gets deleted on clean)
>> >> >> >> >>> 2b. call protoc and compile after 2a
>> >> >> >> >>>
>> >> >> >> >>> I would suggest this solution as I think it¹s the most
>>robust
>> >>and
>> >> >> >> >>>ensures
>> >> >> >> >>> we always have a cas-resource that includes mesos and
>>compiled
>> >> >> >> >>>correctly.
>> >> >> >> >>>
>> >> >> >> >>> Cheers,
>> >> >> >> >>> Chris
>> >> >> >> >>>
>> >> >> >> >>> >
>> >> >> >> >>> >Any ideas on how to continue?
>> >> >> >> >>> >
>> >> >> >> >>> >Michael
>> >> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
>> >> >> >><ch...@gmail.com>
>> >> >> >> >>> >wrote:
>> >> >> >> >>> >
>> >> >> >> >>> >> Hi Mike,
>> >> >> >> >>> >>
>> >> >> >> >>> >> Great discussion. It would be nice if there was
>> >> >> >> >>> >> a protoc Maven plugin:
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >> Looks like there is. My suggestion:
>> >> >> >> >>> >>
>> >> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos and
>> >> >> >> >>> >> then when activated;
>> >> >> >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
>> >> >> >> >>> >> in the resource manager
>> >> >> >> >>> >>
>> >> >> >> >>> >> Sound good?
>> >> >> >> >>> >>
>> >> >> >> >>> >> Cheers,
>> >> >> >> >>> >> Chris
>> >> >> >> >>> >>
>> >> >> >> >>> >> ------------------------
>> >> >> >> >>> >> Chris Mattmann
>> >> >> >> >>> >> chris.mattmann@gmail.com
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >> -----Original Message-----
>> >> >> >> >>> >> From: Michael Starch <st...@umich.edu>
>> >> >> >> >>> >> Reply-To: <de...@oodt.apache.org>
>> >> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
>> >> >> >> >>> >> To: <de...@oodt.apache.org>
>> >> >> >> >>> >> Subject: Extra Compiler Tools
>> >> >> >> >>> >>
>> >> >> >> >>> >> >All,
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >I am trying to integrate apache-mesos with our resource
>> >> >>manager.
>> >> >> >> >>> >>However,
>> >> >> >> >>> >> >mesos uses a technology called "protobuff" from Google
>>for
>> >> >> >> >>> >> >marshaling/unmarshaling data.
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >This requires running a tool called "protoc" to
>>generate a
>> >> >> >>source
>> >> >> >> >>>file
>> >> >> >> >>> >>in
>> >> >> >> >>> >> >java. What is the best way to integrate this step into
>> >>our
>> >> >> >>build
>> >> >> >> >>> >>process?
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >Options I can conceive of:
>> >> >> >> >>> >> > -Check in generated java file
>> >> >> >> >>> >> > -Require "protoc" installation to build resource
>> >>manager
>> >> >> >> >>> >> > -Separate extra resource package into new module
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >None of these ideas are very clean.
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >Any other ideas? I tried setting up a profile to only
>> >> >>compile
>> >> >> >> >>>these
>> >> >> >> >>> >> >sources when selected, but that turned out not to work.
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >-Michael Starch
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>
>>
Re: Extra Compiler Tools
Posted by Michael Starch <st...@umich.edu>.
Chris,
Did you ever come to a conclusion on this?
-Michael
On Wed, Nov 5, 2014 at 7:14 PM, Chris Mattmann <ch...@gmail.com>
wrote:
> OK, let me think about this tonight.
> Maybe we can figure this out tomorrow,
> I won’t hold this up longer than that.
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Michael Starch <st...@umich.edu>
> Reply-To: <de...@oodt.apache.org>
> Date: Wednesday, November 5, 2014 at 8:23 PM
> To: <de...@oodt.apache.org>
> Subject: Re: Extra Compiler Tools
>
> >According to the specs, any subclass that implements Serializable must
> >manually implement the serialization of the parents' members. I tested
> >this and it fails exactly as expected. The parent's members aren't
> >serialized.
> >
> >Also, JobInput is an interface.... so I would have no way of catching all
> >of the possible implementations that could come at me.
> >
> >Michael
> >On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> Got it, Mike.
> >>
> >> Hmm, how about simply creating SerializableJobSpec and
> >> SerializableJob and SerializableJobInput and then making
> >> them sub-class their parents and implement Serializable.
> >> Then, use these classes in your Mesos implementation.
> >> That seems self-contained, doesn’t change core classes,
> >> and pretty easy, right?
> >>
> >> Cheers,
> >> Chris
> >>
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Chief Architect
> >> Instrument Software and Science Data Systems Section (398)
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 168-519, Mailstop: 168-527
> >> Email: chris.a.mattmann@nasa.gov
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Associate Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Michael Starch <st...@umich.edu>
> >> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> >> Date: Wednesday, November 5, 2014 at 8:00 PM
> >> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> >> Subject: Re: Extra Compiler Tools
> >>
> >> >I need to serialize a JobSpec and children (Job and JobInput) to a
> >>byte[].
> >> >Java can do this automatically by marking all three as Serializable.
> >> >
> >> >The work around is to manually serialize to a private inner struct and
> >> >back
> >> >out again. The inner class will have members for each member in the
> >> >JobSpec and children. Java can the auto-serialize that without
> >>changing
> >> >the other three.
> >> >
> >> >It is ugly, and essentially a reimplementation of those three
> >> >classes....but it is entirely self-contained.
> >> >
> >> >Michael
> >> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" <ch...@gmail.com>
> >> wrote:
> >> >
> >> >> Hey Mike,
> >> >>
> >> >> Hmm, what’s the work around just so I know
> >> >> what we’re trading against?
> >> >>
> >> >> Cheers,
> >> >> Chris
> >> >>
> >> >> ------------------------
> >> >> Chris Mattmann
> >> >> chris.mattmann@gmail.com
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> -----Original Message-----
> >> >> From: Michael Starch <st...@umich.edu>
> >> >> Reply-To: <de...@oodt.apache.org>
> >> >> Date: Wednesday, November 5, 2014 at 6:31 PM
> >> >> To: <de...@oodt.apache.org>
> >> >> Subject: Re: Extra Compiler Tools
> >> >>
> >> >> >That is basically what I did. Regardless, protobuff proves to be
> >> >>overkill.
> >> >> >
> >> >> >If I mark those classes as serializable, the correct solution is 2
> >> >>lines
> >> >> >of
> >> >> >code. (protobuff was like 20). Wrote a test case, and it works
> >> >> >perfectly.
> >> >> >
> >> >> >If I cannot make JobSpec Job and JonInput implement Serializable
> >>then
> >> >>the
> >> >> >work around is simple too.
> >> >> >
> >> >> >What do you think? Should I mark them as Serializable, or use a
> >> >> >work-around. Either is a better solution than protobuff.
> >> >> >
> >> >> >Michael
> >> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann" <ch...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> Mike, have you looked at this yet?
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >>
> >>
> http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
> >> >> >>co
> >> >> >> l-buffers-from-maven/
> >> >> >>
> >> >> >>
> >> >> >> I’m going to play with it tonight and see if
> >> >> >> I can help here. Do you have some files I can test
> >> >> >> with? Can you attach them to JIRA or dropbox them to me
> >> >> >> so I can scope?
> >> >> >>
> >> >> >> Cheers,
> >> >> >> Chris
> >> >> >>
> >> >> >> ------------------------
> >> >> >> Chris Mattmann
> >> >> >> chris.mattmann@gmail.com
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> -----Original Message-----
> >> >> >> From: Michael Starch <st...@umich.edu>
> >> >> >> Reply-To: <de...@oodt.apache.org>
> >> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM
> >> >> >> To: <de...@oodt.apache.org>
> >> >> >> Subject: Re: Extra Compiler Tools
> >> >> >>
> >> >> >> >Ok....time for an audible. Protoc needs to be built from
> >>source, no
> >> >> >> >binary
> >> >> >> >distributions available. Thus I am going to purge proto-buffers
> >> >>from
> >> >> >>the
> >> >> >> >new code and be done with it.
> >> >> >> >
> >> >> >> >Any problem making the following classes/interfaces implement
> >> >> >> >java.io.Serializable:
> >> >> >> >
> >> >> >> >JobSpec
> >> >> >> >Job
> >> >> >> >JobInput
> >> >> >> >
> >> >> >> >Doing so would allow apache and native java serialization and
> >>thus
> >> >>we
> >> >> >> >wouldn't need something like proto-buffers.
> >> >> >> >
> >> >> >> >-Michael
> >> >> >> >Thanks Mike +1
> >> >> >> >
> >> >> >> >------------------------
> >> >> >> >Chris Mattmann
> >> >> >> >chris.mattmann@gmail.com
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >-----Original Message-----
> >> >> >> >From: Michael Starch <st...@umich.edu>
> >> >> >> >Reply-To: <de...@oodt.apache.org>
> >> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
> >> >> >> >To: <de...@oodt.apache.org>
> >> >> >> >Subject: Re: Extra Compiler Tools
> >> >> >> >
> >> >> >> >>Looks like you followed the same reasoning chain that I did.
> >>Yes,
> >> >>I
> >> >> >>came
> >> >> >> >>to the same conclusion that ant-build was best.
> >> >> >> >>
> >> >> >> >>I wasn't sure how to download protoc, but you just answered
> >> >> >>that....so I
> >> >> >> >>think this is a great solution!
> >> >> >> >>
> >> >> >> >>Thanks,
> >> >> >> >>
> >> >> >> >>Michael
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
> >> >> >> >><ch...@gmail.com>
> >> >> >> >>wrote:
> >> >> >> >>
> >> >> >> >>> Hi Mike,
> >> >> >> >>>
> >> >> >> >>> Thanks for flushing this out.
> >> >> >> >>>
> >> >> >> >>> My thoughts on the below:
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> -----Original Message-----
> >> >> >> >>> From: Michael Starch <st...@umich.edu>
> >> >> >> >>> Reply-To: <de...@oodt.apache.org>
> >> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
> >> >> >> >>> To: <de...@oodt.apache.org>
> >> >> >> >>> Subject: Re: Extra Compiler Tools
> >> >> >> >>>
> >> >> >> >>> >I tried this approach. The plugin requires a path to the
> >> >>"protoc"
> >> >> >>tool
> >> >> >> >>>and
> >> >> >> >>> >thus a working installation. This is what prompted the
> >> >>discussion.
> >> >> >> >>>
> >> >> >> >>> Ah - no worries, what you could do is:
> >> >> >> >>>
> >> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
> >> >> >> >>>
> >> >> >> >>> >
> >> >> >> >>> >Running the plugin under a profile works.
> >> >> >> >>>
> >> >> >> >>> Yep.
> >> >> >> >>>
> >> >> >> >>> > However, not running the plugin
> >> >> >> >>> >causes compile errors in dependant code. Excluding this code
> >> >> >>except
> >> >> >> >>> >within
> >> >> >> >>> >the profile doesn't seem to work, and is considered by some
> >>to
> >> >>be
> >> >> >>bad
> >> >> >> >>>form
> >> >> >> >>> >because there is nothing inside the jar file that notes which
> >> >> >>profiles
> >> >> >> >>> >were
> >> >> >> >>> >used to compile.
> >> >> >> >>>
> >> >> >> >>> Got it. Suggestion here would be:
> >> >> >> >>>
> >> >> >> >>> 2. create a new module, cas-resource-mesos, and inside of that
> >> >> >>module,
> >> >> >> >>> take one of the following approaches, assuming the module is
> >> >> >>activated
> >> >> >> >>> when -Pwith-mesos is enabled:
> >> >> >> >>>
> >> >> >> >>> 2a. Maven Antrun like so (in this old example):
> >> >> >> >>>
> >> >> >> >>>
> >> >> >>
> >> >>
> >>
> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
> >> >> >> >>>t
> >> >> >> >>>o-
> >> >> >> >>> maven2-build
> >> >> >> >>>
> >> >> >> >>> (pro: more flexibility in case protoc isn¹t there; to fail on
> >> >> >>error; to
> >> >> >> >>> only compile if
> >> >> >> >>> protoc is available
> >> >> >> >>>
> >> >> >> >>> 2b. Maven protobuf plugin
> >> >> >> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >> >> >> >>>
> >> >> >> >>> Here¹s how to enable a module with a profile:
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >>
> >> >>
> >>
> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
> >> >> >> >>>a
> >> >> >> >>>ct
> >> >> >> >>> ive-modules-by-profiles-in-maven/
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> It seems like that is a bad idea though, based on that
> >> >>discussion.
> >> >> >> >>>
> >> >> >> >>> So, here¹s another option:
> >> >> >> >>>
> >> >> >> >>> 1. Inside of cas-resource (no special new module or anything
> >> >>else)
> >> >> >> >>> 2. include some custom Ant magic via a build.xml file and the
> >> >>Maven
> >> >> >> >>> AntRun plugin:
> >> >> >> >>> 2a. test if protoc is on the system path, and if not,
> >>download
> >> >>it,
> >> >> >> >>>e.g.,
> >> >> >> >>> into the target directory (gets deleted on clean)
> >> >> >> >>> 2b. call protoc and compile after 2a
> >> >> >> >>>
> >> >> >> >>> I would suggest this solution as I think it¹s the most robust
> >>and
> >> >> >> >>>ensures
> >> >> >> >>> we always have a cas-resource that includes mesos and compiled
> >> >> >> >>>correctly.
> >> >> >> >>>
> >> >> >> >>> Cheers,
> >> >> >> >>> Chris
> >> >> >> >>>
> >> >> >> >>> >
> >> >> >> >>> >Any ideas on how to continue?
> >> >> >> >>> >
> >> >> >> >>> >Michael
> >> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
> >> >> >><ch...@gmail.com>
> >> >> >> >>> >wrote:
> >> >> >> >>> >
> >> >> >> >>> >> Hi Mike,
> >> >> >> >>> >>
> >> >> >> >>> >> Great discussion. It would be nice if there was
> >> >> >> >>> >> a protoc Maven plugin:
> >> >> >> >>> >>
> >> >> >> >>> >>
> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >> Looks like there is. My suggestion:
> >> >> >> >>> >>
> >> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos and
> >> >> >> >>> >> then when activated;
> >> >> >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
> >> >> >> >>> >> in the resource manager
> >> >> >> >>> >>
> >> >> >> >>> >> Sound good?
> >> >> >> >>> >>
> >> >> >> >>> >> Cheers,
> >> >> >> >>> >> Chris
> >> >> >> >>> >>
> >> >> >> >>> >> ------------------------
> >> >> >> >>> >> Chris Mattmann
> >> >> >> >>> >> chris.mattmann@gmail.com
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >> -----Original Message-----
> >> >> >> >>> >> From: Michael Starch <st...@umich.edu>
> >> >> >> >>> >> Reply-To: <de...@oodt.apache.org>
> >> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
> >> >> >> >>> >> To: <de...@oodt.apache.org>
> >> >> >> >>> >> Subject: Extra Compiler Tools
> >> >> >> >>> >>
> >> >> >> >>> >> >All,
> >> >> >> >>> >> >
> >> >> >> >>> >> >I am trying to integrate apache-mesos with our resource
> >> >>manager.
> >> >> >> >>> >>However,
> >> >> >> >>> >> >mesos uses a technology called "protobuff" from Google for
> >> >> >> >>> >> >marshaling/unmarshaling data.
> >> >> >> >>> >> >
> >> >> >> >>> >> >This requires running a tool called "protoc" to generate a
> >> >> >>source
> >> >> >> >>>file
> >> >> >> >>> >>in
> >> >> >> >>> >> >java. What is the best way to integrate this step into
> >>our
> >> >> >>build
> >> >> >> >>> >>process?
> >> >> >> >>> >> >
> >> >> >> >>> >> >Options I can conceive of:
> >> >> >> >>> >> > -Check in generated java file
> >> >> >> >>> >> > -Require "protoc" installation to build resource
> >>manager
> >> >> >> >>> >> > -Separate extra resource package into new module
> >> >> >> >>> >> >
> >> >> >> >>> >> >None of these ideas are very clean.
> >> >> >> >>> >> >
> >> >> >> >>> >> >Any other ideas? I tried setting up a profile to only
> >> >>compile
> >> >> >> >>>these
> >> >> >> >>> >> >sources when selected, but that turned out not to work.
> >> >> >> >>> >> >
> >> >> >> >>> >> >-Michael Starch
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >> >>
> >>
> >>
>
>
>
Re: Extra Compiler Tools
Posted by Chris Mattmann <ch...@gmail.com>.
OK, let me think about this tonight.
Maybe we can figure this out tomorrow,
I won’t hold this up longer than that.
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: <de...@oodt.apache.org>
Date: Wednesday, November 5, 2014 at 8:23 PM
To: <de...@oodt.apache.org>
Subject: Re: Extra Compiler Tools
>According to the specs, any subclass that implements Serializable must
>manually implement the serialization of the parents' members. I tested
>this and it fails exactly as expected. The parent's members aren't
>serialized.
>
>Also, JobInput is an interface.... so I would have no way of catching all
>of the possible implementations that could come at me.
>
>Michael
>On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Got it, Mike.
>>
>> Hmm, how about simply creating SerializableJobSpec and
>> SerializableJob and SerializableJobInput and then making
>> them sub-class their parents and implement Serializable.
>> Then, use these classes in your Mesos implementation.
>> That seems self-contained, doesn’t change core classes,
>> and pretty easy, right?
>>
>> Cheers,
>> Chris
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Chief Architect
>> Instrument Software and Science Data Systems Section (398)
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 168-519, Mailstop: 168-527
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Associate Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Michael Starch <st...@umich.edu>
>> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> Date: Wednesday, November 5, 2014 at 8:00 PM
>> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> Subject: Re: Extra Compiler Tools
>>
>> >I need to serialize a JobSpec and children (Job and JobInput) to a
>>byte[].
>> >Java can do this automatically by marking all three as Serializable.
>> >
>> >The work around is to manually serialize to a private inner struct and
>> >back
>> >out again. The inner class will have members for each member in the
>> >JobSpec and children. Java can the auto-serialize that without
>>changing
>> >the other three.
>> >
>> >It is ugly, and essentially a reimplementation of those three
>> >classes....but it is entirely self-contained.
>> >
>> >Michael
>> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" <ch...@gmail.com>
>> wrote:
>> >
>> >> Hey Mike,
>> >>
>> >> Hmm, what’s the work around just so I know
>> >> what we’re trading against?
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> ------------------------
>> >> Chris Mattmann
>> >> chris.mattmann@gmail.com
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Michael Starch <st...@umich.edu>
>> >> Reply-To: <de...@oodt.apache.org>
>> >> Date: Wednesday, November 5, 2014 at 6:31 PM
>> >> To: <de...@oodt.apache.org>
>> >> Subject: Re: Extra Compiler Tools
>> >>
>> >> >That is basically what I did. Regardless, protobuff proves to be
>> >>overkill.
>> >> >
>> >> >If I mark those classes as serializable, the correct solution is 2
>> >>lines
>> >> >of
>> >> >code. (protobuff was like 20). Wrote a test case, and it works
>> >> >perfectly.
>> >> >
>> >> >If I cannot make JobSpec Job and JonInput implement Serializable
>>then
>> >>the
>> >> >work around is simple too.
>> >> >
>> >> >What do you think? Should I mark them as Serializable, or use a
>> >> >work-around. Either is a better solution than protobuff.
>> >> >
>> >> >Michael
>> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann" <ch...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Mike, have you looked at this yet?
>> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
>> >> >>co
>> >> >> l-buffers-from-maven/
>> >> >>
>> >> >>
>> >> >> I’m going to play with it tonight and see if
>> >> >> I can help here. Do you have some files I can test
>> >> >> with? Can you attach them to JIRA or dropbox them to me
>> >> >> so I can scope?
>> >> >>
>> >> >> Cheers,
>> >> >> Chris
>> >> >>
>> >> >> ------------------------
>> >> >> Chris Mattmann
>> >> >> chris.mattmann@gmail.com
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Michael Starch <st...@umich.edu>
>> >> >> Reply-To: <de...@oodt.apache.org>
>> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM
>> >> >> To: <de...@oodt.apache.org>
>> >> >> Subject: Re: Extra Compiler Tools
>> >> >>
>> >> >> >Ok....time for an audible. Protoc needs to be built from
>>source, no
>> >> >> >binary
>> >> >> >distributions available. Thus I am going to purge proto-buffers
>> >>from
>> >> >>the
>> >> >> >new code and be done with it.
>> >> >> >
>> >> >> >Any problem making the following classes/interfaces implement
>> >> >> >java.io.Serializable:
>> >> >> >
>> >> >> >JobSpec
>> >> >> >Job
>> >> >> >JobInput
>> >> >> >
>> >> >> >Doing so would allow apache and native java serialization and
>>thus
>> >>we
>> >> >> >wouldn't need something like proto-buffers.
>> >> >> >
>> >> >> >-Michael
>> >> >> >Thanks Mike +1
>> >> >> >
>> >> >> >------------------------
>> >> >> >Chris Mattmann
>> >> >> >chris.mattmann@gmail.com
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >-----Original Message-----
>> >> >> >From: Michael Starch <st...@umich.edu>
>> >> >> >Reply-To: <de...@oodt.apache.org>
>> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
>> >> >> >To: <de...@oodt.apache.org>
>> >> >> >Subject: Re: Extra Compiler Tools
>> >> >> >
>> >> >> >>Looks like you followed the same reasoning chain that I did.
>>Yes,
>> >>I
>> >> >>came
>> >> >> >>to the same conclusion that ant-build was best.
>> >> >> >>
>> >> >> >>I wasn't sure how to download protoc, but you just answered
>> >> >>that....so I
>> >> >> >>think this is a great solution!
>> >> >> >>
>> >> >> >>Thanks,
>> >> >> >>
>> >> >> >>Michael
>> >> >> >>
>> >> >> >>
>> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
>> >> >> >><ch...@gmail.com>
>> >> >> >>wrote:
>> >> >> >>
>> >> >> >>> Hi Mike,
>> >> >> >>>
>> >> >> >>> Thanks for flushing this out.
>> >> >> >>>
>> >> >> >>> My thoughts on the below:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> -----Original Message-----
>> >> >> >>> From: Michael Starch <st...@umich.edu>
>> >> >> >>> Reply-To: <de...@oodt.apache.org>
>> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
>> >> >> >>> To: <de...@oodt.apache.org>
>> >> >> >>> Subject: Re: Extra Compiler Tools
>> >> >> >>>
>> >> >> >>> >I tried this approach. The plugin requires a path to the
>> >>"protoc"
>> >> >>tool
>> >> >> >>>and
>> >> >> >>> >thus a working installation. This is what prompted the
>> >>discussion.
>> >> >> >>>
>> >> >> >>> Ah - no worries, what you could do is:
>> >> >> >>>
>> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
>> >> >> >>>
>> >> >> >>> >
>> >> >> >>> >Running the plugin under a profile works.
>> >> >> >>>
>> >> >> >>> Yep.
>> >> >> >>>
>> >> >> >>> > However, not running the plugin
>> >> >> >>> >causes compile errors in dependant code. Excluding this code
>> >> >>except
>> >> >> >>> >within
>> >> >> >>> >the profile doesn't seem to work, and is considered by some
>>to
>> >>be
>> >> >>bad
>> >> >> >>>form
>> >> >> >>> >because there is nothing inside the jar file that notes which
>> >> >>profiles
>> >> >> >>> >were
>> >> >> >>> >used to compile.
>> >> >> >>>
>> >> >> >>> Got it. Suggestion here would be:
>> >> >> >>>
>> >> >> >>> 2. create a new module, cas-resource-mesos, and inside of that
>> >> >>module,
>> >> >> >>> take one of the following approaches, assuming the module is
>> >> >>activated
>> >> >> >>> when -Pwith-mesos is enabled:
>> >> >> >>>
>> >> >> >>> 2a. Maven Antrun like so (in this old example):
>> >> >> >>>
>> >> >> >>>
>> >> >>
>> >>
>> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
>> >> >> >>>t
>> >> >> >>>o-
>> >> >> >>> maven2-build
>> >> >> >>>
>> >> >> >>> (pro: more flexibility in case protoc isn¹t there; to fail on
>> >> >>error; to
>> >> >> >>> only compile if
>> >> >> >>> protoc is available
>> >> >> >>>
>> >> >> >>> 2b. Maven protobuf plugin
>> >> >> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >> >>>
>> >> >> >>> Here¹s how to enable a module with a profile:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >>
>> >>
>> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
>> >> >> >>>a
>> >> >> >>>ct
>> >> >> >>> ive-modules-by-profiles-in-maven/
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> It seems like that is a bad idea though, based on that
>> >>discussion.
>> >> >> >>>
>> >> >> >>> So, here¹s another option:
>> >> >> >>>
>> >> >> >>> 1. Inside of cas-resource (no special new module or anything
>> >>else)
>> >> >> >>> 2. include some custom Ant magic via a build.xml file and the
>> >>Maven
>> >> >> >>> AntRun plugin:
>> >> >> >>> 2a. test if protoc is on the system path, and if not,
>>download
>> >>it,
>> >> >> >>>e.g.,
>> >> >> >>> into the target directory (gets deleted on clean)
>> >> >> >>> 2b. call protoc and compile after 2a
>> >> >> >>>
>> >> >> >>> I would suggest this solution as I think it¹s the most robust
>>and
>> >> >> >>>ensures
>> >> >> >>> we always have a cas-resource that includes mesos and compiled
>> >> >> >>>correctly.
>> >> >> >>>
>> >> >> >>> Cheers,
>> >> >> >>> Chris
>> >> >> >>>
>> >> >> >>> >
>> >> >> >>> >Any ideas on how to continue?
>> >> >> >>> >
>> >> >> >>> >Michael
>> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
>> >> >><ch...@gmail.com>
>> >> >> >>> >wrote:
>> >> >> >>> >
>> >> >> >>> >> Hi Mike,
>> >> >> >>> >>
>> >> >> >>> >> Great discussion. It would be nice if there was
>> >> >> >>> >> a protoc Maven plugin:
>> >> >> >>> >>
>> >> >> >>> >>
>>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> Looks like there is. My suggestion:
>> >> >> >>> >>
>> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos and
>> >> >> >>> >> then when activated;
>> >> >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
>> >> >> >>> >> in the resource manager
>> >> >> >>> >>
>> >> >> >>> >> Sound good?
>> >> >> >>> >>
>> >> >> >>> >> Cheers,
>> >> >> >>> >> Chris
>> >> >> >>> >>
>> >> >> >>> >> ------------------------
>> >> >> >>> >> Chris Mattmann
>> >> >> >>> >> chris.mattmann@gmail.com
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> -----Original Message-----
>> >> >> >>> >> From: Michael Starch <st...@umich.edu>
>> >> >> >>> >> Reply-To: <de...@oodt.apache.org>
>> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
>> >> >> >>> >> To: <de...@oodt.apache.org>
>> >> >> >>> >> Subject: Extra Compiler Tools
>> >> >> >>> >>
>> >> >> >>> >> >All,
>> >> >> >>> >> >
>> >> >> >>> >> >I am trying to integrate apache-mesos with our resource
>> >>manager.
>> >> >> >>> >>However,
>> >> >> >>> >> >mesos uses a technology called "protobuff" from Google for
>> >> >> >>> >> >marshaling/unmarshaling data.
>> >> >> >>> >> >
>> >> >> >>> >> >This requires running a tool called "protoc" to generate a
>> >> >>source
>> >> >> >>>file
>> >> >> >>> >>in
>> >> >> >>> >> >java. What is the best way to integrate this step into
>>our
>> >> >>build
>> >> >> >>> >>process?
>> >> >> >>> >> >
>> >> >> >>> >> >Options I can conceive of:
>> >> >> >>> >> > -Check in generated java file
>> >> >> >>> >> > -Require "protoc" installation to build resource
>>manager
>> >> >> >>> >> > -Separate extra resource package into new module
>> >> >> >>> >> >
>> >> >> >>> >> >None of these ideas are very clean.
>> >> >> >>> >> >
>> >> >> >>> >> >Any other ideas? I tried setting up a profile to only
>> >>compile
>> >> >> >>>these
>> >> >> >>> >> >sources when selected, but that turned out not to work.
>> >> >> >>> >> >
>> >> >> >>> >> >-Michael Starch
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> >>
>>
>>
Re: Extra Compiler Tools
Posted by Michael Starch <st...@umich.edu>.
According to the specs, any subclass that implements Serializable must
manually implement the serialization of the parents' members. I tested
this and it fails exactly as expected. The parent's members aren't
serialized.
Also, JobInput is an interface.... so I would have no way of catching all
of the possible implementations that could come at me.
Michael
On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Got it, Mike.
>
> Hmm, how about simply creating SerializableJobSpec and
> SerializableJob and SerializableJobInput and then making
> them sub-class their parents and implement Serializable.
> Then, use these classes in your Mesos implementation.
> That seems self-contained, doesn’t change core classes,
> and pretty easy, right?
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Michael Starch <st...@umich.edu>
> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Date: Wednesday, November 5, 2014 at 8:00 PM
> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Extra Compiler Tools
>
> >I need to serialize a JobSpec and children (Job and JobInput) to a byte[].
> >Java can do this automatically by marking all three as Serializable.
> >
> >The work around is to manually serialize to a private inner struct and
> >back
> >out again. The inner class will have members for each member in the
> >JobSpec and children. Java can the auto-serialize that without changing
> >the other three.
> >
> >It is ugly, and essentially a reimplementation of those three
> >classes....but it is entirely self-contained.
> >
> >Michael
> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" <ch...@gmail.com>
> wrote:
> >
> >> Hey Mike,
> >>
> >> Hmm, what’s the work around just so I know
> >> what we’re trading against?
> >>
> >> Cheers,
> >> Chris
> >>
> >> ------------------------
> >> Chris Mattmann
> >> chris.mattmann@gmail.com
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Michael Starch <st...@umich.edu>
> >> Reply-To: <de...@oodt.apache.org>
> >> Date: Wednesday, November 5, 2014 at 6:31 PM
> >> To: <de...@oodt.apache.org>
> >> Subject: Re: Extra Compiler Tools
> >>
> >> >That is basically what I did. Regardless, protobuff proves to be
> >>overkill.
> >> >
> >> >If I mark those classes as serializable, the correct solution is 2
> >>lines
> >> >of
> >> >code. (protobuff was like 20). Wrote a test case, and it works
> >> >perfectly.
> >> >
> >> >If I cannot make JobSpec Job and JonInput implement Serializable then
> >>the
> >> >work around is simple too.
> >> >
> >> >What do you think? Should I mark them as Serializable, or use a
> >> >work-around. Either is a better solution than protobuff.
> >> >
> >> >Michael
> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann" <ch...@gmail.com>
> >> wrote:
> >> >
> >> >> Mike, have you looked at this yet?
> >> >>
> >> >>
> >> >>
> >>
> >>
> http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
> >> >>co
> >> >> l-buffers-from-maven/
> >> >>
> >> >>
> >> >> I’m going to play with it tonight and see if
> >> >> I can help here. Do you have some files I can test
> >> >> with? Can you attach them to JIRA or dropbox them to me
> >> >> so I can scope?
> >> >>
> >> >> Cheers,
> >> >> Chris
> >> >>
> >> >> ------------------------
> >> >> Chris Mattmann
> >> >> chris.mattmann@gmail.com
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> -----Original Message-----
> >> >> From: Michael Starch <st...@umich.edu>
> >> >> Reply-To: <de...@oodt.apache.org>
> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM
> >> >> To: <de...@oodt.apache.org>
> >> >> Subject: Re: Extra Compiler Tools
> >> >>
> >> >> >Ok....time for an audible. Protoc needs to be built from source, no
> >> >> >binary
> >> >> >distributions available. Thus I am going to purge proto-buffers
> >>from
> >> >>the
> >> >> >new code and be done with it.
> >> >> >
> >> >> >Any problem making the following classes/interfaces implement
> >> >> >java.io.Serializable:
> >> >> >
> >> >> >JobSpec
> >> >> >Job
> >> >> >JobInput
> >> >> >
> >> >> >Doing so would allow apache and native java serialization and thus
> >>we
> >> >> >wouldn't need something like proto-buffers.
> >> >> >
> >> >> >-Michael
> >> >> >Thanks Mike +1
> >> >> >
> >> >> >------------------------
> >> >> >Chris Mattmann
> >> >> >chris.mattmann@gmail.com
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >-----Original Message-----
> >> >> >From: Michael Starch <st...@umich.edu>
> >> >> >Reply-To: <de...@oodt.apache.org>
> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
> >> >> >To: <de...@oodt.apache.org>
> >> >> >Subject: Re: Extra Compiler Tools
> >> >> >
> >> >> >>Looks like you followed the same reasoning chain that I did. Yes,
> >>I
> >> >>came
> >> >> >>to the same conclusion that ant-build was best.
> >> >> >>
> >> >> >>I wasn't sure how to download protoc, but you just answered
> >> >>that....so I
> >> >> >>think this is a great solution!
> >> >> >>
> >> >> >>Thanks,
> >> >> >>
> >> >> >>Michael
> >> >> >>
> >> >> >>
> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
> >> >> >><ch...@gmail.com>
> >> >> >>wrote:
> >> >> >>
> >> >> >>> Hi Mike,
> >> >> >>>
> >> >> >>> Thanks for flushing this out.
> >> >> >>>
> >> >> >>> My thoughts on the below:
> >> >> >>>
> >> >> >>>
> >> >> >>> -----Original Message-----
> >> >> >>> From: Michael Starch <st...@umich.edu>
> >> >> >>> Reply-To: <de...@oodt.apache.org>
> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
> >> >> >>> To: <de...@oodt.apache.org>
> >> >> >>> Subject: Re: Extra Compiler Tools
> >> >> >>>
> >> >> >>> >I tried this approach. The plugin requires a path to the
> >>"protoc"
> >> >>tool
> >> >> >>>and
> >> >> >>> >thus a working installation. This is what prompted the
> >>discussion.
> >> >> >>>
> >> >> >>> Ah - no worries, what you could do is:
> >> >> >>>
> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
> >> >> >>>
> >> >> >>> >
> >> >> >>> >Running the plugin under a profile works.
> >> >> >>>
> >> >> >>> Yep.
> >> >> >>>
> >> >> >>> > However, not running the plugin
> >> >> >>> >causes compile errors in dependant code. Excluding this code
> >> >>except
> >> >> >>> >within
> >> >> >>> >the profile doesn't seem to work, and is considered by some to
> >>be
> >> >>bad
> >> >> >>>form
> >> >> >>> >because there is nothing inside the jar file that notes which
> >> >>profiles
> >> >> >>> >were
> >> >> >>> >used to compile.
> >> >> >>>
> >> >> >>> Got it. Suggestion here would be:
> >> >> >>>
> >> >> >>> 2. create a new module, cas-resource-mesos, and inside of that
> >> >>module,
> >> >> >>> take one of the following approaches, assuming the module is
> >> >>activated
> >> >> >>> when -Pwith-mesos is enabled:
> >> >> >>>
> >> >> >>> 2a. Maven Antrun like so (in this old example):
> >> >> >>>
> >> >> >>>
> >> >>
> >>
> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
> >> >> >>>t
> >> >> >>>o-
> >> >> >>> maven2-build
> >> >> >>>
> >> >> >>> (pro: more flexibility in case protoc isn¹t there; to fail on
> >> >>error; to
> >> >> >>> only compile if
> >> >> >>> protoc is available
> >> >> >>>
> >> >> >>> 2b. Maven protobuf plugin
> >> >> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >> >> >>>
> >> >> >>> Here¹s how to enable a module with a profile:
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >>
> >>
> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
> >> >> >>>a
> >> >> >>>ct
> >> >> >>> ive-modules-by-profiles-in-maven/
> >> >> >>>
> >> >> >>>
> >> >> >>> It seems like that is a bad idea though, based on that
> >>discussion.
> >> >> >>>
> >> >> >>> So, here¹s another option:
> >> >> >>>
> >> >> >>> 1. Inside of cas-resource (no special new module or anything
> >>else)
> >> >> >>> 2. include some custom Ant magic via a build.xml file and the
> >>Maven
> >> >> >>> AntRun plugin:
> >> >> >>> 2a. test if protoc is on the system path, and if not, download
> >>it,
> >> >> >>>e.g.,
> >> >> >>> into the target directory (gets deleted on clean)
> >> >> >>> 2b. call protoc and compile after 2a
> >> >> >>>
> >> >> >>> I would suggest this solution as I think it¹s the most robust and
> >> >> >>>ensures
> >> >> >>> we always have a cas-resource that includes mesos and compiled
> >> >> >>>correctly.
> >> >> >>>
> >> >> >>> Cheers,
> >> >> >>> Chris
> >> >> >>>
> >> >> >>> >
> >> >> >>> >Any ideas on how to continue?
> >> >> >>> >
> >> >> >>> >Michael
> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
> >> >><ch...@gmail.com>
> >> >> >>> >wrote:
> >> >> >>> >
> >> >> >>> >> Hi Mike,
> >> >> >>> >>
> >> >> >>> >> Great discussion. It would be nice if there was
> >> >> >>> >> a protoc Maven plugin:
> >> >> >>> >>
> >> >> >>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> Looks like there is. My suggestion:
> >> >> >>> >>
> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos and
> >> >> >>> >> then when activated;
> >> >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
> >> >> >>> >> in the resource manager
> >> >> >>> >>
> >> >> >>> >> Sound good?
> >> >> >>> >>
> >> >> >>> >> Cheers,
> >> >> >>> >> Chris
> >> >> >>> >>
> >> >> >>> >> ------------------------
> >> >> >>> >> Chris Mattmann
> >> >> >>> >> chris.mattmann@gmail.com
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> -----Original Message-----
> >> >> >>> >> From: Michael Starch <st...@umich.edu>
> >> >> >>> >> Reply-To: <de...@oodt.apache.org>
> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
> >> >> >>> >> To: <de...@oodt.apache.org>
> >> >> >>> >> Subject: Extra Compiler Tools
> >> >> >>> >>
> >> >> >>> >> >All,
> >> >> >>> >> >
> >> >> >>> >> >I am trying to integrate apache-mesos with our resource
> >>manager.
> >> >> >>> >>However,
> >> >> >>> >> >mesos uses a technology called "protobuff" from Google for
> >> >> >>> >> >marshaling/unmarshaling data.
> >> >> >>> >> >
> >> >> >>> >> >This requires running a tool called "protoc" to generate a
> >> >>source
> >> >> >>>file
> >> >> >>> >>in
> >> >> >>> >> >java. What is the best way to integrate this step into our
> >> >>build
> >> >> >>> >>process?
> >> >> >>> >> >
> >> >> >>> >> >Options I can conceive of:
> >> >> >>> >> > -Check in generated java file
> >> >> >>> >> > -Require "protoc" installation to build resource manager
> >> >> >>> >> > -Separate extra resource package into new module
> >> >> >>> >> >
> >> >> >>> >> >None of these ideas are very clean.
> >> >> >>> >> >
> >> >> >>> >> >Any other ideas? I tried setting up a profile to only
> >>compile
> >> >> >>>these
> >> >> >>> >> >sources when selected, but that turned out not to work.
> >> >> >>> >> >
> >> >> >>> >> >-Michael Starch
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
>
>
Re: Extra Compiler Tools
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Got it, Mike.
Hmm, how about simply creating SerializableJobSpec and
SerializableJob and SerializableJobInput and then making
them sub-class their parents and implement Serializable.
Then, use these classes in your Mesos implementation.
That seems self-contained, doesn’t change core classes,
and pretty easy, right?
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Date: Wednesday, November 5, 2014 at 8:00 PM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Re: Extra Compiler Tools
>I need to serialize a JobSpec and children (Job and JobInput) to a byte[].
>Java can do this automatically by marking all three as Serializable.
>
>The work around is to manually serialize to a private inner struct and
>back
>out again. The inner class will have members for each member in the
>JobSpec and children. Java can the auto-serialize that without changing
>the other three.
>
>It is ugly, and essentially a reimplementation of those three
>classes....but it is entirely self-contained.
>
>Michael
>On Nov 5, 2014 6:45 PM, "Chris Mattmann" <ch...@gmail.com> wrote:
>
>> Hey Mike,
>>
>> Hmm, what’s the work around just so I know
>> what we’re trading against?
>>
>> Cheers,
>> Chris
>>
>> ------------------------
>> Chris Mattmann
>> chris.mattmann@gmail.com
>>
>>
>>
>>
>> -----Original Message-----
>> From: Michael Starch <st...@umich.edu>
>> Reply-To: <de...@oodt.apache.org>
>> Date: Wednesday, November 5, 2014 at 6:31 PM
>> To: <de...@oodt.apache.org>
>> Subject: Re: Extra Compiler Tools
>>
>> >That is basically what I did. Regardless, protobuff proves to be
>>overkill.
>> >
>> >If I mark those classes as serializable, the correct solution is 2
>>lines
>> >of
>> >code. (protobuff was like 20). Wrote a test case, and it works
>> >perfectly.
>> >
>> >If I cannot make JobSpec Job and JonInput implement Serializable then
>>the
>> >work around is simple too.
>> >
>> >What do you think? Should I mark them as Serializable, or use a
>> >work-around. Either is a better solution than protobuff.
>> >
>> >Michael
>> >On Nov 5, 2014 4:44 PM, "Chris Mattmann" <ch...@gmail.com>
>> wrote:
>> >
>> >> Mike, have you looked at this yet?
>> >>
>> >>
>> >>
>>
>>http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
>> >>co
>> >> l-buffers-from-maven/
>> >>
>> >>
>> >> I’m going to play with it tonight and see if
>> >> I can help here. Do you have some files I can test
>> >> with? Can you attach them to JIRA or dropbox them to me
>> >> so I can scope?
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> ------------------------
>> >> Chris Mattmann
>> >> chris.mattmann@gmail.com
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Michael Starch <st...@umich.edu>
>> >> Reply-To: <de...@oodt.apache.org>
>> >> Date: Wednesday, November 5, 2014 at 5:37 PM
>> >> To: <de...@oodt.apache.org>
>> >> Subject: Re: Extra Compiler Tools
>> >>
>> >> >Ok....time for an audible. Protoc needs to be built from source, no
>> >> >binary
>> >> >distributions available. Thus I am going to purge proto-buffers
>>from
>> >>the
>> >> >new code and be done with it.
>> >> >
>> >> >Any problem making the following classes/interfaces implement
>> >> >java.io.Serializable:
>> >> >
>> >> >JobSpec
>> >> >Job
>> >> >JobInput
>> >> >
>> >> >Doing so would allow apache and native java serialization and thus
>>we
>> >> >wouldn't need something like proto-buffers.
>> >> >
>> >> >-Michael
>> >> >Thanks Mike +1
>> >> >
>> >> >------------------------
>> >> >Chris Mattmann
>> >> >chris.mattmann@gmail.com
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >-----Original Message-----
>> >> >From: Michael Starch <st...@umich.edu>
>> >> >Reply-To: <de...@oodt.apache.org>
>> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
>> >> >To: <de...@oodt.apache.org>
>> >> >Subject: Re: Extra Compiler Tools
>> >> >
>> >> >>Looks like you followed the same reasoning chain that I did. Yes,
>>I
>> >>came
>> >> >>to the same conclusion that ant-build was best.
>> >> >>
>> >> >>I wasn't sure how to download protoc, but you just answered
>> >>that....so I
>> >> >>think this is a great solution!
>> >> >>
>> >> >>Thanks,
>> >> >>
>> >> >>Michael
>> >> >>
>> >> >>
>> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
>> >> >><ch...@gmail.com>
>> >> >>wrote:
>> >> >>
>> >> >>> Hi Mike,
>> >> >>>
>> >> >>> Thanks for flushing this out.
>> >> >>>
>> >> >>> My thoughts on the below:
>> >> >>>
>> >> >>>
>> >> >>> -----Original Message-----
>> >> >>> From: Michael Starch <st...@umich.edu>
>> >> >>> Reply-To: <de...@oodt.apache.org>
>> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
>> >> >>> To: <de...@oodt.apache.org>
>> >> >>> Subject: Re: Extra Compiler Tools
>> >> >>>
>> >> >>> >I tried this approach. The plugin requires a path to the
>>"protoc"
>> >>tool
>> >> >>>and
>> >> >>> >thus a working installation. This is what prompted the
>>discussion.
>> >> >>>
>> >> >>> Ah - no worries, what you could do is:
>> >> >>>
>> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
>> >> >>>
>> >> >>> >
>> >> >>> >Running the plugin under a profile works.
>> >> >>>
>> >> >>> Yep.
>> >> >>>
>> >> >>> > However, not running the plugin
>> >> >>> >causes compile errors in dependant code. Excluding this code
>> >>except
>> >> >>> >within
>> >> >>> >the profile doesn't seem to work, and is considered by some to
>>be
>> >>bad
>> >> >>>form
>> >> >>> >because there is nothing inside the jar file that notes which
>> >>profiles
>> >> >>> >were
>> >> >>> >used to compile.
>> >> >>>
>> >> >>> Got it. Suggestion here would be:
>> >> >>>
>> >> >>> 2. create a new module, cas-resource-mesos, and inside of that
>> >>module,
>> >> >>> take one of the following approaches, assuming the module is
>> >>activated
>> >> >>> when -Pwith-mesos is enabled:
>> >> >>>
>> >> >>> 2a. Maven Antrun like so (in this old example):
>> >> >>>
>> >> >>>
>> >>
>> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
>> >> >>>t
>> >> >>>o-
>> >> >>> maven2-build
>> >> >>>
>> >> >>> (pro: more flexibility in case protoc isn¹t there; to fail on
>> >>error; to
>> >> >>> only compile if
>> >> >>> protoc is available
>> >> >>>
>> >> >>> 2b. Maven protobuf plugin
>> >> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >>>
>> >> >>> Here¹s how to enable a module with a profile:
>> >> >>>
>> >> >>>
>> >> >>>
>> >>
>> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
>> >> >>>a
>> >> >>>ct
>> >> >>> ive-modules-by-profiles-in-maven/
>> >> >>>
>> >> >>>
>> >> >>> It seems like that is a bad idea though, based on that
>>discussion.
>> >> >>>
>> >> >>> So, here¹s another option:
>> >> >>>
>> >> >>> 1. Inside of cas-resource (no special new module or anything
>>else)
>> >> >>> 2. include some custom Ant magic via a build.xml file and the
>>Maven
>> >> >>> AntRun plugin:
>> >> >>> 2a. test if protoc is on the system path, and if not, download
>>it,
>> >> >>>e.g.,
>> >> >>> into the target directory (gets deleted on clean)
>> >> >>> 2b. call protoc and compile after 2a
>> >> >>>
>> >> >>> I would suggest this solution as I think it¹s the most robust and
>> >> >>>ensures
>> >> >>> we always have a cas-resource that includes mesos and compiled
>> >> >>>correctly.
>> >> >>>
>> >> >>> Cheers,
>> >> >>> Chris
>> >> >>>
>> >> >>> >
>> >> >>> >Any ideas on how to continue?
>> >> >>> >
>> >> >>> >Michael
>> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
>> >><ch...@gmail.com>
>> >> >>> >wrote:
>> >> >>> >
>> >> >>> >> Hi Mike,
>> >> >>> >>
>> >> >>> >> Great discussion. It would be nice if there was
>> >> >>> >> a protoc Maven plugin:
>> >> >>> >>
>> >> >>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> Looks like there is. My suggestion:
>> >> >>> >>
>> >> >>> >> 1. use a Profile, something like -Pwith-mesos and
>> >> >>> >> then when activated;
>> >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
>> >> >>> >> in the resource manager
>> >> >>> >>
>> >> >>> >> Sound good?
>> >> >>> >>
>> >> >>> >> Cheers,
>> >> >>> >> Chris
>> >> >>> >>
>> >> >>> >> ------------------------
>> >> >>> >> Chris Mattmann
>> >> >>> >> chris.mattmann@gmail.com
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> -----Original Message-----
>> >> >>> >> From: Michael Starch <st...@umich.edu>
>> >> >>> >> Reply-To: <de...@oodt.apache.org>
>> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
>> >> >>> >> To: <de...@oodt.apache.org>
>> >> >>> >> Subject: Extra Compiler Tools
>> >> >>> >>
>> >> >>> >> >All,
>> >> >>> >> >
>> >> >>> >> >I am trying to integrate apache-mesos with our resource
>>manager.
>> >> >>> >>However,
>> >> >>> >> >mesos uses a technology called "protobuff" from Google for
>> >> >>> >> >marshaling/unmarshaling data.
>> >> >>> >> >
>> >> >>> >> >This requires running a tool called "protoc" to generate a
>> >>source
>> >> >>>file
>> >> >>> >>in
>> >> >>> >> >java. What is the best way to integrate this step into our
>> >>build
>> >> >>> >>process?
>> >> >>> >> >
>> >> >>> >> >Options I can conceive of:
>> >> >>> >> > -Check in generated java file
>> >> >>> >> > -Require "protoc" installation to build resource manager
>> >> >>> >> > -Separate extra resource package into new module
>> >> >>> >> >
>> >> >>> >> >None of these ideas are very clean.
>> >> >>> >> >
>> >> >>> >> >Any other ideas? I tried setting up a profile to only
>>compile
>> >> >>>these
>> >> >>> >> >sources when selected, but that turned out not to work.
>> >> >>> >> >
>> >> >>> >> >-Michael Starch
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>>
>> >> >>>
>> >> >>>
>> >>
>> >>
>> >>
>>
>>
>>
Re: Extra Compiler Tools
Posted by Tyler Palsulich <tp...@gmail.com>.
What is the disadvantage of using Java's built in Serialization?
Tyler
On Wed, Nov 5, 2014 at 10:00 PM, Michael Starch <st...@umich.edu> wrote:
> I need to serialize a JobSpec and children (Job and JobInput) to a byte[].
> Java can do this automatically by marking all three as Serializable.
>
> The work around is to manually serialize to a private inner struct and back
> out again. The inner class will have members for each member in the
> JobSpec and children. Java can the auto-serialize that without changing
> the other three.
>
> It is ugly, and essentially a reimplementation of those three
> classes....but it is entirely self-contained.
>
> Michael
> On Nov 5, 2014 6:45 PM, "Chris Mattmann" <ch...@gmail.com> wrote:
>
> > Hey Mike,
> >
> > Hmm, what’s the work around just so I know
> > what we’re trading against?
> >
> > Cheers,
> > Chris
> >
> > ------------------------
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> > -----Original Message-----
> > From: Michael Starch <st...@umich.edu>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Wednesday, November 5, 2014 at 6:31 PM
> > To: <de...@oodt.apache.org>
> > Subject: Re: Extra Compiler Tools
> >
> > >That is basically what I did. Regardless, protobuff proves to be
> overkill.
> > >
> > >If I mark those classes as serializable, the correct solution is 2 lines
> > >of
> > >code. (protobuff was like 20). Wrote a test case, and it works
> > >perfectly.
> > >
> > >If I cannot make JobSpec Job and JonInput implement Serializable then
> the
> > >work around is simple too.
> > >
> > >What do you think? Should I mark them as Serializable, or use a
> > >work-around. Either is a better solution than protobuff.
> > >
> > >Michael
> > >On Nov 5, 2014 4:44 PM, "Chris Mattmann" <ch...@gmail.com>
> > wrote:
> > >
> > >> Mike, have you looked at this yet?
> > >>
> > >>
> > >>
> >
> http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
> > >>co
> > >> l-buffers-from-maven/
> > >>
> > >>
> > >> I’m going to play with it tonight and see if
> > >> I can help here. Do you have some files I can test
> > >> with? Can you attach them to JIRA or dropbox them to me
> > >> so I can scope?
> > >>
> > >> Cheers,
> > >> Chris
> > >>
> > >> ------------------------
> > >> Chris Mattmann
> > >> chris.mattmann@gmail.com
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Michael Starch <st...@umich.edu>
> > >> Reply-To: <de...@oodt.apache.org>
> > >> Date: Wednesday, November 5, 2014 at 5:37 PM
> > >> To: <de...@oodt.apache.org>
> > >> Subject: Re: Extra Compiler Tools
> > >>
> > >> >Ok....time for an audible. Protoc needs to be built from source, no
> > >> >binary
> > >> >distributions available. Thus I am going to purge proto-buffers from
> > >>the
> > >> >new code and be done with it.
> > >> >
> > >> >Any problem making the following classes/interfaces implement
> > >> >java.io.Serializable:
> > >> >
> > >> >JobSpec
> > >> >Job
> > >> >JobInput
> > >> >
> > >> >Doing so would allow apache and native java serialization and thus we
> > >> >wouldn't need something like proto-buffers.
> > >> >
> > >> >-Michael
> > >> >Thanks Mike +1
> > >> >
> > >> >------------------------
> > >> >Chris Mattmann
> > >> >chris.mattmann@gmail.com
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >-----Original Message-----
> > >> >From: Michael Starch <st...@umich.edu>
> > >> >Reply-To: <de...@oodt.apache.org>
> > >> >Date: Wednesday, November 5, 2014 at 12:31 PM
> > >> >To: <de...@oodt.apache.org>
> > >> >Subject: Re: Extra Compiler Tools
> > >> >
> > >> >>Looks like you followed the same reasoning chain that I did. Yes, I
> > >>came
> > >> >>to the same conclusion that ant-build was best.
> > >> >>
> > >> >>I wasn't sure how to download protoc, but you just answered
> > >>that....so I
> > >> >>think this is a great solution!
> > >> >>
> > >> >>Thanks,
> > >> >>
> > >> >>Michael
> > >> >>
> > >> >>
> > >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
> > >> >><ch...@gmail.com>
> > >> >>wrote:
> > >> >>
> > >> >>> Hi Mike,
> > >> >>>
> > >> >>> Thanks for flushing this out.
> > >> >>>
> > >> >>> My thoughts on the below:
> > >> >>>
> > >> >>>
> > >> >>> -----Original Message-----
> > >> >>> From: Michael Starch <st...@umich.edu>
> > >> >>> Reply-To: <de...@oodt.apache.org>
> > >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
> > >> >>> To: <de...@oodt.apache.org>
> > >> >>> Subject: Re: Extra Compiler Tools
> > >> >>>
> > >> >>> >I tried this approach. The plugin requires a path to the "protoc"
> > >>tool
> > >> >>>and
> > >> >>> >thus a working installation. This is what prompted the
> discussion.
> > >> >>>
> > >> >>> Ah - no worries, what you could do is:
> > >> >>>
> > >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
> > >> >>>
> > >> >>> >
> > >> >>> >Running the plugin under a profile works.
> > >> >>>
> > >> >>> Yep.
> > >> >>>
> > >> >>> > However, not running the plugin
> > >> >>> >causes compile errors in dependant code. Excluding this code
> > >>except
> > >> >>> >within
> > >> >>> >the profile doesn't seem to work, and is considered by some to be
> > >>bad
> > >> >>>form
> > >> >>> >because there is nothing inside the jar file that notes which
> > >>profiles
> > >> >>> >were
> > >> >>> >used to compile.
> > >> >>>
> > >> >>> Got it. Suggestion here would be:
> > >> >>>
> > >> >>> 2. create a new module, cas-resource-mesos, and inside of that
> > >>module,
> > >> >>> take one of the following approaches, assuming the module is
> > >>activated
> > >> >>> when -Pwith-mesos is enabled:
> > >> >>>
> > >> >>> 2a. Maven Antrun like so (in this old example):
> > >> >>>
> > >> >>>
> > >>
> > http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
> > >> >>>t
> > >> >>>o-
> > >> >>> maven2-build
> > >> >>>
> > >> >>> (pro: more flexibility in case protoc isn¹t there; to fail on
> > >>error; to
> > >> >>> only compile if
> > >> >>> protoc is available
> > >> >>>
> > >> >>> 2b. Maven protobuf plugin
> > >> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> > >> >>>
> > >> >>> Here¹s how to enable a module with a profile:
> > >> >>>
> > >> >>>
> > >> >>>
> > >>
> > http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
> > >> >>>a
> > >> >>>ct
> > >> >>> ive-modules-by-profiles-in-maven/
> > >> >>>
> > >> >>>
> > >> >>> It seems like that is a bad idea though, based on that discussion.
> > >> >>>
> > >> >>> So, here¹s another option:
> > >> >>>
> > >> >>> 1. Inside of cas-resource (no special new module or anything else)
> > >> >>> 2. include some custom Ant magic via a build.xml file and the
> Maven
> > >> >>> AntRun plugin:
> > >> >>> 2a. test if protoc is on the system path, and if not, download
> it,
> > >> >>>e.g.,
> > >> >>> into the target directory (gets deleted on clean)
> > >> >>> 2b. call protoc and compile after 2a
> > >> >>>
> > >> >>> I would suggest this solution as I think it¹s the most robust and
> > >> >>>ensures
> > >> >>> we always have a cas-resource that includes mesos and compiled
> > >> >>>correctly.
> > >> >>>
> > >> >>> Cheers,
> > >> >>> Chris
> > >> >>>
> > >> >>> >
> > >> >>> >Any ideas on how to continue?
> > >> >>> >
> > >> >>> >Michael
> > >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
> > >><ch...@gmail.com>
> > >> >>> >wrote:
> > >> >>> >
> > >> >>> >> Hi Mike,
> > >> >>> >>
> > >> >>> >> Great discussion. It would be nice if there was
> > >> >>> >> a protoc Maven plugin:
> > >> >>> >>
> > >> >>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> > >> >>> >>
> > >> >>> >>
> > >> >>> >> Looks like there is. My suggestion:
> > >> >>> >>
> > >> >>> >> 1. use a Profile, something like -Pwith-mesos and
> > >> >>> >> then when activated;
> > >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
> > >> >>> >> in the resource manager
> > >> >>> >>
> > >> >>> >> Sound good?
> > >> >>> >>
> > >> >>> >> Cheers,
> > >> >>> >> Chris
> > >> >>> >>
> > >> >>> >> ------------------------
> > >> >>> >> Chris Mattmann
> > >> >>> >> chris.mattmann@gmail.com
> > >> >>> >>
> > >> >>> >>
> > >> >>> >>
> > >> >>> >>
> > >> >>> >> -----Original Message-----
> > >> >>> >> From: Michael Starch <st...@umich.edu>
> > >> >>> >> Reply-To: <de...@oodt.apache.org>
> > >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
> > >> >>> >> To: <de...@oodt.apache.org>
> > >> >>> >> Subject: Extra Compiler Tools
> > >> >>> >>
> > >> >>> >> >All,
> > >> >>> >> >
> > >> >>> >> >I am trying to integrate apache-mesos with our resource
> manager.
> > >> >>> >>However,
> > >> >>> >> >mesos uses a technology called "protobuff" from Google for
> > >> >>> >> >marshaling/unmarshaling data.
> > >> >>> >> >
> > >> >>> >> >This requires running a tool called "protoc" to generate a
> > >>source
> > >> >>>file
> > >> >>> >>in
> > >> >>> >> >java. What is the best way to integrate this step into our
> > >>build
> > >> >>> >>process?
> > >> >>> >> >
> > >> >>> >> >Options I can conceive of:
> > >> >>> >> > -Check in generated java file
> > >> >>> >> > -Require "protoc" installation to build resource manager
> > >> >>> >> > -Separate extra resource package into new module
> > >> >>> >> >
> > >> >>> >> >None of these ideas are very clean.
> > >> >>> >> >
> > >> >>> >> >Any other ideas? I tried setting up a profile to only compile
> > >> >>>these
> > >> >>> >> >sources when selected, but that turned out not to work.
> > >> >>> >> >
> > >> >>> >> >-Michael Starch
> > >> >>> >>
> > >> >>> >>
> > >> >>> >>
> > >> >>>
> > >> >>>
> > >> >>>
> > >>
> > >>
> > >>
> >
> >
> >
>
Re: Extra Compiler Tools
Posted by Michael Starch <st...@umich.edu>.
I need to serialize a JobSpec and children (Job and JobInput) to a byte[].
Java can do this automatically by marking all three as Serializable.
The work around is to manually serialize to a private inner struct and back
out again. The inner class will have members for each member in the
JobSpec and children. Java can the auto-serialize that without changing
the other three.
It is ugly, and essentially a reimplementation of those three
classes....but it is entirely self-contained.
Michael
On Nov 5, 2014 6:45 PM, "Chris Mattmann" <ch...@gmail.com> wrote:
> Hey Mike,
>
> Hmm, what’s the work around just so I know
> what we’re trading against?
>
> Cheers,
> Chris
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Michael Starch <st...@umich.edu>
> Reply-To: <de...@oodt.apache.org>
> Date: Wednesday, November 5, 2014 at 6:31 PM
> To: <de...@oodt.apache.org>
> Subject: Re: Extra Compiler Tools
>
> >That is basically what I did. Regardless, protobuff proves to be overkill.
> >
> >If I mark those classes as serializable, the correct solution is 2 lines
> >of
> >code. (protobuff was like 20). Wrote a test case, and it works
> >perfectly.
> >
> >If I cannot make JobSpec Job and JonInput implement Serializable then the
> >work around is simple too.
> >
> >What do you think? Should I mark them as Serializable, or use a
> >work-around. Either is a better solution than protobuff.
> >
> >Michael
> >On Nov 5, 2014 4:44 PM, "Chris Mattmann" <ch...@gmail.com>
> wrote:
> >
> >> Mike, have you looked at this yet?
> >>
> >>
> >>
> http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
> >>co
> >> l-buffers-from-maven/
> >>
> >>
> >> I’m going to play with it tonight and see if
> >> I can help here. Do you have some files I can test
> >> with? Can you attach them to JIRA or dropbox them to me
> >> so I can scope?
> >>
> >> Cheers,
> >> Chris
> >>
> >> ------------------------
> >> Chris Mattmann
> >> chris.mattmann@gmail.com
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Michael Starch <st...@umich.edu>
> >> Reply-To: <de...@oodt.apache.org>
> >> Date: Wednesday, November 5, 2014 at 5:37 PM
> >> To: <de...@oodt.apache.org>
> >> Subject: Re: Extra Compiler Tools
> >>
> >> >Ok....time for an audible. Protoc needs to be built from source, no
> >> >binary
> >> >distributions available. Thus I am going to purge proto-buffers from
> >>the
> >> >new code and be done with it.
> >> >
> >> >Any problem making the following classes/interfaces implement
> >> >java.io.Serializable:
> >> >
> >> >JobSpec
> >> >Job
> >> >JobInput
> >> >
> >> >Doing so would allow apache and native java serialization and thus we
> >> >wouldn't need something like proto-buffers.
> >> >
> >> >-Michael
> >> >Thanks Mike +1
> >> >
> >> >------------------------
> >> >Chris Mattmann
> >> >chris.mattmann@gmail.com
> >> >
> >> >
> >> >
> >> >
> >> >-----Original Message-----
> >> >From: Michael Starch <st...@umich.edu>
> >> >Reply-To: <de...@oodt.apache.org>
> >> >Date: Wednesday, November 5, 2014 at 12:31 PM
> >> >To: <de...@oodt.apache.org>
> >> >Subject: Re: Extra Compiler Tools
> >> >
> >> >>Looks like you followed the same reasoning chain that I did. Yes, I
> >>came
> >> >>to the same conclusion that ant-build was best.
> >> >>
> >> >>I wasn't sure how to download protoc, but you just answered
> >>that....so I
> >> >>think this is a great solution!
> >> >>
> >> >>Thanks,
> >> >>
> >> >>Michael
> >> >>
> >> >>
> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
> >> >><ch...@gmail.com>
> >> >>wrote:
> >> >>
> >> >>> Hi Mike,
> >> >>>
> >> >>> Thanks for flushing this out.
> >> >>>
> >> >>> My thoughts on the below:
> >> >>>
> >> >>>
> >> >>> -----Original Message-----
> >> >>> From: Michael Starch <st...@umich.edu>
> >> >>> Reply-To: <de...@oodt.apache.org>
> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
> >> >>> To: <de...@oodt.apache.org>
> >> >>> Subject: Re: Extra Compiler Tools
> >> >>>
> >> >>> >I tried this approach. The plugin requires a path to the "protoc"
> >>tool
> >> >>>and
> >> >>> >thus a working installation. This is what prompted the discussion.
> >> >>>
> >> >>> Ah - no worries, what you could do is:
> >> >>>
> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
> >> >>>
> >> >>> >
> >> >>> >Running the plugin under a profile works.
> >> >>>
> >> >>> Yep.
> >> >>>
> >> >>> > However, not running the plugin
> >> >>> >causes compile errors in dependant code. Excluding this code
> >>except
> >> >>> >within
> >> >>> >the profile doesn't seem to work, and is considered by some to be
> >>bad
> >> >>>form
> >> >>> >because there is nothing inside the jar file that notes which
> >>profiles
> >> >>> >were
> >> >>> >used to compile.
> >> >>>
> >> >>> Got it. Suggestion here would be:
> >> >>>
> >> >>> 2. create a new module, cas-resource-mesos, and inside of that
> >>module,
> >> >>> take one of the following approaches, assuming the module is
> >>activated
> >> >>> when -Pwith-mesos is enabled:
> >> >>>
> >> >>> 2a. Maven Antrun like so (in this old example):
> >> >>>
> >> >>>
> >>
> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
> >> >>>t
> >> >>>o-
> >> >>> maven2-build
> >> >>>
> >> >>> (pro: more flexibility in case protoc isn¹t there; to fail on
> >>error; to
> >> >>> only compile if
> >> >>> protoc is available
> >> >>>
> >> >>> 2b. Maven protobuf plugin
> >> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >> >>>
> >> >>> Here¹s how to enable a module with a profile:
> >> >>>
> >> >>>
> >> >>>
> >>
> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
> >> >>>a
> >> >>>ct
> >> >>> ive-modules-by-profiles-in-maven/
> >> >>>
> >> >>>
> >> >>> It seems like that is a bad idea though, based on that discussion.
> >> >>>
> >> >>> So, here¹s another option:
> >> >>>
> >> >>> 1. Inside of cas-resource (no special new module or anything else)
> >> >>> 2. include some custom Ant magic via a build.xml file and the Maven
> >> >>> AntRun plugin:
> >> >>> 2a. test if protoc is on the system path, and if not, download it,
> >> >>>e.g.,
> >> >>> into the target directory (gets deleted on clean)
> >> >>> 2b. call protoc and compile after 2a
> >> >>>
> >> >>> I would suggest this solution as I think it¹s the most robust and
> >> >>>ensures
> >> >>> we always have a cas-resource that includes mesos and compiled
> >> >>>correctly.
> >> >>>
> >> >>> Cheers,
> >> >>> Chris
> >> >>>
> >> >>> >
> >> >>> >Any ideas on how to continue?
> >> >>> >
> >> >>> >Michael
> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
> >><ch...@gmail.com>
> >> >>> >wrote:
> >> >>> >
> >> >>> >> Hi Mike,
> >> >>> >>
> >> >>> >> Great discussion. It would be nice if there was
> >> >>> >> a protoc Maven plugin:
> >> >>> >>
> >> >>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >> >>> >>
> >> >>> >>
> >> >>> >> Looks like there is. My suggestion:
> >> >>> >>
> >> >>> >> 1. use a Profile, something like -Pwith-mesos and
> >> >>> >> then when activated;
> >> >>> >> 2. call the above plugin if -Pwith-mesos is activated
> >> >>> >> in the resource manager
> >> >>> >>
> >> >>> >> Sound good?
> >> >>> >>
> >> >>> >> Cheers,
> >> >>> >> Chris
> >> >>> >>
> >> >>> >> ------------------------
> >> >>> >> Chris Mattmann
> >> >>> >> chris.mattmann@gmail.com
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> -----Original Message-----
> >> >>> >> From: Michael Starch <st...@umich.edu>
> >> >>> >> Reply-To: <de...@oodt.apache.org>
> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
> >> >>> >> To: <de...@oodt.apache.org>
> >> >>> >> Subject: Extra Compiler Tools
> >> >>> >>
> >> >>> >> >All,
> >> >>> >> >
> >> >>> >> >I am trying to integrate apache-mesos with our resource manager.
> >> >>> >>However,
> >> >>> >> >mesos uses a technology called "protobuff" from Google for
> >> >>> >> >marshaling/unmarshaling data.
> >> >>> >> >
> >> >>> >> >This requires running a tool called "protoc" to generate a
> >>source
> >> >>>file
> >> >>> >>in
> >> >>> >> >java. What is the best way to integrate this step into our
> >>build
> >> >>> >>process?
> >> >>> >> >
> >> >>> >> >Options I can conceive of:
> >> >>> >> > -Check in generated java file
> >> >>> >> > -Require "protoc" installation to build resource manager
> >> >>> >> > -Separate extra resource package into new module
> >> >>> >> >
> >> >>> >> >None of these ideas are very clean.
> >> >>> >> >
> >> >>> >> >Any other ideas? I tried setting up a profile to only compile
> >> >>>these
> >> >>> >> >sources when selected, but that turned out not to work.
> >> >>> >> >
> >> >>> >> >-Michael Starch
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>>
> >> >>>
> >> >>>
> >>
> >>
> >>
>
>
>
Re: Extra Compiler Tools
Posted by Chris Mattmann <ch...@gmail.com>.
Hey Mike,
Hmm, what’s the work around just so I know
what we’re trading against?
Cheers,
Chris
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: <de...@oodt.apache.org>
Date: Wednesday, November 5, 2014 at 6:31 PM
To: <de...@oodt.apache.org>
Subject: Re: Extra Compiler Tools
>That is basically what I did. Regardless, protobuff proves to be overkill.
>
>If I mark those classes as serializable, the correct solution is 2 lines
>of
>code. (protobuff was like 20). Wrote a test case, and it works
>perfectly.
>
>If I cannot make JobSpec Job and JonInput implement Serializable then the
>work around is simple too.
>
>What do you think? Should I mark them as Serializable, or use a
>work-around. Either is a better solution than protobuff.
>
>Michael
>On Nov 5, 2014 4:44 PM, "Chris Mattmann" <ch...@gmail.com> wrote:
>
>> Mike, have you looked at this yet?
>>
>>
>>http://techtraits.com/build%20management/maven/2011/09/09/compiling-proto
>>co
>> l-buffers-from-maven/
>>
>>
>> I’m going to play with it tonight and see if
>> I can help here. Do you have some files I can test
>> with? Can you attach them to JIRA or dropbox them to me
>> so I can scope?
>>
>> Cheers,
>> Chris
>>
>> ------------------------
>> Chris Mattmann
>> chris.mattmann@gmail.com
>>
>>
>>
>>
>> -----Original Message-----
>> From: Michael Starch <st...@umich.edu>
>> Reply-To: <de...@oodt.apache.org>
>> Date: Wednesday, November 5, 2014 at 5:37 PM
>> To: <de...@oodt.apache.org>
>> Subject: Re: Extra Compiler Tools
>>
>> >Ok....time for an audible. Protoc needs to be built from source, no
>> >binary
>> >distributions available. Thus I am going to purge proto-buffers from
>>the
>> >new code and be done with it.
>> >
>> >Any problem making the following classes/interfaces implement
>> >java.io.Serializable:
>> >
>> >JobSpec
>> >Job
>> >JobInput
>> >
>> >Doing so would allow apache and native java serialization and thus we
>> >wouldn't need something like proto-buffers.
>> >
>> >-Michael
>> >Thanks Mike +1
>> >
>> >------------------------
>> >Chris Mattmann
>> >chris.mattmann@gmail.com
>> >
>> >
>> >
>> >
>> >-----Original Message-----
>> >From: Michael Starch <st...@umich.edu>
>> >Reply-To: <de...@oodt.apache.org>
>> >Date: Wednesday, November 5, 2014 at 12:31 PM
>> >To: <de...@oodt.apache.org>
>> >Subject: Re: Extra Compiler Tools
>> >
>> >>Looks like you followed the same reasoning chain that I did. Yes, I
>>came
>> >>to the same conclusion that ant-build was best.
>> >>
>> >>I wasn't sure how to download protoc, but you just answered
>>that....so I
>> >>think this is a great solution!
>> >>
>> >>Thanks,
>> >>
>> >>Michael
>> >>
>> >>
>> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
>> >><ch...@gmail.com>
>> >>wrote:
>> >>
>> >>> Hi Mike,
>> >>>
>> >>> Thanks for flushing this out.
>> >>>
>> >>> My thoughts on the below:
>> >>>
>> >>>
>> >>> -----Original Message-----
>> >>> From: Michael Starch <st...@umich.edu>
>> >>> Reply-To: <de...@oodt.apache.org>
>> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
>> >>> To: <de...@oodt.apache.org>
>> >>> Subject: Re: Extra Compiler Tools
>> >>>
>> >>> >I tried this approach. The plugin requires a path to the "protoc"
>>tool
>> >>>and
>> >>> >thus a working installation. This is what prompted the discussion.
>> >>>
>> >>> Ah - no worries, what you could do is:
>> >>>
>> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
>> >>>
>> >>> >
>> >>> >Running the plugin under a profile works.
>> >>>
>> >>> Yep.
>> >>>
>> >>> > However, not running the plugin
>> >>> >causes compile errors in dependant code. Excluding this code
>>except
>> >>> >within
>> >>> >the profile doesn't seem to work, and is considered by some to be
>>bad
>> >>>form
>> >>> >because there is nothing inside the jar file that notes which
>>profiles
>> >>> >were
>> >>> >used to compile.
>> >>>
>> >>> Got it. Suggestion here would be:
>> >>>
>> >>> 2. create a new module, cas-resource-mesos, and inside of that
>>module,
>> >>> take one of the following approaches, assuming the module is
>>activated
>> >>> when -Pwith-mesos is enabled:
>> >>>
>> >>> 2a. Maven Antrun like so (in this old example):
>> >>>
>> >>>
>> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
>> >>>t
>> >>>o-
>> >>> maven2-build
>> >>>
>> >>> (pro: more flexibility in case protoc isn¹t there; to fail on
>>error; to
>> >>> only compile if
>> >>> protoc is available
>> >>>
>> >>> 2b. Maven protobuf plugin
>> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >>>
>> >>> Here¹s how to enable a module with a profile:
>> >>>
>> >>>
>> >>>
>> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
>> >>>a
>> >>>ct
>> >>> ive-modules-by-profiles-in-maven/
>> >>>
>> >>>
>> >>> It seems like that is a bad idea though, based on that discussion.
>> >>>
>> >>> So, here¹s another option:
>> >>>
>> >>> 1. Inside of cas-resource (no special new module or anything else)
>> >>> 2. include some custom Ant magic via a build.xml file and the Maven
>> >>> AntRun plugin:
>> >>> 2a. test if protoc is on the system path, and if not, download it,
>> >>>e.g.,
>> >>> into the target directory (gets deleted on clean)
>> >>> 2b. call protoc and compile after 2a
>> >>>
>> >>> I would suggest this solution as I think it¹s the most robust and
>> >>>ensures
>> >>> we always have a cas-resource that includes mesos and compiled
>> >>>correctly.
>> >>>
>> >>> Cheers,
>> >>> Chris
>> >>>
>> >>> >
>> >>> >Any ideas on how to continue?
>> >>> >
>> >>> >Michael
>> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann"
>><ch...@gmail.com>
>> >>> >wrote:
>> >>> >
>> >>> >> Hi Mike,
>> >>> >>
>> >>> >> Great discussion. It would be nice if there was
>> >>> >> a protoc Maven plugin:
>> >>> >>
>> >>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >>> >>
>> >>> >>
>> >>> >> Looks like there is. My suggestion:
>> >>> >>
>> >>> >> 1. use a Profile, something like -Pwith-mesos and
>> >>> >> then when activated;
>> >>> >> 2. call the above plugin if -Pwith-mesos is activated
>> >>> >> in the resource manager
>> >>> >>
>> >>> >> Sound good?
>> >>> >>
>> >>> >> Cheers,
>> >>> >> Chris
>> >>> >>
>> >>> >> ------------------------
>> >>> >> Chris Mattmann
>> >>> >> chris.mattmann@gmail.com
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> -----Original Message-----
>> >>> >> From: Michael Starch <st...@umich.edu>
>> >>> >> Reply-To: <de...@oodt.apache.org>
>> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
>> >>> >> To: <de...@oodt.apache.org>
>> >>> >> Subject: Extra Compiler Tools
>> >>> >>
>> >>> >> >All,
>> >>> >> >
>> >>> >> >I am trying to integrate apache-mesos with our resource manager.
>> >>> >>However,
>> >>> >> >mesos uses a technology called "protobuff" from Google for
>> >>> >> >marshaling/unmarshaling data.
>> >>> >> >
>> >>> >> >This requires running a tool called "protoc" to generate a
>>source
>> >>>file
>> >>> >>in
>> >>> >> >java. What is the best way to integrate this step into our
>>build
>> >>> >>process?
>> >>> >> >
>> >>> >> >Options I can conceive of:
>> >>> >> > -Check in generated java file
>> >>> >> > -Require "protoc" installation to build resource manager
>> >>> >> > -Separate extra resource package into new module
>> >>> >> >
>> >>> >> >None of these ideas are very clean.
>> >>> >> >
>> >>> >> >Any other ideas? I tried setting up a profile to only compile
>> >>>these
>> >>> >> >sources when selected, but that turned out not to work.
>> >>> >> >
>> >>> >> >-Michael Starch
>> >>> >>
>> >>> >>
>> >>> >>
>> >>>
>> >>>
>> >>>
>>
>>
>>
Re: Extra Compiler Tools
Posted by Michael Starch <st...@umich.edu>.
That is basically what I did. Regardless, protobuff proves to be overkill.
If I mark those classes as serializable, the correct solution is 2 lines of
code. (protobuff was like 20). Wrote a test case, and it works perfectly.
If I cannot make JobSpec Job and JonInput implement Serializable then the
work around is simple too.
What do you think? Should I mark them as Serializable, or use a
work-around. Either is a better solution than protobuff.
Michael
On Nov 5, 2014 4:44 PM, "Chris Mattmann" <ch...@gmail.com> wrote:
> Mike, have you looked at this yet?
>
> http://techtraits.com/build%20management/maven/2011/09/09/compiling-protoco
> l-buffers-from-maven/
>
>
> I’m going to play with it tonight and see if
> I can help here. Do you have some files I can test
> with? Can you attach them to JIRA or dropbox them to me
> so I can scope?
>
> Cheers,
> Chris
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Michael Starch <st...@umich.edu>
> Reply-To: <de...@oodt.apache.org>
> Date: Wednesday, November 5, 2014 at 5:37 PM
> To: <de...@oodt.apache.org>
> Subject: Re: Extra Compiler Tools
>
> >Ok....time for an audible. Protoc needs to be built from source, no
> >binary
> >distributions available. Thus I am going to purge proto-buffers from the
> >new code and be done with it.
> >
> >Any problem making the following classes/interfaces implement
> >java.io.Serializable:
> >
> >JobSpec
> >Job
> >JobInput
> >
> >Doing so would allow apache and native java serialization and thus we
> >wouldn't need something like proto-buffers.
> >
> >-Michael
> >Thanks Mike +1
> >
> >------------------------
> >Chris Mattmann
> >chris.mattmann@gmail.com
> >
> >
> >
> >
> >-----Original Message-----
> >From: Michael Starch <st...@umich.edu>
> >Reply-To: <de...@oodt.apache.org>
> >Date: Wednesday, November 5, 2014 at 12:31 PM
> >To: <de...@oodt.apache.org>
> >Subject: Re: Extra Compiler Tools
> >
> >>Looks like you followed the same reasoning chain that I did. Yes, I came
> >>to the same conclusion that ant-build was best.
> >>
> >>I wasn't sure how to download protoc, but you just answered that....so I
> >>think this is a great solution!
> >>
> >>Thanks,
> >>
> >>Michael
> >>
> >>
> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
> >><ch...@gmail.com>
> >>wrote:
> >>
> >>> Hi Mike,
> >>>
> >>> Thanks for flushing this out.
> >>>
> >>> My thoughts on the below:
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Michael Starch <st...@umich.edu>
> >>> Reply-To: <de...@oodt.apache.org>
> >>> Date: Wednesday, November 5, 2014 at 12:12 PM
> >>> To: <de...@oodt.apache.org>
> >>> Subject: Re: Extra Compiler Tools
> >>>
> >>> >I tried this approach. The plugin requires a path to the "protoc" tool
> >>>and
> >>> >thus a working installation. This is what prompted the discussion.
> >>>
> >>> Ah - no worries, what you could do is:
> >>>
> >>> 1. only enable to plugin if -Pwith-mesos is enabled; and
> >>>
> >>> >
> >>> >Running the plugin under a profile works.
> >>>
> >>> Yep.
> >>>
> >>> > However, not running the plugin
> >>> >causes compile errors in dependant code. Excluding this code except
> >>> >within
> >>> >the profile doesn't seem to work, and is considered by some to be bad
> >>>form
> >>> >because there is nothing inside the jar file that notes which profiles
> >>> >were
> >>> >used to compile.
> >>>
> >>> Got it. Suggestion here would be:
> >>>
> >>> 2. create a new module, cas-resource-mesos, and inside of that module,
> >>> take one of the following approaches, assuming the module is activated
> >>> when -Pwith-mesos is enabled:
> >>>
> >>> 2a. Maven Antrun like so (in this old example):
> >>>
> >>>
> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
> >>>t
> >>>o-
> >>> maven2-build
> >>>
> >>> (pro: more flexibility in case protoc isn¹t there; to fail on error; to
> >>> only compile if
> >>> protoc is available
> >>>
> >>> 2b. Maven protobuf plugin
> >>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >>>
> >>> Here¹s how to enable a module with a profile:
> >>>
> >>>
> >>>
> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
> >>>a
> >>>ct
> >>> ive-modules-by-profiles-in-maven/
> >>>
> >>>
> >>> It seems like that is a bad idea though, based on that discussion.
> >>>
> >>> So, here¹s another option:
> >>>
> >>> 1. Inside of cas-resource (no special new module or anything else)
> >>> 2. include some custom Ant magic via a build.xml file and the Maven
> >>> AntRun plugin:
> >>> 2a. test if protoc is on the system path, and if not, download it,
> >>>e.g.,
> >>> into the target directory (gets deleted on clean)
> >>> 2b. call protoc and compile after 2a
> >>>
> >>> I would suggest this solution as I think it¹s the most robust and
> >>>ensures
> >>> we always have a cas-resource that includes mesos and compiled
> >>>correctly.
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> >
> >>> >Any ideas on how to continue?
> >>> >
> >>> >Michael
> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann" <ch...@gmail.com>
> >>> >wrote:
> >>> >
> >>> >> Hi Mike,
> >>> >>
> >>> >> Great discussion. It would be nice if there was
> >>> >> a protoc Maven plugin:
> >>> >>
> >>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >>> >>
> >>> >>
> >>> >> Looks like there is. My suggestion:
> >>> >>
> >>> >> 1. use a Profile, something like -Pwith-mesos and
> >>> >> then when activated;
> >>> >> 2. call the above plugin if -Pwith-mesos is activated
> >>> >> in the resource manager
> >>> >>
> >>> >> Sound good?
> >>> >>
> >>> >> Cheers,
> >>> >> Chris
> >>> >>
> >>> >> ------------------------
> >>> >> Chris Mattmann
> >>> >> chris.mattmann@gmail.com
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> -----Original Message-----
> >>> >> From: Michael Starch <st...@umich.edu>
> >>> >> Reply-To: <de...@oodt.apache.org>
> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
> >>> >> To: <de...@oodt.apache.org>
> >>> >> Subject: Extra Compiler Tools
> >>> >>
> >>> >> >All,
> >>> >> >
> >>> >> >I am trying to integrate apache-mesos with our resource manager.
> >>> >>However,
> >>> >> >mesos uses a technology called "protobuff" from Google for
> >>> >> >marshaling/unmarshaling data.
> >>> >> >
> >>> >> >This requires running a tool called "protoc" to generate a source
> >>>file
> >>> >>in
> >>> >> >java. What is the best way to integrate this step into our build
> >>> >>process?
> >>> >> >
> >>> >> >Options I can conceive of:
> >>> >> > -Check in generated java file
> >>> >> > -Require "protoc" installation to build resource manager
> >>> >> > -Separate extra resource package into new module
> >>> >> >
> >>> >> >None of these ideas are very clean.
> >>> >> >
> >>> >> >Any other ideas? I tried setting up a profile to only compile
> >>>these
> >>> >> >sources when selected, but that turned out not to work.
> >>> >> >
> >>> >> >-Michael Starch
> >>> >>
> >>> >>
> >>> >>
> >>>
> >>>
> >>>
>
>
>
Re: Extra Compiler Tools
Posted by Chris Mattmann <ch...@gmail.com>.
Mike, have you looked at this yet?
http://techtraits.com/build%20management/maven/2011/09/09/compiling-protoco
l-buffers-from-maven/
I’m going to play with it tonight and see if
I can help here. Do you have some files I can test
with? Can you attach them to JIRA or dropbox them to me
so I can scope?
Cheers,
Chris
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: <de...@oodt.apache.org>
Date: Wednesday, November 5, 2014 at 5:37 PM
To: <de...@oodt.apache.org>
Subject: Re: Extra Compiler Tools
>Ok....time for an audible. Protoc needs to be built from source, no
>binary
>distributions available. Thus I am going to purge proto-buffers from the
>new code and be done with it.
>
>Any problem making the following classes/interfaces implement
>java.io.Serializable:
>
>JobSpec
>Job
>JobInput
>
>Doing so would allow apache and native java serialization and thus we
>wouldn't need something like proto-buffers.
>
>-Michael
>Thanks Mike +1
>
>------------------------
>Chris Mattmann
>chris.mattmann@gmail.com
>
>
>
>
>-----Original Message-----
>From: Michael Starch <st...@umich.edu>
>Reply-To: <de...@oodt.apache.org>
>Date: Wednesday, November 5, 2014 at 12:31 PM
>To: <de...@oodt.apache.org>
>Subject: Re: Extra Compiler Tools
>
>>Looks like you followed the same reasoning chain that I did. Yes, I came
>>to the same conclusion that ant-build was best.
>>
>>I wasn't sure how to download protoc, but you just answered that....so I
>>think this is a great solution!
>>
>>Thanks,
>>
>>Michael
>>
>>
>>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann
>><ch...@gmail.com>
>>wrote:
>>
>>> Hi Mike,
>>>
>>> Thanks for flushing this out.
>>>
>>> My thoughts on the below:
>>>
>>>
>>> -----Original Message-----
>>> From: Michael Starch <st...@umich.edu>
>>> Reply-To: <de...@oodt.apache.org>
>>> Date: Wednesday, November 5, 2014 at 12:12 PM
>>> To: <de...@oodt.apache.org>
>>> Subject: Re: Extra Compiler Tools
>>>
>>> >I tried this approach. The plugin requires a path to the "protoc" tool
>>>and
>>> >thus a working installation. This is what prompted the discussion.
>>>
>>> Ah - no worries, what you could do is:
>>>
>>> 1. only enable to plugin if -Pwith-mesos is enabled; and
>>>
>>> >
>>> >Running the plugin under a profile works.
>>>
>>> Yep.
>>>
>>> > However, not running the plugin
>>> >causes compile errors in dependant code. Excluding this code except
>>> >within
>>> >the profile doesn't seem to work, and is considered by some to be bad
>>>form
>>> >because there is nothing inside the jar file that notes which profiles
>>> >were
>>> >used to compile.
>>>
>>> Got it. Suggestion here would be:
>>>
>>> 2. create a new module, cas-resource-mesos, and inside of that module,
>>> take one of the following approaches, assuming the module is activated
>>> when -Pwith-mesos is enabled:
>>>
>>> 2a. Maven Antrun like so (in this old example):
>>>
>>>http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in
>>>t
>>>o-
>>> maven2-build
>>>
>>> (pro: more flexibility in case protoc isn¹t there; to fail on error; to
>>> only compile if
>>> protoc is available
>>>
>>> 2b. Maven protobuf plugin
>>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>>>
>>> Here¹s how to enable a module with a profile:
>>>
>>>
>>>http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de
>>>a
>>>ct
>>> ive-modules-by-profiles-in-maven/
>>>
>>>
>>> It seems like that is a bad idea though, based on that discussion.
>>>
>>> So, here¹s another option:
>>>
>>> 1. Inside of cas-resource (no special new module or anything else)
>>> 2. include some custom Ant magic via a build.xml file and the Maven
>>> AntRun plugin:
>>> 2a. test if protoc is on the system path, and if not, download it,
>>>e.g.,
>>> into the target directory (gets deleted on clean)
>>> 2b. call protoc and compile after 2a
>>>
>>> I would suggest this solution as I think it¹s the most robust and
>>>ensures
>>> we always have a cas-resource that includes mesos and compiled
>>>correctly.
>>>
>>> Cheers,
>>> Chris
>>>
>>> >
>>> >Any ideas on how to continue?
>>> >
>>> >Michael
>>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann" <ch...@gmail.com>
>>> >wrote:
>>> >
>>> >> Hi Mike,
>>> >>
>>> >> Great discussion. It would be nice if there was
>>> >> a protoc Maven plugin:
>>> >>
>>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>>> >>
>>> >>
>>> >> Looks like there is. My suggestion:
>>> >>
>>> >> 1. use a Profile, something like -Pwith-mesos and
>>> >> then when activated;
>>> >> 2. call the above plugin if -Pwith-mesos is activated
>>> >> in the resource manager
>>> >>
>>> >> Sound good?
>>> >>
>>> >> Cheers,
>>> >> Chris
>>> >>
>>> >> ------------------------
>>> >> Chris Mattmann
>>> >> chris.mattmann@gmail.com
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: Michael Starch <st...@umich.edu>
>>> >> Reply-To: <de...@oodt.apache.org>
>>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
>>> >> To: <de...@oodt.apache.org>
>>> >> Subject: Extra Compiler Tools
>>> >>
>>> >> >All,
>>> >> >
>>> >> >I am trying to integrate apache-mesos with our resource manager.
>>> >>However,
>>> >> >mesos uses a technology called "protobuff" from Google for
>>> >> >marshaling/unmarshaling data.
>>> >> >
>>> >> >This requires running a tool called "protoc" to generate a source
>>>file
>>> >>in
>>> >> >java. What is the best way to integrate this step into our build
>>> >>process?
>>> >> >
>>> >> >Options I can conceive of:
>>> >> > -Check in generated java file
>>> >> > -Require "protoc" installation to build resource manager
>>> >> > -Separate extra resource package into new module
>>> >> >
>>> >> >None of these ideas are very clean.
>>> >> >
>>> >> >Any other ideas? I tried setting up a profile to only compile
>>>these
>>> >> >sources when selected, but that turned out not to work.
>>> >> >
>>> >> >-Michael Starch
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
Re: Extra Compiler Tools
Posted by Michael Starch <st...@umich.edu>.
Ok....time for an audible. Protoc needs to be built from source, no binary
distributions available. Thus I am going to purge proto-buffers from the
new code and be done with it.
Any problem making the following classes/interfaces implement
java.io.Serializable:
JobSpec
Job
JobInput
Doing so would allow apache and native java serialization and thus we
wouldn't need something like proto-buffers.
-Michael
Thanks Mike +1
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: <de...@oodt.apache.org>
Date: Wednesday, November 5, 2014 at 12:31 PM
To: <de...@oodt.apache.org>
Subject: Re: Extra Compiler Tools
>Looks like you followed the same reasoning chain that I did. Yes, I came
>to the same conclusion that ant-build was best.
>
>I wasn't sure how to download protoc, but you just answered that....so I
>think this is a great solution!
>
>Thanks,
>
>Michael
>
>
>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann <ch...@gmail.com>
>wrote:
>
>> Hi Mike,
>>
>> Thanks for flushing this out.
>>
>> My thoughts on the below:
>>
>>
>> -----Original Message-----
>> From: Michael Starch <st...@umich.edu>
>> Reply-To: <de...@oodt.apache.org>
>> Date: Wednesday, November 5, 2014 at 12:12 PM
>> To: <de...@oodt.apache.org>
>> Subject: Re: Extra Compiler Tools
>>
>> >I tried this approach. The plugin requires a path to the "protoc" tool
>>and
>> >thus a working installation. This is what prompted the discussion.
>>
>> Ah - no worries, what you could do is:
>>
>> 1. only enable to plugin if -Pwith-mesos is enabled; and
>>
>> >
>> >Running the plugin under a profile works.
>>
>> Yep.
>>
>> > However, not running the plugin
>> >causes compile errors in dependant code. Excluding this code except
>> >within
>> >the profile doesn't seem to work, and is considered by some to be bad
>>form
>> >because there is nothing inside the jar file that notes which profiles
>> >were
>> >used to compile.
>>
>> Got it. Suggestion here would be:
>>
>> 2. create a new module, cas-resource-mesos, and inside of that module,
>> take one of the following approaches, assuming the module is activated
>> when -Pwith-mesos is enabled:
>>
>> 2a. Maven Antrun like so (in this old example):
>>
>>http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-int
>>o-
>> maven2-build
>>
>> (pro: more flexibility in case protoc isn¹t there; to fail on error; to
>> only compile if
>> protoc is available
>>
>> 2b. Maven protobuf plugin
>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>>
>> Here¹s how to enable a module with a profile:
>>
>>
>>http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-dea
>>ct
>> ive-modules-by-profiles-in-maven/
>>
>>
>> It seems like that is a bad idea though, based on that discussion.
>>
>> So, here¹s another option:
>>
>> 1. Inside of cas-resource (no special new module or anything else)
>> 2. include some custom Ant magic via a build.xml file and the Maven
>> AntRun plugin:
>> 2a. test if protoc is on the system path, and if not, download it,
>>e.g.,
>> into the target directory (gets deleted on clean)
>> 2b. call protoc and compile after 2a
>>
>> I would suggest this solution as I think it¹s the most robust and
>>ensures
>> we always have a cas-resource that includes mesos and compiled
>>correctly.
>>
>> Cheers,
>> Chris
>>
>> >
>> >Any ideas on how to continue?
>> >
>> >Michael
>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann" <ch...@gmail.com>
>> >wrote:
>> >
>> >> Hi Mike,
>> >>
>> >> Great discussion. It would be nice if there was
>> >> a protoc Maven plugin:
>> >>
>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >>
>> >>
>> >> Looks like there is. My suggestion:
>> >>
>> >> 1. use a Profile, something like -Pwith-mesos and
>> >> then when activated;
>> >> 2. call the above plugin if -Pwith-mesos is activated
>> >> in the resource manager
>> >>
>> >> Sound good?
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> ------------------------
>> >> Chris Mattmann
>> >> chris.mattmann@gmail.com
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Michael Starch <st...@umich.edu>
>> >> Reply-To: <de...@oodt.apache.org>
>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
>> >> To: <de...@oodt.apache.org>
>> >> Subject: Extra Compiler Tools
>> >>
>> >> >All,
>> >> >
>> >> >I am trying to integrate apache-mesos with our resource manager.
>> >>However,
>> >> >mesos uses a technology called "protobuff" from Google for
>> >> >marshaling/unmarshaling data.
>> >> >
>> >> >This requires running a tool called "protoc" to generate a source
>>file
>> >>in
>> >> >java. What is the best way to integrate this step into our build
>> >>process?
>> >> >
>> >> >Options I can conceive of:
>> >> > -Check in generated java file
>> >> > -Require "protoc" installation to build resource manager
>> >> > -Separate extra resource package into new module
>> >> >
>> >> >None of these ideas are very clean.
>> >> >
>> >> >Any other ideas? I tried setting up a profile to only compile these
>> >> >sources when selected, but that turned out not to work.
>> >> >
>> >> >-Michael Starch
>> >>
>> >>
>> >>
>>
>>
>>
Re: Extra Compiler Tools
Posted by Chris Mattmann <ch...@gmail.com>.
Thanks Mike +1
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: <de...@oodt.apache.org>
Date: Wednesday, November 5, 2014 at 12:31 PM
To: <de...@oodt.apache.org>
Subject: Re: Extra Compiler Tools
>Looks like you followed the same reasoning chain that I did. Yes, I came
>to the same conclusion that ant-build was best.
>
>I wasn't sure how to download protoc, but you just answered that....so I
>think this is a great solution!
>
>Thanks,
>
>Michael
>
>
>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann <ch...@gmail.com>
>wrote:
>
>> Hi Mike,
>>
>> Thanks for flushing this out.
>>
>> My thoughts on the below:
>>
>>
>> -----Original Message-----
>> From: Michael Starch <st...@umich.edu>
>> Reply-To: <de...@oodt.apache.org>
>> Date: Wednesday, November 5, 2014 at 12:12 PM
>> To: <de...@oodt.apache.org>
>> Subject: Re: Extra Compiler Tools
>>
>> >I tried this approach. The plugin requires a path to the "protoc" tool
>>and
>> >thus a working installation. This is what prompted the discussion.
>>
>> Ah - no worries, what you could do is:
>>
>> 1. only enable to plugin if -Pwith-mesos is enabled; and
>>
>> >
>> >Running the plugin under a profile works.
>>
>> Yep.
>>
>> > However, not running the plugin
>> >causes compile errors in dependant code. Excluding this code except
>> >within
>> >the profile doesn't seem to work, and is considered by some to be bad
>>form
>> >because there is nothing inside the jar file that notes which profiles
>> >were
>> >used to compile.
>>
>> Got it. Suggestion here would be:
>>
>> 2. create a new module, cas-resource-mesos, and inside of that module,
>> take one of the following approaches, assuming the module is activated
>> when -Pwith-mesos is enabled:
>>
>> 2a. Maven Antrun like so (in this old example):
>>
>>http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-int
>>o-
>> maven2-build
>>
>> (pro: more flexibility in case protoc isn¹t there; to fail on error; to
>> only compile if
>> protoc is available
>>
>> 2b. Maven protobuf plugin
>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>>
>> Here¹s how to enable a module with a profile:
>>
>>
>>http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-dea
>>ct
>> ive-modules-by-profiles-in-maven/
>>
>>
>> It seems like that is a bad idea though, based on that discussion.
>>
>> So, here¹s another option:
>>
>> 1. Inside of cas-resource (no special new module or anything else)
>> 2. include some custom Ant magic via a build.xml file and the Maven
>> AntRun plugin:
>> 2a. test if protoc is on the system path, and if not, download it,
>>e.g.,
>> into the target directory (gets deleted on clean)
>> 2b. call protoc and compile after 2a
>>
>> I would suggest this solution as I think it¹s the most robust and
>>ensures
>> we always have a cas-resource that includes mesos and compiled
>>correctly.
>>
>> Cheers,
>> Chris
>>
>> >
>> >Any ideas on how to continue?
>> >
>> >Michael
>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann" <ch...@gmail.com>
>> >wrote:
>> >
>> >> Hi Mike,
>> >>
>> >> Great discussion. It would be nice if there was
>> >> a protoc Maven plugin:
>> >>
>> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>> >>
>> >>
>> >> Looks like there is. My suggestion:
>> >>
>> >> 1. use a Profile, something like -Pwith-mesos and
>> >> then when activated;
>> >> 2. call the above plugin if -Pwith-mesos is activated
>> >> in the resource manager
>> >>
>> >> Sound good?
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> ------------------------
>> >> Chris Mattmann
>> >> chris.mattmann@gmail.com
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Michael Starch <st...@umich.edu>
>> >> Reply-To: <de...@oodt.apache.org>
>> >> Date: Wednesday, November 5, 2014 at 11:46 AM
>> >> To: <de...@oodt.apache.org>
>> >> Subject: Extra Compiler Tools
>> >>
>> >> >All,
>> >> >
>> >> >I am trying to integrate apache-mesos with our resource manager.
>> >>However,
>> >> >mesos uses a technology called "protobuff" from Google for
>> >> >marshaling/unmarshaling data.
>> >> >
>> >> >This requires running a tool called "protoc" to generate a source
>>file
>> >>in
>> >> >java. What is the best way to integrate this step into our build
>> >>process?
>> >> >
>> >> >Options I can conceive of:
>> >> > -Check in generated java file
>> >> > -Require "protoc" installation to build resource manager
>> >> > -Separate extra resource package into new module
>> >> >
>> >> >None of these ideas are very clean.
>> >> >
>> >> >Any other ideas? I tried setting up a profile to only compile these
>> >> >sources when selected, but that turned out not to work.
>> >> >
>> >> >-Michael Starch
>> >>
>> >>
>> >>
>>
>>
>>
Re: Extra Compiler Tools
Posted by Michael Starch <st...@umich.edu>.
Looks like you followed the same reasoning chain that I did. Yes, I came
to the same conclusion that ant-build was best.
I wasn't sure how to download protoc, but you just answered that....so I
think this is a great solution!
Thanks,
Michael
On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann <ch...@gmail.com>
wrote:
> Hi Mike,
>
> Thanks for flushing this out.
>
> My thoughts on the below:
>
>
> -----Original Message-----
> From: Michael Starch <st...@umich.edu>
> Reply-To: <de...@oodt.apache.org>
> Date: Wednesday, November 5, 2014 at 12:12 PM
> To: <de...@oodt.apache.org>
> Subject: Re: Extra Compiler Tools
>
> >I tried this approach. The plugin requires a path to the "protoc" tool and
> >thus a working installation. This is what prompted the discussion.
>
> Ah - no worries, what you could do is:
>
> 1. only enable to plugin if -Pwith-mesos is enabled; and
>
> >
> >Running the plugin under a profile works.
>
> Yep.
>
> > However, not running the plugin
> >causes compile errors in dependant code. Excluding this code except
> >within
> >the profile doesn't seem to work, and is considered by some to be bad form
> >because there is nothing inside the jar file that notes which profiles
> >were
> >used to compile.
>
> Got it. Suggestion here would be:
>
> 2. create a new module, cas-resource-mesos, and inside of that module,
> take one of the following approaches, assuming the module is activated
> when -Pwith-mesos is enabled:
>
> 2a. Maven Antrun like so (in this old example):
> http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-into-
> maven2-build
>
> (pro: more flexibility in case protoc isn¹t there; to fail on error; to
> only compile if
> protoc is available
>
> 2b. Maven protobuf plugin
> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>
> Here¹s how to enable a module with a profile:
>
> http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-deact
> ive-modules-by-profiles-in-maven/
>
>
> It seems like that is a bad idea though, based on that discussion.
>
> So, here¹s another option:
>
> 1. Inside of cas-resource (no special new module or anything else)
> 2. include some custom Ant magic via a build.xml file and the Maven
> AntRun plugin:
> 2a. test if protoc is on the system path, and if not, download it, e.g.,
> into the target directory (gets deleted on clean)
> 2b. call protoc and compile after 2a
>
> I would suggest this solution as I think it¹s the most robust and ensures
> we always have a cas-resource that includes mesos and compiled correctly.
>
> Cheers,
> Chris
>
> >
> >Any ideas on how to continue?
> >
> >Michael
> > On Nov 5, 2014 11:04 AM, "Chris Mattmann" <ch...@gmail.com>
> >wrote:
> >
> >> Hi Mike,
> >>
> >> Great discussion. It would be nice if there was
> >> a protoc Maven plugin:
> >>
> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
> >>
> >>
> >> Looks like there is. My suggestion:
> >>
> >> 1. use a Profile, something like -Pwith-mesos and
> >> then when activated;
> >> 2. call the above plugin if -Pwith-mesos is activated
> >> in the resource manager
> >>
> >> Sound good?
> >>
> >> Cheers,
> >> Chris
> >>
> >> ------------------------
> >> Chris Mattmann
> >> chris.mattmann@gmail.com
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Michael Starch <st...@umich.edu>
> >> Reply-To: <de...@oodt.apache.org>
> >> Date: Wednesday, November 5, 2014 at 11:46 AM
> >> To: <de...@oodt.apache.org>
> >> Subject: Extra Compiler Tools
> >>
> >> >All,
> >> >
> >> >I am trying to integrate apache-mesos with our resource manager.
> >>However,
> >> >mesos uses a technology called "protobuff" from Google for
> >> >marshaling/unmarshaling data.
> >> >
> >> >This requires running a tool called "protoc" to generate a source file
> >>in
> >> >java. What is the best way to integrate this step into our build
> >>process?
> >> >
> >> >Options I can conceive of:
> >> > -Check in generated java file
> >> > -Require "protoc" installation to build resource manager
> >> > -Separate extra resource package into new module
> >> >
> >> >None of these ideas are very clean.
> >> >
> >> >Any other ideas? I tried setting up a profile to only compile these
> >> >sources when selected, but that turned out not to work.
> >> >
> >> >-Michael Starch
> >>
> >>
> >>
>
>
>
Re: Extra Compiler Tools
Posted by Chris Mattmann <ch...@gmail.com>.
Hi Mike,
Thanks for flushing this out.
My thoughts on the below:
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: <de...@oodt.apache.org>
Date: Wednesday, November 5, 2014 at 12:12 PM
To: <de...@oodt.apache.org>
Subject: Re: Extra Compiler Tools
>I tried this approach. The plugin requires a path to the "protoc" tool and
>thus a working installation. This is what prompted the discussion.
Ah - no worries, what you could do is:
1. only enable to plugin if -Pwith-mesos is enabled; and
>
>Running the plugin under a profile works.
Yep.
> However, not running the plugin
>causes compile errors in dependant code. Excluding this code except
>within
>the profile doesn't seem to work, and is considered by some to be bad form
>because there is nothing inside the jar file that notes which profiles
>were
>used to compile.
Got it. Suggestion here would be:
2. create a new module, cas-resource-mesos, and inside of that module,
take one of the following approaches, assuming the module is activated
when -Pwith-mesos is enabled:
2a. Maven Antrun like so (in this old example):
http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-into-
maven2-build
(pro: more flexibility in case protoc isn¹t there; to fail on error; to
only compile if
protoc is available
2b. Maven protobuf plugin
http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
Here¹s how to enable a module with a profile:
http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-deact
ive-modules-by-profiles-in-maven/
It seems like that is a bad idea though, based on that discussion.
So, here¹s another option:
1. Inside of cas-resource (no special new module or anything else)
2. include some custom Ant magic via a build.xml file and the Maven
AntRun plugin:
2a. test if protoc is on the system path, and if not, download it, e.g.,
into the target directory (gets deleted on clean)
2b. call protoc and compile after 2a
I would suggest this solution as I think it¹s the most robust and ensures
we always have a cas-resource that includes mesos and compiled correctly.
Cheers,
Chris
>
>Any ideas on how to continue?
>
>Michael
> On Nov 5, 2014 11:04 AM, "Chris Mattmann" <ch...@gmail.com>
>wrote:
>
>> Hi Mike,
>>
>> Great discussion. It would be nice if there was
>> a protoc Maven plugin:
>>
>> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>>
>>
>> Looks like there is. My suggestion:
>>
>> 1. use a Profile, something like -Pwith-mesos and
>> then when activated;
>> 2. call the above plugin if -Pwith-mesos is activated
>> in the resource manager
>>
>> Sound good?
>>
>> Cheers,
>> Chris
>>
>> ------------------------
>> Chris Mattmann
>> chris.mattmann@gmail.com
>>
>>
>>
>>
>> -----Original Message-----
>> From: Michael Starch <st...@umich.edu>
>> Reply-To: <de...@oodt.apache.org>
>> Date: Wednesday, November 5, 2014 at 11:46 AM
>> To: <de...@oodt.apache.org>
>> Subject: Extra Compiler Tools
>>
>> >All,
>> >
>> >I am trying to integrate apache-mesos with our resource manager.
>>However,
>> >mesos uses a technology called "protobuff" from Google for
>> >marshaling/unmarshaling data.
>> >
>> >This requires running a tool called "protoc" to generate a source file
>>in
>> >java. What is the best way to integrate this step into our build
>>process?
>> >
>> >Options I can conceive of:
>> > -Check in generated java file
>> > -Require "protoc" installation to build resource manager
>> > -Separate extra resource package into new module
>> >
>> >None of these ideas are very clean.
>> >
>> >Any other ideas? I tried setting up a profile to only compile these
>> >sources when selected, but that turned out not to work.
>> >
>> >-Michael Starch
>>
>>
>>
Re: Extra Compiler Tools
Posted by Michael Starch <st...@umich.edu>.
I tried this approach. The plugin requires a path to the "protoc" tool and
thus a working installation. This is what prompted the discussion.
Running the plugin under a profile works. However, not running the plugin
causes compile errors in dependant code. Excluding this code except within
the profile doesn't seem to work, and is considered by some to be bad form
because there is nothing inside the jar file that notes which profiles were
used to compile.
Any ideas on how to continue?
Michael
On Nov 5, 2014 11:04 AM, "Chris Mattmann" <ch...@gmail.com> wrote:
> Hi Mike,
>
> Great discussion. It would be nice if there was
> a protoc Maven plugin:
>
> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html
>
>
> Looks like there is. My suggestion:
>
> 1. use a Profile, something like -Pwith-mesos and
> then when activated;
> 2. call the above plugin if -Pwith-mesos is activated
> in the resource manager
>
> Sound good?
>
> Cheers,
> Chris
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Michael Starch <st...@umich.edu>
> Reply-To: <de...@oodt.apache.org>
> Date: Wednesday, November 5, 2014 at 11:46 AM
> To: <de...@oodt.apache.org>
> Subject: Extra Compiler Tools
>
> >All,
> >
> >I am trying to integrate apache-mesos with our resource manager. However,
> >mesos uses a technology called "protobuff" from Google for
> >marshaling/unmarshaling data.
> >
> >This requires running a tool called "protoc" to generate a source file in
> >java. What is the best way to integrate this step into our build process?
> >
> >Options I can conceive of:
> > -Check in generated java file
> > -Require "protoc" installation to build resource manager
> > -Separate extra resource package into new module
> >
> >None of these ideas are very clean.
> >
> >Any other ideas? I tried setting up a profile to only compile these
> >sources when selected, but that turned out not to work.
> >
> >-Michael Starch
>
>
>