Re: Apache Apex as component in Apache Bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

​Hello Bigtop Community,

As a part of second phase of integration, I've created a pull request for
adding smoke test of Apex package to Bigtop that I would like to get
reviewed and committed to Bigtop repository:
https://github.com/apache/bigtop/pull/94

The pull request contains 2 basic tests which validates correctness of the
installed apex package.

If you have some time, could you please review the pull request and provide
the feedback so that we can get these tests merged in?

Thanks a ton!!

- Chinmay.



On Fri, Feb 26, 2016 at 9:22 PM, Konstantin Boudnik <co...@apache.org> wrote:

> On Fri, Feb 26, 2016 at 02:26PM, Chinmay Kolhatkar wrote:
> > Thank you Bigtop community for efficiently consuming the first patch
> > (packing of apex).
>
> yeah, you don't want to see the regurgitation process ;)
>
> > For the second phase, I'll soon be creating a PR for Smoke tests for apex
> > package.
>
> Keep 'em coming and thanks for the contributions!
>   Cos
>
> > On Thu, Feb 25, 2016 at 12:30 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Thanks Jay.
> > > There are already some comments from YoungWoo Kim which are taken care
> of
> > > in that PR.
> > >
> > > On Tue, Feb 23, 2016 at 6:05 PM, Jay Vyas <jayunit100.apache@gmail.com
> >
> > > wrote:
> > >
> > >> Thanks again for this , don't apologize for the email.
> > >>
> > >> We actually are excited about apex, thanks for this! We will get it
> > >> reviewed within the next couple of days I'm sure (I'll look some time
> after
> > >> work if nobody else gets to it first).
> > >>
> > >> > On Feb 23, 2016, at 6:09 AM, Chinmay Kolhatkar <ch...@apache.org>
> > >> wrote:
> > >> >
> > >> > Sorry for another mail. Just want to mention how I tested the patch
> > >> before
> > >> > creating PR:
> > >> >
> > >> > Apex package was built using with following docker images:
> > >> > 1) Ubuntu trusty:
> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> > >> bigtop/slaves:trunk-ubuntu-14.04
> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
> > >> >
> > >> > 2) CentOS 7
> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> bigtop/slaves:trunk-centos-7
> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
> > >> >
> > >> > 3) CentOS 6
> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> bigtop/slaves:trunk-centos-6
> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
> > >> >
> > >> > 4) OpenSUSE 13.2
> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> > >> > bigtop/slaves:trunk-opensuse-13.2 -l -c './gradlew allclean ;
> ./gradlew
> > >> > apex-pkg'
> > >> >
> > >> >
> > >> > Apex package was installed on following docker containers and
> verified
> > >> for
> > >> > sanity:
> > >> > 1) bigtop/slaves:trunk-ubuntu-14.04
> > >> > 2) bigtop/slaves:trunk-centos-6
> > >> > 3) bigtop/slaves:trunk-centos-7
> > >> > 4) bigtop/slaves:trunk-opensuse-13.2
> > >> >
> > >> > When you have some time, please provide the feedback on the pull
> > >> request (
> > >> > https://github.com/apache/bigtop/pull/92)
> > >> >
> > >> > Thank you,
> > >> > Chinmay.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Feb 22, 2016 at 9:43 PM, Chinmay Kolhatkar <
> chinmay@apache.org>
> > >> > wrote:
> > >> >
> > >> >> Hello Bigtop Community,
> > >> >>
> > >> >> I've created a pull request for packing of Apache Apex into bigtop
> that
> > >> >> I'd like to get reviewd and committed to Bigtop repository:
> > >> >> https://github.com/apache/bigtop/pull/92
> > >> >>
> > >> >> The pull request contains code for packing of apex for both deb
> and rpm
> > >> >> packing.
> > >> >> This does not include smoke tests which will be the very next steps
> > >> once
> > >> >> this gets accepted.
> > >> >>
> > >> >> If you have some time, could you please review this pull request
> and
> > >> >> provide feedback so that we can get these merged in?
> > >> >>
> > >> >> Also, there is a question related to what should be included in
> package
> > >> >> with respect to packing of runtime dependent jars.
> > >> >> Currently, I've included license files of non-ASF based licenses in
> > >> >> package.
> > >> >> Is there any specific guidelines from bigtop that integrating
> projects
> > >> >> needs to follow on that front?
> > >> >>
> > >> >> In the meantime, I would like to get some feedback on pull request
> for
> > >> if
> > >> >> there are any code change required.
> > >> >>
> > >> >> Thank you in advance!!
> > >> >>
> > >> >> Chinmay.
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Wed, Feb 17, 2016 at 11:10 AM, Chinmay Kolhatkar <
> > >> chinmay@apache.org>
> > >> >> wrote:
> > >> >>
> > >> >>> Cos,
> > >> >>> Thank for a quick reply. We (apex community) has decided to follow
> > >> your
> > >> >>> recommendation i.e. drop "-incubating".
> > >> >>>
> > >> >>> Roman,
> > >> >>> Apologies, I did not understand your last question earlier.
> > >> >>> Yes, I can certainly provide the prototype for rpm+deb packing of
> Apex
> > >> >>> with bigtop.
> > >> >>> Here is the link to GH where development is WIP:
> > >> >>> https://github.com/chinmaykolhatkar/bigtop/tree/BIGTOP-2314
> > >> >>>
> > >> >>> Thanks,
> > >> >>> Chinmay.
> > >> >>>
> > >> >>>
> > >> >>> On Tue, Feb 16, 2016 at 9:15 PM, Konstantin Boudnik <
> cos@apache.org>
> > >> >>> wrote:
> > >> >>>
> > >> >>>> Dude to known limitations of RPMs "-" aren't allowed for sure.
> Deb
> > >> >>>> doesn't
> > >> >>>> have such idiosyncrasy, but there are ppl who are using RPM,
> which is
> > >> >>>> beyond
> > >> >>>> me. Anyway...
> > >> >>>>
> > >> >>>> This is a long standing issue for us as well, as we have a few
> > >> incubating
> > >> >>>> projects in the stack. My recommendation would be to drop
> > >> "-incubating"
> > >> >>>> part
> > >> >>>> from the package version. Check out how it is done for zeppelin.
> > >> >>>>
> > >> >>>> Cos
> > >> >>>>
> > >> >>>>> On Tue, Feb 16, 2016 at 09:04PM, Chinmay Kolhatkar wrote:
> > >> >>>>> Hello Everyone,
> > >> >>>>>
> > >> >>>>> I'm facing an issue while setting base version for rpm package.
> > >> >>>>> We want to have version of Apache Apex as "3.3.0-incubating".
> > >> >>>>> But when I try to set version in bigtop.bom file, I get error
> while
> > >> >>>>> generating rpm package as rpm does not allow hyphen ('-') in
> > >> version.
> > >> >>>>>
> > >> >>>>> I also tried to set 'incubating' as a value as release: version
> {
> > >> base
> > >> >>>> =
> > >> >>>>> '3.3.0'; pkg = base; release = 'incubating' }
> > >> >>>>> But that doesn't seem to take any effect.
> > >> >>>>>
> > >> >>>>> Can you please suggest how can we achieve this?
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> Chinmay.
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On Mon, Feb 15, 2016 at 11:38 AM, Chinmay Kolhatkar <
> > >> >>>> chinmay@apache.org>
> > >> >>>>> wrote:
> > >> >>>>>
> > >> >>>>>> Hi Roman,
> > >> >>>>>>
> > >> >>>>>> Thanks for the feedback.
> > >> >>>>>> Here are the answers to the question in the same order:
> > >> >>>>>> 1. Apex is a YARN-native platform. Hence apex needs YARN to be
> > >> there.
> > >> >>>>>> 2. conf/ directory recently got removed. This is not needed
> anymore
> > >> >>>> in the
> > >> >>>>>> package.
> > >> >>>>>> 3. Here is the link to GH repo for apex platform:
> > >> >>>>>>     https://github.com/apache/incubator-apex-core
> > >> >>>>>>   The source as of now does not output any package.
> do-component
> > >> >>>> build
> > >> >>>>>> would compile the apex platform and generate a rpm/deb package
> > >> using
> > >> >>>> files
> > >> >>>>>> present in source code. Hence there is no prototype of the
> package.
> > >> >>>>>>   Is that the prototype you're looking for?
> > >> >>>>>>
> > >> >>>>>> Thanks,
> > >> >>>>>> Chinmay.
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Mon, Feb 15, 2016 at 11:11 AM, Roman Shaposhnik <
> > >> >>>> roman@shaposhnik.org>
> > >> >>>>>> wrote:
> > >> >>>>>>
> > >> >>>>>>> Looks good to me. Three questions:
> > >> >>>>>>>   1. Does apex provide standalone daemon or does only run on
> YARN
> > >> ?
> > >> >>>>>>>   2. What's the intended use for content under
> /usr/lib/apex/conf
> > >> ?
> > >> >>>>>>>   3. Any chance you can provide a pointer to GH repo with a
> > >> >>>> prototype?
> > >> >>>>>>>
> > >> >>>>>>> Thanks,
> > >> >>>>>>> Roman.
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Sun, Feb 14, 2016 at 8:36 PM, Chinmay Kolhatkar <
> > >> >>>> chinmay@apache.org>
> > >> >>>>>>> wrote:
> > >> >>>>>>>> Hi Everyone,
> > >> >>>>>>>>
> > >> >>>>>>>> Here is some details regarding packaging apex component into
> > >> >>>> bigtop:
> > >> >>>>>>>> 1. Apex component will dependent on hadoop. This will be
> > >> >>>> mentioned in
> > >> >>>>>>>> bigtop.bom file.
> > >> >>>>>>>> 2. The component added to bom file will look like following:
> > >> >>>>>>>>
> > >> >>>>>>>> 'apex' {
> > >> >>>>>>>>      name    = 'apex'
> > >> >>>>>>>>      relNotes = 'Apache Apex (incubating)'
> > >> >>>>>>>>      version { base = '3.3.0-incubating'; pkg = base;
> release =
> > >> >>>> 1 }
> > >> >>>>>>>>      tarball { source      =
> > >> >>>>>>> "$name-${version.base}-source-release.tar.gz"
> > >> >>>>>>>>                destination = "$name-${version.base}.tar.gz" }
> > >> >>>>>>>>      url     { download_path =
> > >> >>>> "/incubator/$name/v${version.base}/"
> > >> >>>>>>>>                site =
> "${apache.APACHE_MIRROR}/${download_path}"
> > >> >>>>>>>>                archive =
> > >> >>>> "${apache.APACHE_ARCHIVE}/${download_path}" }
> > >> >>>>>>>> }
> > >> >>>>>>>>
> > >> >>>>>>>> 3. After installation of package (deb+rpm), the directory
> > >> >>>> structure
> > >> >>>>>>> related
> > >> >>>>>>>> to apex will look like following:
> > >> >>>>>>>>
> > >> >>>>>>>> /
> > >> >>>>>>>> `-- usr
> > >> >>>>>>>>    |-- bin
> > >> >>>>>>>>    |   |-- apex -> ../lib/apex/bin/apex
> > >> >>>>>>>>    |   `-- dtcli -> ../lib/apex/bin/apex
> > >> >>>>>>>>    |-- lib
> > >> >>>>>>>>    |   `-- apex
> > >> >>>>>>>>    |       |-- bin
> > >> >>>>>>>>    |       |   |-- apex
> > >> >>>>>>>>    |       |   `-- dtcli -> apex
> > >> >>>>>>>>    |       |-- conf
> > >> >>>>>>>>    |       |   `-- dt-env.sh
> > >> >>>>>>>>    |       `-- lib
> > >> >>>>>>>>    |           |-- apex-api-3.3.0-incubating.jar
> > >> >>>>>>>>    |           |-- apex-bufferserver-3.3.0-incubating.jar
> > >> >>>>>>>>    |           |-- apex-common-3.3.0-incubating.jar
> > >> >>>>>>>>    |           |-- apex-engine.jar
> > >> >>>>>>>>    |           `-- <other dependency jars>
> > >> >>>>>>>>    `-- share
> > >> >>>>>>>>        `-- doc
> > >> >>>>>>>>            `-- apex
> > >> >>>>>>>>                |-- changelog.Debian.gz
> > >> >>>>>>>>                |-- copyright
> > >> >>>>>>>>                `-- license
> > >> >>>>>>>>                    `-- LICENSE.gz
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> Please share your thoughts on above approach.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> Thanks,
> > >> >>>>>>>> Chinmay.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> On Sat, Feb 13, 2016 at 8:38 AM, Thomas Weise <
> thw@apache.org>
> > >> >>>> wrote:
> > >> >>>>>>>>
> > >> >>>>>>>>> Chinmay,
> > >> >>>>>>>>>
> > >> >>>>>>>>> Can you share some details regarding the packaging?
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks
> > >> >>>>>>>>>
> > >> >>>>>>>>> On Thu, Feb 11, 2016 at 12:35 AM, Chinmay Kolhatkar <
> > >> >>>>>>>>> chinmay@datatorrent.com
> > >> >>>>>>>>>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>>> Dear Community,
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Here is the Jira that is created for adding Apex into
> Bigtop.
> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/BIGTOP-2313
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I've started working on creating packages for Apex.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Please share your thoughts on approach as well as on
> questions
> > >> >>>>>>> related to
> > >> >>>>>>>>>> packaging in my previous mail.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Thanks,
> > >> >>>>>>>>>> Chinmay.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> On Wed, Feb 10, 2016 at 9:55 PM, Chinmay Kolhatkar <
> > >> >>>>>>>>>> chinmay@datatorrent.com>
> > >> >>>>>>>>>> wrote:
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>> Hello Bigtop Community,
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Continuing with the packing effort, here is how we're
> > >> >>>> planning to
> > >> >>>>>>>>> package
> > >> >>>>>>>>>>> Apache Apex.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Tentative package directory structure for apex would look
> > >> >>>> like
> > >> >>>>>>>>> following:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli
> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-api-3.3.0-incubating.jar
> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-engine-3.3.0-incubating.jar
> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-bufferserver-3.3.0-incubating.jar
> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-common-3.3.0-incubating.jar
> > >> >>>>>>>>>>> /usr/lib/apex/lib/<other dependent jars>
> > >> >>>>>>>>>>> /usr/lib/apex/conf/dt-env.sh
> > >> >>>>>>>>>>> /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > >> >>>>>>>>>>> /usr/share/doc/man/man1/dtcli.1.gz
> > >> >>>>>>>>>>> /usr/share/doc/apex/license/LICENSE.txt
> > >> >>>>>>>>>>> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > >> >>>>>>>>>>> /usr/share/doc/apex/CHANGELOG
> > >> >>>>>>>>>>> /usr/share/doc/apex/NOTICE
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Addition contents to bigtop.bom:
> > >> >>>>>>>>>>> 1. Dependency of apex will be on "hadoop"
> > >> >>>>>>>>>>> 2. Component addition would looks like this:
> > >> >>>>>>>>>>> 'apex' {
> > >> >>>>>>>>>>>      name    = 'apex'
> > >> >>>>>>>>>>>      relNotes = 'Apache Apex (incubating)'
> > >> >>>>>>>>>>>      version { base = '3.3.0-incubating'; pkg = base;
> > >> >>>> release = 1
> > >> >>>>>>> }
> > >> >>>>>>>>>>>      tarball { source      =
> > >> >>>>>>>>>> "$name-${version.base}-source-release.tar.gz"
> > >> >>>>>>>>>>>                destination =
> "$name-${version.base}.tar.gz"
> > >> >>>> }
> > >> >>>>>>>>>>>      url     { download_path =
> > >> >>>>>>> "/incubator/$name/v${version.base}/"
> > >> >>>>>>>>>>>                site =
> > >> >>>> "${apache.APACHE_MIRROR}/${download_path}"
> > >> >>>>>>>>>>>                archive =
> > >> >>>>>>> "${apache.APACHE_ARCHIVE}/${download_path}" }
> > >> >>>>>>>>>>> }
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I have couple of questions related to packaging:
> > >> >>>>>>>>>>> 1. What is the best possible way to test the package
> locally?
> > >> >>>>>>>>>>> 2. Do you have any specific size limitation on size of the
> > >> >>>> package
> > >> >>>>>>> (rpm
> > >> >>>>>>>>>> OR
> > >> >>>>>>>>>>> deb)?
> > >> >>>>>>>>>>> 3. Is above proposed packaging information inline with
> bigtop
> > >> >>>>>>>>> standards?
> > >> >>>>>>>>>>> 4. Am I missing anything there?
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Thanks,
> > >> >>>>>>>>>>> Chinmay.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> On Fri, Feb 5, 2016 at 12:01 AM, Chinmay Kolhatkar <
> > >> >>>>>>>>>>> chinmay@datatorrent.com> wrote:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>> Hello Bigtop community!
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> My name is Chinmay Kolhatkar and I am a committer in
> Apache
> > >> >>>> Apex
> > >> >>>>>>>>>>>> (incubating) project.
> > >> >>>>>>>>>>>> Our website is at: http://apex.incubator.apache.org/
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> We want to explore and integrate Apache Apex into the
> > >> >>>> Bigtop as a
> > >> >>>>>>>>>>>> component.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Please treat this mail as a continuation of a mail
> thread:
> > >> >>>>
> > >>
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201601.mbox/%3CCA%2B5xAo20CAHuYtdxw3mtH2X9whREKd7fLdz4xmuTMqkaTiiskg%40mail.gmail.com%3E
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> We're currently confirming with requirements for
> components
> > >> >>>> and
> > >> >>>>>>> making
> > >> >>>>>>>>>>>> sure that we meet the expectations:
> > >> >>>>
> > >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Once requirements are through, we're planning to add
> > >> >>>> component
> > >> >>>>>>> into
> > >> >>>>>>>>>>>> Apache Bigtop in following phases (as separate
> patches/pull
> > >> >>>>>>> requests):
> > >> >>>>>>>>>>>> 1. Packaging
> > >> >>>>>>>>>>>>    - Creating of package i.e. rpm & deb files.
> > >> >>>>>>>>>>>>    - documentations/READMEs, LICENSE, DISCLAMER, NOTES
> etc
> > >> >>>> if any
> > >> >>>>>>>>>> needed.
> > >> >>>>>>>>>>>>    - Any documentation that need to be added to
> > >> >>>> distribution of
> > >> >>>>>>> our
> > >> >>>>>>>>>>>> software.
> > >> >>>>>>>>>>>>    - Any license information of dependencies required to
> be
> > >> >>>>>>> added to
> > >> >>>>>>>>>>>> package
> > >> >>>>>>>>>>>> 2. Smoke tests
> > >> >>>>>>>>>>>>    - Adding smoke test for packaging.
> > >> >>>>>>>>>>>> 3. Puppet recipes for automatic deployment and
> configuration
> > >> >>>>>>>>>>>>    - Add puppet recipes for automatic deployment
> > >> >>>>>>>>>>>> 4. license clearance:
> > >> >>>>>>>>>>>>    - Run 'gradle rat' to make sure all new stuff is
> > >> >>>> compliant
> > >> >>>>>>> with
> > >> >>>>>>>>> ASF
> > >> >>>>>>>>>>>> license requirements.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Please share your thoughts on above approach.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> If the approach seems fine, we can create a Jira items
> and
> > >> >>>>>>>>> corresponding
> > >> >>>>>>>>>>>> subtasks to start working.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Thanks,
> > >> >>>>>>>>>>>> Chinmay.
> > >> >>
> > >>
> > >
> > >
>

Re: Apache Apex as component in Apache Bigtop

[Konstantin Boudnik <co...@apache.org>]

Lately we've been doing a couple of releases a year, which approximately put
the next one around mid-summer module the will of the community to do it
earlier/later.

Cos

On Fri, Apr 08, 2016 at 11:45AM, nate@reactor8.com wrote:
> Hi Chinmay,
> 
> Great to hear last apex items are ready, will be great to have as a component.
> 
> As far as a next release, there is this jira currently:
> 
> https://issues.apache.org/jira/browse/BIGTOP-2282
> 
> I know besides some fixes and updates, I saw some msgs about flink related
> work to join apex as latest supported components.  Also some talked to some
> folks from openpower community this week at their event and they had wanted
> to fix couple component issues from 1.1.
> 
> Would be good if schedules and efforts and coalesce in time for Apache
> Bigdata timeframe in May, or shortly thereafter.  We have couple bigtop
> presentations at the event so we add new components and updates, including
> apex regardless if release hits before/during or after the event
> 
> 
> -----Original Message-----
> From: Chinmay Kolhatkar [mailto:chinmay@apache.org] 
> Sent: Friday, April 8, 2016 10:43 AM
> To: dev@bigtop.apache.org
> Subject: Re: Apache Apex as component in Apache Bigtop
> 
> Hello Bigtop Community,
> 
> The last PR for adding apex as component is bigtop is merged. This completes the Jira BIGTOP-2313.
> 
> A huge thanks to everyone for accepting the patches and adding apex as a component in bigtop github repository.
> 
> We're excited to share "yum/apt-get install apex" commands with users of apex.
> Any idea on when will be the next release of bigtop and when yum/apt-get repositories will get updated?
> 
> Thanks,
> Chinmay.
> 
> 
> 
> On Fri, Mar 4, 2016 at 12:24 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
> 
> > ​Hello Bigtop Community,
> >
> > As a part of second phase of integration, I've created a pull request 
> > for adding smoke test of Apex package to Bigtop that I would like to 
> > get reviewed and committed to Bigtop repository:
> > https://github.com/apache/bigtop/pull/94
> >
> > The pull request contains 2 basic tests which validates correctness of 
> > the installed apex package.
> >
> > If you have some time, could you please review the pull request and 
> > provide the feedback so that we can get these tests merged in?
> >
> > Thanks a ton!!
> >
> > - Chinmay.
> >
> >
> >
> > On Fri, Feb 26, 2016 at 9:22 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> >
> >> On Fri, Feb 26, 2016 at 02:26PM, Chinmay Kolhatkar wrote:
> >> > Thank you Bigtop community for efficiently consuming the first 
> >> > patch (packing of apex).
> >>
> >> yeah, you don't want to see the regurgitation process ;)
> >>
> >> > For the second phase, I'll soon be creating a PR for Smoke tests 
> >> > for
> >> apex
> >> > package.
> >>
> >> Keep 'em coming and thanks for the contributions!
> >>   Cos
> >>
> >> > On Thu, Feb 25, 2016 at 12:30 PM, Chinmay Kolhatkar 
> >> > <chinmay@apache.org
> >> >
> >> > wrote:
> >> >
> >> > > Thanks Jay.
> >> > > There are already some comments from YoungWoo Kim which are taken
> >> care of
> >> > > in that PR.
> >> > >
> >> > > On Tue, Feb 23, 2016 at 6:05 PM, Jay Vyas <
> >> jayunit100.apache@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> Thanks again for this , don't apologize for the email.
> >> > >>
> >> > >> We actually are excited about apex, thanks for this! We will get 
> >> > >> it reviewed within the next couple of days I'm sure (I'll look 
> >> > >> some
> >> time after
> >> > >> work if nobody else gets to it first).
> >> > >>
> >> > >> > On Feb 23, 2016, at 6:09 AM, Chinmay Kolhatkar 
> >> > >> > <chinmay@apache.org
> >> >
> >> > >> wrote:
> >> > >> >
> >> > >> > Sorry for another mail. Just want to mention how I tested the 
> >> > >> > patch
> >> > >> before
> >> > >> > creating PR:
> >> > >> >
> >> > >> > Apex package was built using with following docker images:
> >> > >> > 1) Ubuntu trusty:
> >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> >> > >> bigtop/slaves:trunk-ubuntu-14.04
> >> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
> >> > >> >
> >> > >> > 2) CentOS 7
> >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> >> bigtop/slaves:trunk-centos-7
> >> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
> >> > >> >
> >> > >> > 3) CentOS 6
> >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> >> bigtop/slaves:trunk-centos-6
> >> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
> >> > >> >
> >> > >> > 4) OpenSUSE 13.2
> >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> >> > >> > bigtop/slaves:trunk-opensuse-13.2 -l -c './gradlew allclean ;
> >> ./gradlew
> >> > >> > apex-pkg'
> >> > >> >
> >> > >> >
> >> > >> > Apex package was installed on following docker containers and
> >> verified
> >> > >> for
> >> > >> > sanity:
> >> > >> > 1) bigtop/slaves:trunk-ubuntu-14.04
> >> > >> > 2) bigtop/slaves:trunk-centos-6
> >> > >> > 3) bigtop/slaves:trunk-centos-7
> >> > >> > 4) bigtop/slaves:trunk-opensuse-13.2
> >> > >> >
> >> > >> > When you have some time, please provide the feedback on the 
> >> > >> > pull
> >> > >> request (
> >> > >> > https://github.com/apache/bigtop/pull/92)
> >> > >> >
> >> > >> > Thank you,
> >> > >> > Chinmay.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Mon, Feb 22, 2016 at 9:43 PM, Chinmay Kolhatkar <
> >> chinmay@apache.org>
> >> > >> > wrote:
> >> > >> >
> >> > >> >> Hello Bigtop Community,
> >> > >> >>
> >> > >> >> I've created a pull request for packing of Apache Apex into
> >> bigtop that
> >> > >> >> I'd like to get reviewd and committed to Bigtop repository:
> >> > >> >> https://github.com/apache/bigtop/pull/92
> >> > >> >>
> >> > >> >> The pull request contains code for packing of apex for both 
> >> > >> >> deb
> >> and rpm
> >> > >> >> packing.
> >> > >> >> This does not include smoke tests which will be the very next
> >> steps
> >> > >> once
> >> > >> >> this gets accepted.
> >> > >> >>
> >> > >> >> If you have some time, could you please review this pull 
> >> > >> >> request
> >> and
> >> > >> >> provide feedback so that we can get these merged in?
> >> > >> >>
> >> > >> >> Also, there is a question related to what should be included 
> >> > >> >> in
> >> package
> >> > >> >> with respect to packing of runtime dependent jars.
> >> > >> >> Currently, I've included license files of non-ASF based 
> >> > >> >> licenses
> >> in
> >> > >> >> package.
> >> > >> >> Is there any specific guidelines from bigtop that integrating
> >> projects
> >> > >> >> needs to follow on that front?
> >> > >> >>
> >> > >> >> In the meantime, I would like to get some feedback on pull
> >> request for
> >> > >> if
> >> > >> >> there are any code change required.
> >> > >> >>
> >> > >> >> Thank you in advance!!
> >> > >> >>
> >> > >> >> Chinmay.
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >> On Wed, Feb 17, 2016 at 11:10 AM, Chinmay Kolhatkar <
> >> > >> chinmay@apache.org>
> >> > >> >> wrote:
> >> > >> >>
> >> > >> >>> Cos,
> >> > >> >>> Thank for a quick reply. We (apex community) has decided to
> >> follow
> >> > >> your
> >> > >> >>> recommendation i.e. drop "-incubating".
> >> > >> >>>
> >> > >> >>> Roman,
> >> > >> >>> Apologies, I did not understand your last question earlier.
> >> > >> >>> Yes, I can certainly provide the prototype for rpm+deb 
> >> > >> >>> packing
> >> of Apex
> >> > >> >>> with bigtop.
> >> > >> >>> Here is the link to GH where development is WIP:
> >> > >> >>> https://github.com/chinmaykolhatkar/bigtop/tree/BIGTOP-2314
> >> > >> >>>
> >> > >> >>> Thanks,
> >> > >> >>> Chinmay.
> >> > >> >>>
> >> > >> >>>
> >> > >> >>> On Tue, Feb 16, 2016 at 9:15 PM, Konstantin Boudnik <
> >> cos@apache.org>
> >> > >> >>> wrote:
> >> > >> >>>
> >> > >> >>>> Dude to known limitations of RPMs "-" aren't allowed for sure.
> >> Deb
> >> > >> >>>> doesn't
> >> > >> >>>> have such idiosyncrasy, but there are ppl who are using 
> >> > >> >>>> RPM,
> >> which is
> >> > >> >>>> beyond
> >> > >> >>>> me. Anyway...
> >> > >> >>>>
> >> > >> >>>> This is a long standing issue for us as well, as we have a 
> >> > >> >>>> few
> >> > >> incubating
> >> > >> >>>> projects in the stack. My recommendation would be to drop
> >> > >> "-incubating"
> >> > >> >>>> part
> >> > >> >>>> from the package version. Check out how it is done for zeppelin.
> >> > >> >>>>
> >> > >> >>>> Cos
> >> > >> >>>>
> >> > >> >>>>> On Tue, Feb 16, 2016 at 09:04PM, Chinmay Kolhatkar wrote:
> >> > >> >>>>> Hello Everyone,
> >> > >> >>>>>
> >> > >> >>>>> I'm facing an issue while setting base version for rpm package.
> >> > >> >>>>> We want to have version of Apache Apex as "3.3.0-incubating".
> >> > >> >>>>> But when I try to set version in bigtop.bom file, I get 
> >> > >> >>>>> error
> >> while
> >> > >> >>>>> generating rpm package as rpm does not allow hyphen ('-') 
> >> > >> >>>>> in
> >> > >> version.
> >> > >> >>>>>
> >> > >> >>>>> I also tried to set 'incubating' as a value as release:
> >> version {
> >> > >> base
> >> > >> >>>> =
> >> > >> >>>>> '3.3.0'; pkg = base; release = 'incubating' } But that 
> >> > >> >>>>> doesn't seem to take any effect.
> >> > >> >>>>>
> >> > >> >>>>> Can you please suggest how can we achieve this?
> >> > >> >>>>>
> >> > >> >>>>> Thanks,
> >> > >> >>>>> Chinmay.
> >> > >> >>>>>
> >> > >> >>>>>
> >> > >> >>>>> On Mon, Feb 15, 2016 at 11:38 AM, Chinmay Kolhatkar <
> >> > >> >>>> chinmay@apache.org>
> >> > >> >>>>> wrote:
> >> > >> >>>>>
> >> > >> >>>>>> Hi Roman,
> >> > >> >>>>>>
> >> > >> >>>>>> Thanks for the feedback.
> >> > >> >>>>>> Here are the answers to the question in the same order:
> >> > >> >>>>>> 1. Apex is a YARN-native platform. Hence apex needs YARN 
> >> > >> >>>>>> to be
> >> > >> there.
> >> > >> >>>>>> 2. conf/ directory recently got removed. This is not 
> >> > >> >>>>>> needed
> >> anymore
> >> > >> >>>> in the
> >> > >> >>>>>> package.
> >> > >> >>>>>> 3. Here is the link to GH repo for apex platform:
> >> > >> >>>>>>     https://github.com/apache/incubator-apex-core
> >> > >> >>>>>>   The source as of now does not output any package.
> >> do-component
> >> > >> >>>> build
> >> > >> >>>>>> would compile the apex platform and generate a rpm/deb 
> >> > >> >>>>>> package
> >> > >> using
> >> > >> >>>> files
> >> > >> >>>>>> present in source code. Hence there is no prototype of 
> >> > >> >>>>>> the
> >> package.
> >> > >> >>>>>>   Is that the prototype you're looking for?
> >> > >> >>>>>>
> >> > >> >>>>>> Thanks,
> >> > >> >>>>>> Chinmay.
> >> > >> >>>>>>
> >> > >> >>>>>>
> >> > >> >>>>>> On Mon, Feb 15, 2016 at 11:11 AM, Roman Shaposhnik <
> >> > >> >>>> roman@shaposhnik.org>
> >> > >> >>>>>> wrote:
> >> > >> >>>>>>
> >> > >> >>>>>>> Looks good to me. Three questions:
> >> > >> >>>>>>>   1. Does apex provide standalone daemon or does only 
> >> > >> >>>>>>> run on
> >> YARN
> >> > >> ?
> >> > >> >>>>>>>   2. What's the intended use for content under
> >> /usr/lib/apex/conf
> >> > >> ?
> >> > >> >>>>>>>   3. Any chance you can provide a pointer to GH repo 
> >> > >> >>>>>>> with a
> >> > >> >>>> prototype?
> >> > >> >>>>>>>
> >> > >> >>>>>>> Thanks,
> >> > >> >>>>>>> Roman.
> >> > >> >>>>>>>
> >> > >> >>>>>>>
> >> > >> >>>>>>> On Sun, Feb 14, 2016 at 8:36 PM, Chinmay Kolhatkar <
> >> > >> >>>> chinmay@apache.org>
> >> > >> >>>>>>> wrote:
> >> > >> >>>>>>>> Hi Everyone,
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> Here is some details regarding packaging apex component 
> >> > >> >>>>>>>> into
> >> > >> >>>> bigtop:
> >> > >> >>>>>>>> 1. Apex component will dependent on hadoop. This will 
> >> > >> >>>>>>>> be
> >> > >> >>>> mentioned in
> >> > >> >>>>>>>> bigtop.bom file.
> >> > >> >>>>>>>> 2. The component added to bom file will look like following:
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> 'apex' {
> >> > >> >>>>>>>>      name    = 'apex'
> >> > >> >>>>>>>>      relNotes = 'Apache Apex (incubating)'
> >> > >> >>>>>>>>      version { base = '3.3.0-incubating'; pkg = base;
> >> release =
> >> > >> >>>> 1 }
> >> > >> >>>>>>>>      tarball { source      =
> >> > >> >>>>>>> "$name-${version.base}-source-release.tar.gz"
> >> > >> >>>>>>>>                destination = "$name-${version.base}.tar.gz"
> >> }
> >> > >> >>>>>>>>      url     { download_path =
> >> > >> >>>> "/incubator/$name/v${version.base}/"
> >> > >> >>>>>>>>                site =
> >> "${apache.APACHE_MIRROR}/${download_path}"
> >> > >> >>>>>>>>                archive =
> >> > >> >>>> "${apache.APACHE_ARCHIVE}/${download_path}" }
> >> > >> >>>>>>>> }
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> 3. After installation of package (deb+rpm), the 
> >> > >> >>>>>>>> directory
> >> > >> >>>> structure
> >> > >> >>>>>>> related
> >> > >> >>>>>>>> to apex will look like following:
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> /
> >> > >> >>>>>>>> `-- usr
> >> > >> >>>>>>>>    |-- bin
> >> > >> >>>>>>>>    |   |-- apex -> ../lib/apex/bin/apex
> >> > >> >>>>>>>>    |   `-- dtcli -> ../lib/apex/bin/apex
> >> > >> >>>>>>>>    |-- lib
> >> > >> >>>>>>>>    |   `-- apex
> >> > >> >>>>>>>>    |       |-- bin
> >> > >> >>>>>>>>    |       |   |-- apex
> >> > >> >>>>>>>>    |       |   `-- dtcli -> apex
> >> > >> >>>>>>>>    |       |-- conf
> >> > >> >>>>>>>>    |       |   `-- dt-env.sh
> >> > >> >>>>>>>>    |       `-- lib
> >> > >> >>>>>>>>    |           |-- apex-api-3.3.0-incubating.jar
> >> > >> >>>>>>>>    |           |-- apex-bufferserver-3.3.0-incubating.jar
> >> > >> >>>>>>>>    |           |-- apex-common-3.3.0-incubating.jar
> >> > >> >>>>>>>>    |           |-- apex-engine.jar
> >> > >> >>>>>>>>    |           `-- <other dependency jars>
> >> > >> >>>>>>>>    `-- share
> >> > >> >>>>>>>>        `-- doc
> >> > >> >>>>>>>>            `-- apex
> >> > >> >>>>>>>>                |-- changelog.Debian.gz
> >> > >> >>>>>>>>                |-- copyright
> >> > >> >>>>>>>>                `-- license
> >> > >> >>>>>>>>                    `-- LICENSE.gz
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> Please share your thoughts on above approach.
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> Thanks,
> >> > >> >>>>>>>> Chinmay.
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> On Sat, Feb 13, 2016 at 8:38 AM, Thomas Weise <
> >> thw@apache.org>
> >> > >> >>>> wrote:
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>> Chinmay,
> >> > >> >>>>>>>>>
> >> > >> >>>>>>>>> Can you share some details regarding the packaging?
> >> > >> >>>>>>>>>
> >> > >> >>>>>>>>> Thanks
> >> > >> >>>>>>>>>
> >> > >> >>>>>>>>> On Thu, Feb 11, 2016 at 12:35 AM, Chinmay Kolhatkar < 
> >> > >> >>>>>>>>> chinmay@datatorrent.com
> >> > >> >>>>>>>>>> wrote:
> >> > >> >>>>>>>>>
> >> > >> >>>>>>>>>> Dear Community,
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>> Here is the Jira that is created for adding Apex into
> >> Bigtop.
> >> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/BIGTOP-2313
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>> I've started working on creating packages for Apex.
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>> Please share your thoughts on approach as well as on
> >> questions
> >> > >> >>>>>>> related to
> >> > >> >>>>>>>>>> packaging in my previous mail.
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>> Thanks,
> >> > >> >>>>>>>>>> Chinmay.
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>> On Wed, Feb 10, 2016 at 9:55 PM, Chinmay Kolhatkar < 
> >> > >> >>>>>>>>>> chinmay@datatorrent.com>
> >> > >> >>>>>>>>>> wrote:
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>>> Hello Bigtop Community,
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> Continuing with the packing effort, here is how 
> >> > >> >>>>>>>>>>> we're
> >> > >> >>>> planning to
> >> > >> >>>>>>>>> package
> >> > >> >>>>>>>>>>> Apache Apex.
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> Tentative package directory structure for apex would 
> >> > >> >>>>>>>>>>> look
> >> > >> >>>> like
> >> > >> >>>>>>>>> following:
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli 
> >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-api-3.3.0-incubating.jar
> >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-engine-3.3.0-incubating.jar
> >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-bufferserver-3.3.0-incubating
> >> > >> >>>>>>>>>>> .jar 
> >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-common-3.3.0-incubating.jar
> >> > >> >>>>>>>>>>> /usr/lib/apex/lib/<other dependent jars> 
> >> > >> >>>>>>>>>>> /usr/lib/apex/conf/dt-env.sh /usr/bin/dtcli -> 
> >> > >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli 
> >> > >> >>>>>>>>>>> /usr/share/doc/man/man1/dtcli.1.gz
> >> > >> >>>>>>>>>>> /usr/share/doc/apex/license/LICENSE.txt
> >> > >> >>>>>>>>>>> /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > >> >>>>>>>>>>> /usr/share/doc/apex/CHANGELOG 
> >> > >> >>>>>>>>>>> /usr/share/doc/apex/NOTICE
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> Addition contents to bigtop.bom:
> >> > >> >>>>>>>>>>> 1. Dependency of apex will be on "hadoop"
> >> > >> >>>>>>>>>>> 2. Component addition would looks like this:
> >> > >> >>>>>>>>>>> 'apex' {
> >> > >> >>>>>>>>>>>      name    = 'apex'
> >> > >> >>>>>>>>>>>      relNotes = 'Apache Apex (incubating)'
> >> > >> >>>>>>>>>>>      version { base = '3.3.0-incubating'; pkg = 
> >> > >> >>>>>>>>>>> base;
> >> > >> >>>> release = 1
> >> > >> >>>>>>> }
> >> > >> >>>>>>>>>>>      tarball { source      =
> >> > >> >>>>>>>>>> "$name-${version.base}-source-release.tar.gz"
> >> > >> >>>>>>>>>>>                destination =
> >> "$name-${version.base}.tar.gz"
> >> > >> >>>> }
> >> > >> >>>>>>>>>>>      url     { download_path =
> >> > >> >>>>>>> "/incubator/$name/v${version.base}/"
> >> > >> >>>>>>>>>>>                site =
> >> > >> >>>> "${apache.APACHE_MIRROR}/${download_path}"
> >> > >> >>>>>>>>>>>                archive =
> >> > >> >>>>>>> "${apache.APACHE_ARCHIVE}/${download_path}" }
> >> > >> >>>>>>>>>>> }
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> I have couple of questions related to packaging:
> >> > >> >>>>>>>>>>> 1. What is the best possible way to test the package
> >> locally?
> >> > >> >>>>>>>>>>> 2. Do you have any specific size limitation on size 
> >> > >> >>>>>>>>>>> of
> >> the
> >> > >> >>>> package
> >> > >> >>>>>>> (rpm
> >> > >> >>>>>>>>>> OR
> >> > >> >>>>>>>>>>> deb)?
> >> > >> >>>>>>>>>>> 3. Is above proposed packaging information inline 
> >> > >> >>>>>>>>>>> with
> >> bigtop
> >> > >> >>>>>>>>> standards?
> >> > >> >>>>>>>>>>> 4. Am I missing anything there?
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> Thanks,
> >> > >> >>>>>>>>>>> Chinmay.
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> On Fri, Feb 5, 2016 at 12:01 AM, Chinmay Kolhatkar < 
> >> > >> >>>>>>>>>>> chinmay@datatorrent.com> wrote:
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>>> Hello Bigtop community!
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> My name is Chinmay Kolhatkar and I am a committer 
> >> > >> >>>>>>>>>>>> in
> >> Apache
> >> > >> >>>> Apex
> >> > >> >>>>>>>>>>>> (incubating) project.
> >> > >> >>>>>>>>>>>> Our website is at: 
> >> > >> >>>>>>>>>>>> http://apex.incubator.apache.org/
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> We want to explore and integrate Apache Apex into 
> >> > >> >>>>>>>>>>>> the
> >> > >> >>>> Bigtop as a
> >> > >> >>>>>>>>>>>> component.
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> Please treat this mail as a continuation of a mail
> >> thread:
> >> > >> >>>>
> >> > >>
> >> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201601.mbox/%3CCA
> >> %2B5xAo20CAHuYtdxw3mtH2X9whREKd7fLdz4xmuTMqkaTiiskg%40mail.gmail.com%
> >> 3E
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> We're currently confirming with requirements for
> >> components
> >> > >> >>>> and
> >> > >> >>>>>>> making
> >> > >> >>>>>>>>>>>> sure that we meet the expectations:
> >> > >> >>>>
> >> > >>
> >> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+ad
> >> ding+a+new+component+to+Bigtop+distribution
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> Once requirements are through, we're planning to 
> >> > >> >>>>>>>>>>>> add
> >> > >> >>>> component
> >> > >> >>>>>>> into
> >> > >> >>>>>>>>>>>> Apache Bigtop in following phases (as separate
> >> patches/pull
> >> > >> >>>>>>> requests):
> >> > >> >>>>>>>>>>>> 1. Packaging
> >> > >> >>>>>>>>>>>>    - Creating of package i.e. rpm & deb files.
> >> > >> >>>>>>>>>>>>    - documentations/READMEs, LICENSE, DISCLAMER, 
> >> > >> >>>>>>>>>>>> NOTES
> >> etc
> >> > >> >>>> if any
> >> > >> >>>>>>>>>> needed.
> >> > >> >>>>>>>>>>>>    - Any documentation that need to be added to
> >> > >> >>>> distribution of
> >> > >> >>>>>>> our
> >> > >> >>>>>>>>>>>> software.
> >> > >> >>>>>>>>>>>>    - Any license information of dependencies 
> >> > >> >>>>>>>>>>>> required
> >> to be
> >> > >> >>>>>>> added to
> >> > >> >>>>>>>>>>>> package
> >> > >> >>>>>>>>>>>> 2. Smoke tests
> >> > >> >>>>>>>>>>>>    - Adding smoke test for packaging.
> >> > >> >>>>>>>>>>>> 3. Puppet recipes for automatic deployment and
> >> configuration
> >> > >> >>>>>>>>>>>>    - Add puppet recipes for automatic deployment 4. 
> >> > >> >>>>>>>>>>>> license clearance:
> >> > >> >>>>>>>>>>>>    - Run 'gradle rat' to make sure all new stuff is
> >> > >> >>>> compliant
> >> > >> >>>>>>> with
> >> > >> >>>>>>>>> ASF
> >> > >> >>>>>>>>>>>> license requirements.
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> Please share your thoughts on above approach.
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> If the approach seems fine, we can create a Jira 
> >> > >> >>>>>>>>>>>> items
> >> and
> >> > >> >>>>>>>>> corresponding
> >> > >> >>>>>>>>>>>> subtasks to start working.
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> Thanks,
> >> > >> >>>>>>>>>>>> Chinmay.
> >> > >> >>
> >> > >>
> >> > >
> >> > >
> >>
> >
> >
> 

RE: Apache Apex as component in Apache Bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Thanks for the information.

It's great to hear bigtop sharing the information about new components.
Do let us know if you need any content about apex.

Thanks,
Chinmay.

---
Sent from mobile.
On 9 Apr 2016 12:16 a.m., <na...@reactor8.com> wrote:

> Hi Chinmay,
>
> Great to hear last apex items are ready, will be great to have as a
> component.
>
> As far as a next release, there is this jira currently:
>
> https://issues.apache.org/jira/browse/BIGTOP-2282
>
> I know besides some fixes and updates, I saw some msgs about flink related
> work to join apex as latest supported components.  Also some talked to some
> folks from openpower community this week at their event and they had wanted
> to fix couple component issues from 1.1.
>
> Would be good if schedules and efforts and coalesce in time for Apache
> Bigdata timeframe in May, or shortly thereafter.  We have couple bigtop
> presentations at the event so we add new components and updates, including
> apex regardless if release hits before/during or after the event
>
>
> -----Original Message-----
> From: Chinmay Kolhatkar [mailto:chinmay@apache.org]
> Sent: Friday, April 8, 2016 10:43 AM
> To: dev@bigtop.apache.org
> Subject: Re: Apache Apex as component in Apache Bigtop
>
> Hello Bigtop Community,
>
> The last PR for adding apex as component is bigtop is merged. This
> completes the Jira BIGTOP-2313.
>
> A huge thanks to everyone for accepting the patches and adding apex as a
> component in bigtop github repository.
>
> We're excited to share "yum/apt-get install apex" commands with users of
> apex.
> Any idea on when will be the next release of bigtop and when yum/apt-get
> repositories will get updated?
>
> Thanks,
> Chinmay.
>
>
>
> On Fri, Mar 4, 2016 at 12:24 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > ​Hello Bigtop Community,
> >
> > As a part of second phase of integration, I've created a pull request
> > for adding smoke test of Apex package to Bigtop that I would like to
> > get reviewed and committed to Bigtop repository:
> > https://github.com/apache/bigtop/pull/94
> >
> > The pull request contains 2 basic tests which validates correctness of
> > the installed apex package.
> >
> > If you have some time, could you please review the pull request and
> > provide the feedback so that we can get these tests merged in?
> >
> > Thanks a ton!!
> >
> > - Chinmay.
> >
> >
> >
> > On Fri, Feb 26, 2016 at 9:22 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> >
> >> On Fri, Feb 26, 2016 at 02:26PM, Chinmay Kolhatkar wrote:
> >> > Thank you Bigtop community for efficiently consuming the first
> >> > patch (packing of apex).
> >>
> >> yeah, you don't want to see the regurgitation process ;)
> >>
> >> > For the second phase, I'll soon be creating a PR for Smoke tests
> >> > for
> >> apex
> >> > package.
> >>
> >> Keep 'em coming and thanks for the contributions!
> >>   Cos
> >>
> >> > On Thu, Feb 25, 2016 at 12:30 PM, Chinmay Kolhatkar
> >> > <chinmay@apache.org
> >> >
> >> > wrote:
> >> >
> >> > > Thanks Jay.
> >> > > There are already some comments from YoungWoo Kim which are taken
> >> care of
> >> > > in that PR.
> >> > >
> >> > > On Tue, Feb 23, 2016 at 6:05 PM, Jay Vyas <
> >> jayunit100.apache@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> Thanks again for this , don't apologize for the email.
> >> > >>
> >> > >> We actually are excited about apex, thanks for this! We will get
> >> > >> it reviewed within the next couple of days I'm sure (I'll look
> >> > >> some
> >> time after
> >> > >> work if nobody else gets to it first).
> >> > >>
> >> > >> > On Feb 23, 2016, at 6:09 AM, Chinmay Kolhatkar
> >> > >> > <chinmay@apache.org
> >> >
> >> > >> wrote:
> >> > >> >
> >> > >> > Sorry for another mail. Just want to mention how I tested the
> >> > >> > patch
> >> > >> before
> >> > >> > creating PR:
> >> > >> >
> >> > >> > Apex package was built using with following docker images:
> >> > >> > 1) Ubuntu trusty:
> >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> >> > >> bigtop/slaves:trunk-ubuntu-14.04
> >> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
> >> > >> >
> >> > >> > 2) CentOS 7
> >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> >> bigtop/slaves:trunk-centos-7
> >> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
> >> > >> >
> >> > >> > 3) CentOS 6
> >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> >> bigtop/slaves:trunk-centos-6
> >> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
> >> > >> >
> >> > >> > 4) OpenSUSE 13.2
> >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
> >> > >> > bigtop/slaves:trunk-opensuse-13.2 -l -c './gradlew allclean ;
> >> ./gradlew
> >> > >> > apex-pkg'
> >> > >> >
> >> > >> >
> >> > >> > Apex package was installed on following docker containers and
> >> verified
> >> > >> for
> >> > >> > sanity:
> >> > >> > 1) bigtop/slaves:trunk-ubuntu-14.04
> >> > >> > 2) bigtop/slaves:trunk-centos-6
> >> > >> > 3) bigtop/slaves:trunk-centos-7
> >> > >> > 4) bigtop/slaves:trunk-opensuse-13.2
> >> > >> >
> >> > >> > When you have some time, please provide the feedback on the
> >> > >> > pull
> >> > >> request (
> >> > >> > https://github.com/apache/bigtop/pull/92)
> >> > >> >
> >> > >> > Thank you,
> >> > >> > Chinmay.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Mon, Feb 22, 2016 at 9:43 PM, Chinmay Kolhatkar <
> >> chinmay@apache.org>
> >> > >> > wrote:
> >> > >> >
> >> > >> >> Hello Bigtop Community,
> >> > >> >>
> >> > >> >> I've created a pull request for packing of Apache Apex into
> >> bigtop that
> >> > >> >> I'd like to get reviewd and committed to Bigtop repository:
> >> > >> >> https://github.com/apache/bigtop/pull/92
> >> > >> >>
> >> > >> >> The pull request contains code for packing of apex for both
> >> > >> >> deb
> >> and rpm
> >> > >> >> packing.
> >> > >> >> This does not include smoke tests which will be the very next
> >> steps
> >> > >> once
> >> > >> >> this gets accepted.
> >> > >> >>
> >> > >> >> If you have some time, could you please review this pull
> >> > >> >> request
> >> and
> >> > >> >> provide feedback so that we can get these merged in?
> >> > >> >>
> >> > >> >> Also, there is a question related to what should be included
> >> > >> >> in
> >> package
> >> > >> >> with respect to packing of runtime dependent jars.
> >> > >> >> Currently, I've included license files of non-ASF based
> >> > >> >> licenses
> >> in
> >> > >> >> package.
> >> > >> >> Is there any specific guidelines from bigtop that integrating
> >> projects
> >> > >> >> needs to follow on that front?
> >> > >> >>
> >> > >> >> In the meantime, I would like to get some feedback on pull
> >> request for
> >> > >> if
> >> > >> >> there are any code change required.
> >> > >> >>
> >> > >> >> Thank you in advance!!
> >> > >> >>
> >> > >> >> Chinmay.
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >> On Wed, Feb 17, 2016 at 11:10 AM, Chinmay Kolhatkar <
> >> > >> chinmay@apache.org>
> >> > >> >> wrote:
> >> > >> >>
> >> > >> >>> Cos,
> >> > >> >>> Thank for a quick reply. We (apex community) has decided to
> >> follow
> >> > >> your
> >> > >> >>> recommendation i.e. drop "-incubating".
> >> > >> >>>
> >> > >> >>> Roman,
> >> > >> >>> Apologies, I did not understand your last question earlier.
> >> > >> >>> Yes, I can certainly provide the prototype for rpm+deb
> >> > >> >>> packing
> >> of Apex
> >> > >> >>> with bigtop.
> >> > >> >>> Here is the link to GH where development is WIP:
> >> > >> >>> https://github.com/chinmaykolhatkar/bigtop/tree/BIGTOP-2314
> >> > >> >>>
> >> > >> >>> Thanks,
> >> > >> >>> Chinmay.
> >> > >> >>>
> >> > >> >>>
> >> > >> >>> On Tue, Feb 16, 2016 at 9:15 PM, Konstantin Boudnik <
> >> cos@apache.org>
> >> > >> >>> wrote:
> >> > >> >>>
> >> > >> >>>> Dude to known limitations of RPMs "-" aren't allowed for sure.
> >> Deb
> >> > >> >>>> doesn't
> >> > >> >>>> have such idiosyncrasy, but there are ppl who are using
> >> > >> >>>> RPM,
> >> which is
> >> > >> >>>> beyond
> >> > >> >>>> me. Anyway...
> >> > >> >>>>
> >> > >> >>>> This is a long standing issue for us as well, as we have a
> >> > >> >>>> few
> >> > >> incubating
> >> > >> >>>> projects in the stack. My recommendation would be to drop
> >> > >> "-incubating"
> >> > >> >>>> part
> >> > >> >>>> from the package version. Check out how it is done for
> zeppelin.
> >> > >> >>>>
> >> > >> >>>> Cos
> >> > >> >>>>
> >> > >> >>>>> On Tue, Feb 16, 2016 at 09:04PM, Chinmay Kolhatkar wrote:
> >> > >> >>>>> Hello Everyone,
> >> > >> >>>>>
> >> > >> >>>>> I'm facing an issue while setting base version for rpm
> package.
> >> > >> >>>>> We want to have version of Apache Apex as "3.3.0-incubating".
> >> > >> >>>>> But when I try to set version in bigtop.bom file, I get
> >> > >> >>>>> error
> >> while
> >> > >> >>>>> generating rpm package as rpm does not allow hyphen ('-')
> >> > >> >>>>> in
> >> > >> version.
> >> > >> >>>>>
> >> > >> >>>>> I also tried to set 'incubating' as a value as release:
> >> version {
> >> > >> base
> >> > >> >>>> =
> >> > >> >>>>> '3.3.0'; pkg = base; release = 'incubating' } But that
> >> > >> >>>>> doesn't seem to take any effect.
> >> > >> >>>>>
> >> > >> >>>>> Can you please suggest how can we achieve this?
> >> > >> >>>>>
> >> > >> >>>>> Thanks,
> >> > >> >>>>> Chinmay.
> >> > >> >>>>>
> >> > >> >>>>>
> >> > >> >>>>> On Mon, Feb 15, 2016 at 11:38 AM, Chinmay Kolhatkar <
> >> > >> >>>> chinmay@apache.org>
> >> > >> >>>>> wrote:
> >> > >> >>>>>
> >> > >> >>>>>> Hi Roman,
> >> > >> >>>>>>
> >> > >> >>>>>> Thanks for the feedback.
> >> > >> >>>>>> Here are the answers to the question in the same order:
> >> > >> >>>>>> 1. Apex is a YARN-native platform. Hence apex needs YARN
> >> > >> >>>>>> to be
> >> > >> there.
> >> > >> >>>>>> 2. conf/ directory recently got removed. This is not
> >> > >> >>>>>> needed
> >> anymore
> >> > >> >>>> in the
> >> > >> >>>>>> package.
> >> > >> >>>>>> 3. Here is the link to GH repo for apex platform:
> >> > >> >>>>>>     https://github.com/apache/incubator-apex-core
> >> > >> >>>>>>   The source as of now does not output any package.
> >> do-component
> >> > >> >>>> build
> >> > >> >>>>>> would compile the apex platform and generate a rpm/deb
> >> > >> >>>>>> package
> >> > >> using
> >> > >> >>>> files
> >> > >> >>>>>> present in source code. Hence there is no prototype of
> >> > >> >>>>>> the
> >> package.
> >> > >> >>>>>>   Is that the prototype you're looking for?
> >> > >> >>>>>>
> >> > >> >>>>>> Thanks,
> >> > >> >>>>>> Chinmay.
> >> > >> >>>>>>
> >> > >> >>>>>>
> >> > >> >>>>>> On Mon, Feb 15, 2016 at 11:11 AM, Roman Shaposhnik <
> >> > >> >>>> roman@shaposhnik.org>
> >> > >> >>>>>> wrote:
> >> > >> >>>>>>
> >> > >> >>>>>>> Looks good to me. Three questions:
> >> > >> >>>>>>>   1. Does apex provide standalone daemon or does only
> >> > >> >>>>>>> run on
> >> YARN
> >> > >> ?
> >> > >> >>>>>>>   2. What's the intended use for content under
> >> /usr/lib/apex/conf
> >> > >> ?
> >> > >> >>>>>>>   3. Any chance you can provide a pointer to GH repo
> >> > >> >>>>>>> with a
> >> > >> >>>> prototype?
> >> > >> >>>>>>>
> >> > >> >>>>>>> Thanks,
> >> > >> >>>>>>> Roman.
> >> > >> >>>>>>>
> >> > >> >>>>>>>
> >> > >> >>>>>>> On Sun, Feb 14, 2016 at 8:36 PM, Chinmay Kolhatkar <
> >> > >> >>>> chinmay@apache.org>
> >> > >> >>>>>>> wrote:
> >> > >> >>>>>>>> Hi Everyone,
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> Here is some details regarding packaging apex component
> >> > >> >>>>>>>> into
> >> > >> >>>> bigtop:
> >> > >> >>>>>>>> 1. Apex component will dependent on hadoop. This will
> >> > >> >>>>>>>> be
> >> > >> >>>> mentioned in
> >> > >> >>>>>>>> bigtop.bom file.
> >> > >> >>>>>>>> 2. The component added to bom file will look like
> following:
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> 'apex' {
> >> > >> >>>>>>>>      name    = 'apex'
> >> > >> >>>>>>>>      relNotes = 'Apache Apex (incubating)'
> >> > >> >>>>>>>>      version { base = '3.3.0-incubating'; pkg = base;
> >> release =
> >> > >> >>>> 1 }
> >> > >> >>>>>>>>      tarball { source      =
> >> > >> >>>>>>> "$name-${version.base}-source-release.tar.gz"
> >> > >> >>>>>>>>                destination =
> "$name-${version.base}.tar.gz"
> >> }
> >> > >> >>>>>>>>      url     { download_path =
> >> > >> >>>> "/incubator/$name/v${version.base}/"
> >> > >> >>>>>>>>                site =
> >> "${apache.APACHE_MIRROR}/${download_path}"
> >> > >> >>>>>>>>                archive =
> >> > >> >>>> "${apache.APACHE_ARCHIVE}/${download_path}" }
> >> > >> >>>>>>>> }
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> 3. After installation of package (deb+rpm), the
> >> > >> >>>>>>>> directory
> >> > >> >>>> structure
> >> > >> >>>>>>> related
> >> > >> >>>>>>>> to apex will look like following:
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> /
> >> > >> >>>>>>>> `-- usr
> >> > >> >>>>>>>>    |-- bin
> >> > >> >>>>>>>>    |   |-- apex -> ../lib/apex/bin/apex
> >> > >> >>>>>>>>    |   `-- dtcli -> ../lib/apex/bin/apex
> >> > >> >>>>>>>>    |-- lib
> >> > >> >>>>>>>>    |   `-- apex
> >> > >> >>>>>>>>    |       |-- bin
> >> > >> >>>>>>>>    |       |   |-- apex
> >> > >> >>>>>>>>    |       |   `-- dtcli -> apex
> >> > >> >>>>>>>>    |       |-- conf
> >> > >> >>>>>>>>    |       |   `-- dt-env.sh
> >> > >> >>>>>>>>    |       `-- lib
> >> > >> >>>>>>>>    |           |-- apex-api-3.3.0-incubating.jar
> >> > >> >>>>>>>>    |           |-- apex-bufferserver-3.3.0-incubating.jar
> >> > >> >>>>>>>>    |           |-- apex-common-3.3.0-incubating.jar
> >> > >> >>>>>>>>    |           |-- apex-engine.jar
> >> > >> >>>>>>>>    |           `-- <other dependency jars>
> >> > >> >>>>>>>>    `-- share
> >> > >> >>>>>>>>        `-- doc
> >> > >> >>>>>>>>            `-- apex
> >> > >> >>>>>>>>                |-- changelog.Debian.gz
> >> > >> >>>>>>>>                |-- copyright
> >> > >> >>>>>>>>                `-- license
> >> > >> >>>>>>>>                    `-- LICENSE.gz
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> Please share your thoughts on above approach.
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> Thanks,
> >> > >> >>>>>>>> Chinmay.
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>
> >> > >> >>>>>>>> On Sat, Feb 13, 2016 at 8:38 AM, Thomas Weise <
> >> thw@apache.org>
> >> > >> >>>> wrote:
> >> > >> >>>>>>>>
> >> > >> >>>>>>>>> Chinmay,
> >> > >> >>>>>>>>>
> >> > >> >>>>>>>>> Can you share some details regarding the packaging?
> >> > >> >>>>>>>>>
> >> > >> >>>>>>>>> Thanks
> >> > >> >>>>>>>>>
> >> > >> >>>>>>>>> On Thu, Feb 11, 2016 at 12:35 AM, Chinmay Kolhatkar <
> >> > >> >>>>>>>>> chinmay@datatorrent.com
> >> > >> >>>>>>>>>> wrote:
> >> > >> >>>>>>>>>
> >> > >> >>>>>>>>>> Dear Community,
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>> Here is the Jira that is created for adding Apex into
> >> Bigtop.
> >> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/BIGTOP-2313
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>> I've started working on creating packages for Apex.
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>> Please share your thoughts on approach as well as on
> >> questions
> >> > >> >>>>>>> related to
> >> > >> >>>>>>>>>> packaging in my previous mail.
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>> Thanks,
> >> > >> >>>>>>>>>> Chinmay.
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>> On Wed, Feb 10, 2016 at 9:55 PM, Chinmay Kolhatkar <
> >> > >> >>>>>>>>>> chinmay@datatorrent.com>
> >> > >> >>>>>>>>>> wrote:
> >> > >> >>>>>>>>>>
> >> > >> >>>>>>>>>>> Hello Bigtop Community,
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> Continuing with the packing effort, here is how
> >> > >> >>>>>>>>>>> we're
> >> > >> >>>> planning to
> >> > >> >>>>>>>>> package
> >> > >> >>>>>>>>>>> Apache Apex.
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> Tentative package directory structure for apex would
> >> > >> >>>>>>>>>>> look
> >> > >> >>>> like
> >> > >> >>>>>>>>> following:
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli
> >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-api-3.3.0-incubating.jar
> >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-engine-3.3.0-incubating.jar
> >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-bufferserver-3.3.0-incubating
> >> > >> >>>>>>>>>>> .jar
> >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-common-3.3.0-incubating.jar
> >> > >> >>>>>>>>>>> /usr/lib/apex/lib/<other dependent jars>
> >> > >> >>>>>>>>>>> /usr/lib/apex/conf/dt-env.sh /usr/bin/dtcli ->
> >> > >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli
> >> > >> >>>>>>>>>>> /usr/share/doc/man/man1/dtcli.1.gz
> >> > >> >>>>>>>>>>> /usr/share/doc/apex/license/LICENSE.txt
> >> > >> >>>>>>>>>>> /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > >> >>>>>>>>>>> /usr/share/doc/apex/CHANGELOG
> >> > >> >>>>>>>>>>> /usr/share/doc/apex/NOTICE
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> Addition contents to bigtop.bom:
> >> > >> >>>>>>>>>>> 1. Dependency of apex will be on "hadoop"
> >> > >> >>>>>>>>>>> 2. Component addition would looks like this:
> >> > >> >>>>>>>>>>> 'apex' {
> >> > >> >>>>>>>>>>>      name    = 'apex'
> >> > >> >>>>>>>>>>>      relNotes = 'Apache Apex (incubating)'
> >> > >> >>>>>>>>>>>      version { base = '3.3.0-incubating'; pkg =
> >> > >> >>>>>>>>>>> base;
> >> > >> >>>> release = 1
> >> > >> >>>>>>> }
> >> > >> >>>>>>>>>>>      tarball { source      =
> >> > >> >>>>>>>>>> "$name-${version.base}-source-release.tar.gz"
> >> > >> >>>>>>>>>>>                destination =
> >> "$name-${version.base}.tar.gz"
> >> > >> >>>> }
> >> > >> >>>>>>>>>>>      url     { download_path =
> >> > >> >>>>>>> "/incubator/$name/v${version.base}/"
> >> > >> >>>>>>>>>>>                site =
> >> > >> >>>> "${apache.APACHE_MIRROR}/${download_path}"
> >> > >> >>>>>>>>>>>                archive =
> >> > >> >>>>>>> "${apache.APACHE_ARCHIVE}/${download_path}" }
> >> > >> >>>>>>>>>>> }
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> I have couple of questions related to packaging:
> >> > >> >>>>>>>>>>> 1. What is the best possible way to test the package
> >> locally?
> >> > >> >>>>>>>>>>> 2. Do you have any specific size limitation on size
> >> > >> >>>>>>>>>>> of
> >> the
> >> > >> >>>> package
> >> > >> >>>>>>> (rpm
> >> > >> >>>>>>>>>> OR
> >> > >> >>>>>>>>>>> deb)?
> >> > >> >>>>>>>>>>> 3. Is above proposed packaging information inline
> >> > >> >>>>>>>>>>> with
> >> bigtop
> >> > >> >>>>>>>>> standards?
> >> > >> >>>>>>>>>>> 4. Am I missing anything there?
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> Thanks,
> >> > >> >>>>>>>>>>> Chinmay.
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>> On Fri, Feb 5, 2016 at 12:01 AM, Chinmay Kolhatkar <
> >> > >> >>>>>>>>>>> chinmay@datatorrent.com> wrote:
> >> > >> >>>>>>>>>>>
> >> > >> >>>>>>>>>>>> Hello Bigtop community!
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> My name is Chinmay Kolhatkar and I am a committer
> >> > >> >>>>>>>>>>>> in
> >> Apache
> >> > >> >>>> Apex
> >> > >> >>>>>>>>>>>> (incubating) project.
> >> > >> >>>>>>>>>>>> Our website is at:
> >> > >> >>>>>>>>>>>> http://apex.incubator.apache.org/
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> We want to explore and integrate Apache Apex into
> >> > >> >>>>>>>>>>>> the
> >> > >> >>>> Bigtop as a
> >> > >> >>>>>>>>>>>> component.
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> Please treat this mail as a continuation of a mail
> >> thread:
> >> > >> >>>>
> >> > >>
> >> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201601.mbox/%3CCA
> >> %2B5xAo20CAHuYtdxw3mtH2X9whREKd7fLdz4xmuTMqkaTiiskg%40mail.gmail.com%
> >> 3E
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> We're currently confirming with requirements for
> >> components
> >> > >> >>>> and
> >> > >> >>>>>>> making
> >> > >> >>>>>>>>>>>> sure that we meet the expectations:
> >> > >> >>>>
> >> > >>
> >> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+ad
> >> ding+a+new+component+to+Bigtop+distribution
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> Once requirements are through, we're planning to
> >> > >> >>>>>>>>>>>> add
> >> > >> >>>> component
> >> > >> >>>>>>> into
> >> > >> >>>>>>>>>>>> Apache Bigtop in following phases (as separate
> >> patches/pull
> >> > >> >>>>>>> requests):
> >> > >> >>>>>>>>>>>> 1. Packaging
> >> > >> >>>>>>>>>>>>    - Creating of package i.e. rpm & deb files.
> >> > >> >>>>>>>>>>>>    - documentations/READMEs, LICENSE, DISCLAMER,
> >> > >> >>>>>>>>>>>> NOTES
> >> etc
> >> > >> >>>> if any
> >> > >> >>>>>>>>>> needed.
> >> > >> >>>>>>>>>>>>    - Any documentation that need to be added to
> >> > >> >>>> distribution of
> >> > >> >>>>>>> our
> >> > >> >>>>>>>>>>>> software.
> >> > >> >>>>>>>>>>>>    - Any license information of dependencies
> >> > >> >>>>>>>>>>>> required
> >> to be
> >> > >> >>>>>>> added to
> >> > >> >>>>>>>>>>>> package
> >> > >> >>>>>>>>>>>> 2. Smoke tests
> >> > >> >>>>>>>>>>>>    - Adding smoke test for packaging.
> >> > >> >>>>>>>>>>>> 3. Puppet recipes for automatic deployment and
> >> configuration
> >> > >> >>>>>>>>>>>>    - Add puppet recipes for automatic deployment 4.
> >> > >> >>>>>>>>>>>> license clearance:
> >> > >> >>>>>>>>>>>>    - Run 'gradle rat' to make sure all new stuff is
> >> > >> >>>> compliant
> >> > >> >>>>>>> with
> >> > >> >>>>>>>>> ASF
> >> > >> >>>>>>>>>>>> license requirements.
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> Please share your thoughts on above approach.
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> If the approach seems fine, we can create a Jira
> >> > >> >>>>>>>>>>>> items
> >> and
> >> > >> >>>>>>>>> corresponding
> >> > >> >>>>>>>>>>>> subtasks to start working.
> >> > >> >>>>>>>>>>>>
> >> > >> >>>>>>>>>>>> Thanks,
> >> > >> >>>>>>>>>>>> Chinmay.
> >> > >> >>
> >> > >>
> >> > >
> >> > >
> >>
> >
> >
>
>

RE: Apache Apex as component in Apache Bigtop

[na...@reactor8.com]

Hi Chinmay,

Great to hear last apex items are ready, will be great to have as a component.

As far as a next release, there is this jira currently:

https://issues.apache.org/jira/browse/BIGTOP-2282

I know besides some fixes and updates, I saw some msgs about flink related work to join apex as latest supported components.  Also some talked to some folks from openpower community this week at their event and they had wanted to fix couple component issues from 1.1.

Would be good if schedules and efforts and coalesce in time for Apache Bigdata timeframe in May, or shortly thereafter.  We have couple bigtop presentations at the event so we add new components and updates, including apex regardless if release hits before/during or after the event


-----Original Message-----
From: Chinmay Kolhatkar [mailto:chinmay@apache.org] 
Sent: Friday, April 8, 2016 10:43 AM
To: dev@bigtop.apache.org
Subject: Re: Apache Apex as component in Apache Bigtop

Hello Bigtop Community,

The last PR for adding apex as component is bigtop is merged. This completes the Jira BIGTOP-2313.

A huge thanks to everyone for accepting the patches and adding apex as a component in bigtop github repository.

We're excited to share "yum/apt-get install apex" commands with users of apex.
Any idea on when will be the next release of bigtop and when yum/apt-get repositories will get updated?

Thanks,
Chinmay.



On Fri, Mar 4, 2016 at 12:24 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> ​Hello Bigtop Community,
>
> As a part of second phase of integration, I've created a pull request 
> for adding smoke test of Apex package to Bigtop that I would like to 
> get reviewed and committed to Bigtop repository:
> https://github.com/apache/bigtop/pull/94
>
> The pull request contains 2 basic tests which validates correctness of 
> the installed apex package.
>
> If you have some time, could you please review the pull request and 
> provide the feedback so that we can get these tests merged in?
>
> Thanks a ton!!
>
> - Chinmay.
>
>
>
> On Fri, Feb 26, 2016 at 9:22 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
>
>> On Fri, Feb 26, 2016 at 02:26PM, Chinmay Kolhatkar wrote:
>> > Thank you Bigtop community for efficiently consuming the first 
>> > patch (packing of apex).
>>
>> yeah, you don't want to see the regurgitation process ;)
>>
>> > For the second phase, I'll soon be creating a PR for Smoke tests 
>> > for
>> apex
>> > package.
>>
>> Keep 'em coming and thanks for the contributions!
>>   Cos
>>
>> > On Thu, Feb 25, 2016 at 12:30 PM, Chinmay Kolhatkar 
>> > <chinmay@apache.org
>> >
>> > wrote:
>> >
>> > > Thanks Jay.
>> > > There are already some comments from YoungWoo Kim which are taken
>> care of
>> > > in that PR.
>> > >
>> > > On Tue, Feb 23, 2016 at 6:05 PM, Jay Vyas <
>> jayunit100.apache@gmail.com>
>> > > wrote:
>> > >
>> > >> Thanks again for this , don't apologize for the email.
>> > >>
>> > >> We actually are excited about apex, thanks for this! We will get 
>> > >> it reviewed within the next couple of days I'm sure (I'll look 
>> > >> some
>> time after
>> > >> work if nobody else gets to it first).
>> > >>
>> > >> > On Feb 23, 2016, at 6:09 AM, Chinmay Kolhatkar 
>> > >> > <chinmay@apache.org
>> >
>> > >> wrote:
>> > >> >
>> > >> > Sorry for another mail. Just want to mention how I tested the 
>> > >> > patch
>> > >> before
>> > >> > creating PR:
>> > >> >
>> > >> > Apex package was built using with following docker images:
>> > >> > 1) Ubuntu trusty:
>> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
>> > >> bigtop/slaves:trunk-ubuntu-14.04
>> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
>> > >> >
>> > >> > 2) CentOS 7
>> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
>> bigtop/slaves:trunk-centos-7
>> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
>> > >> >
>> > >> > 3) CentOS 6
>> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
>> bigtop/slaves:trunk-centos-6
>> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
>> > >> >
>> > >> > 4) OpenSUSE 13.2
>> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
>> > >> > bigtop/slaves:trunk-opensuse-13.2 -l -c './gradlew allclean ;
>> ./gradlew
>> > >> > apex-pkg'
>> > >> >
>> > >> >
>> > >> > Apex package was installed on following docker containers and
>> verified
>> > >> for
>> > >> > sanity:
>> > >> > 1) bigtop/slaves:trunk-ubuntu-14.04
>> > >> > 2) bigtop/slaves:trunk-centos-6
>> > >> > 3) bigtop/slaves:trunk-centos-7
>> > >> > 4) bigtop/slaves:trunk-opensuse-13.2
>> > >> >
>> > >> > When you have some time, please provide the feedback on the 
>> > >> > pull
>> > >> request (
>> > >> > https://github.com/apache/bigtop/pull/92)
>> > >> >
>> > >> > Thank you,
>> > >> > Chinmay.
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Mon, Feb 22, 2016 at 9:43 PM, Chinmay Kolhatkar <
>> chinmay@apache.org>
>> > >> > wrote:
>> > >> >
>> > >> >> Hello Bigtop Community,
>> > >> >>
>> > >> >> I've created a pull request for packing of Apache Apex into
>> bigtop that
>> > >> >> I'd like to get reviewd and committed to Bigtop repository:
>> > >> >> https://github.com/apache/bigtop/pull/92
>> > >> >>
>> > >> >> The pull request contains code for packing of apex for both 
>> > >> >> deb
>> and rpm
>> > >> >> packing.
>> > >> >> This does not include smoke tests which will be the very next
>> steps
>> > >> once
>> > >> >> this gets accepted.
>> > >> >>
>> > >> >> If you have some time, could you please review this pull 
>> > >> >> request
>> and
>> > >> >> provide feedback so that we can get these merged in?
>> > >> >>
>> > >> >> Also, there is a question related to what should be included 
>> > >> >> in
>> package
>> > >> >> with respect to packing of runtime dependent jars.
>> > >> >> Currently, I've included license files of non-ASF based 
>> > >> >> licenses
>> in
>> > >> >> package.
>> > >> >> Is there any specific guidelines from bigtop that integrating
>> projects
>> > >> >> needs to follow on that front?
>> > >> >>
>> > >> >> In the meantime, I would like to get some feedback on pull
>> request for
>> > >> if
>> > >> >> there are any code change required.
>> > >> >>
>> > >> >> Thank you in advance!!
>> > >> >>
>> > >> >> Chinmay.
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> On Wed, Feb 17, 2016 at 11:10 AM, Chinmay Kolhatkar <
>> > >> chinmay@apache.org>
>> > >> >> wrote:
>> > >> >>
>> > >> >>> Cos,
>> > >> >>> Thank for a quick reply. We (apex community) has decided to
>> follow
>> > >> your
>> > >> >>> recommendation i.e. drop "-incubating".
>> > >> >>>
>> > >> >>> Roman,
>> > >> >>> Apologies, I did not understand your last question earlier.
>> > >> >>> Yes, I can certainly provide the prototype for rpm+deb 
>> > >> >>> packing
>> of Apex
>> > >> >>> with bigtop.
>> > >> >>> Here is the link to GH where development is WIP:
>> > >> >>> https://github.com/chinmaykolhatkar/bigtop/tree/BIGTOP-2314
>> > >> >>>
>> > >> >>> Thanks,
>> > >> >>> Chinmay.
>> > >> >>>
>> > >> >>>
>> > >> >>> On Tue, Feb 16, 2016 at 9:15 PM, Konstantin Boudnik <
>> cos@apache.org>
>> > >> >>> wrote:
>> > >> >>>
>> > >> >>>> Dude to known limitations of RPMs "-" aren't allowed for sure.
>> Deb
>> > >> >>>> doesn't
>> > >> >>>> have such idiosyncrasy, but there are ppl who are using 
>> > >> >>>> RPM,
>> which is
>> > >> >>>> beyond
>> > >> >>>> me. Anyway...
>> > >> >>>>
>> > >> >>>> This is a long standing issue for us as well, as we have a 
>> > >> >>>> few
>> > >> incubating
>> > >> >>>> projects in the stack. My recommendation would be to drop
>> > >> "-incubating"
>> > >> >>>> part
>> > >> >>>> from the package version. Check out how it is done for zeppelin.
>> > >> >>>>
>> > >> >>>> Cos
>> > >> >>>>
>> > >> >>>>> On Tue, Feb 16, 2016 at 09:04PM, Chinmay Kolhatkar wrote:
>> > >> >>>>> Hello Everyone,
>> > >> >>>>>
>> > >> >>>>> I'm facing an issue while setting base version for rpm package.
>> > >> >>>>> We want to have version of Apache Apex as "3.3.0-incubating".
>> > >> >>>>> But when I try to set version in bigtop.bom file, I get 
>> > >> >>>>> error
>> while
>> > >> >>>>> generating rpm package as rpm does not allow hyphen ('-') 
>> > >> >>>>> in
>> > >> version.
>> > >> >>>>>
>> > >> >>>>> I also tried to set 'incubating' as a value as release:
>> version {
>> > >> base
>> > >> >>>> =
>> > >> >>>>> '3.3.0'; pkg = base; release = 'incubating' } But that 
>> > >> >>>>> doesn't seem to take any effect.
>> > >> >>>>>
>> > >> >>>>> Can you please suggest how can we achieve this?
>> > >> >>>>>
>> > >> >>>>> Thanks,
>> > >> >>>>> Chinmay.
>> > >> >>>>>
>> > >> >>>>>
>> > >> >>>>> On Mon, Feb 15, 2016 at 11:38 AM, Chinmay Kolhatkar <
>> > >> >>>> chinmay@apache.org>
>> > >> >>>>> wrote:
>> > >> >>>>>
>> > >> >>>>>> Hi Roman,
>> > >> >>>>>>
>> > >> >>>>>> Thanks for the feedback.
>> > >> >>>>>> Here are the answers to the question in the same order:
>> > >> >>>>>> 1. Apex is a YARN-native platform. Hence apex needs YARN 
>> > >> >>>>>> to be
>> > >> there.
>> > >> >>>>>> 2. conf/ directory recently got removed. This is not 
>> > >> >>>>>> needed
>> anymore
>> > >> >>>> in the
>> > >> >>>>>> package.
>> > >> >>>>>> 3. Here is the link to GH repo for apex platform:
>> > >> >>>>>>     https://github.com/apache/incubator-apex-core
>> > >> >>>>>>   The source as of now does not output any package.
>> do-component
>> > >> >>>> build
>> > >> >>>>>> would compile the apex platform and generate a rpm/deb 
>> > >> >>>>>> package
>> > >> using
>> > >> >>>> files
>> > >> >>>>>> present in source code. Hence there is no prototype of 
>> > >> >>>>>> the
>> package.
>> > >> >>>>>>   Is that the prototype you're looking for?
>> > >> >>>>>>
>> > >> >>>>>> Thanks,
>> > >> >>>>>> Chinmay.
>> > >> >>>>>>
>> > >> >>>>>>
>> > >> >>>>>> On Mon, Feb 15, 2016 at 11:11 AM, Roman Shaposhnik <
>> > >> >>>> roman@shaposhnik.org>
>> > >> >>>>>> wrote:
>> > >> >>>>>>
>> > >> >>>>>>> Looks good to me. Three questions:
>> > >> >>>>>>>   1. Does apex provide standalone daemon or does only 
>> > >> >>>>>>> run on
>> YARN
>> > >> ?
>> > >> >>>>>>>   2. What's the intended use for content under
>> /usr/lib/apex/conf
>> > >> ?
>> > >> >>>>>>>   3. Any chance you can provide a pointer to GH repo 
>> > >> >>>>>>> with a
>> > >> >>>> prototype?
>> > >> >>>>>>>
>> > >> >>>>>>> Thanks,
>> > >> >>>>>>> Roman.
>> > >> >>>>>>>
>> > >> >>>>>>>
>> > >> >>>>>>> On Sun, Feb 14, 2016 at 8:36 PM, Chinmay Kolhatkar <
>> > >> >>>> chinmay@apache.org>
>> > >> >>>>>>> wrote:
>> > >> >>>>>>>> Hi Everyone,
>> > >> >>>>>>>>
>> > >> >>>>>>>> Here is some details regarding packaging apex component 
>> > >> >>>>>>>> into
>> > >> >>>> bigtop:
>> > >> >>>>>>>> 1. Apex component will dependent on hadoop. This will 
>> > >> >>>>>>>> be
>> > >> >>>> mentioned in
>> > >> >>>>>>>> bigtop.bom file.
>> > >> >>>>>>>> 2. The component added to bom file will look like following:
>> > >> >>>>>>>>
>> > >> >>>>>>>> 'apex' {
>> > >> >>>>>>>>      name    = 'apex'
>> > >> >>>>>>>>      relNotes = 'Apache Apex (incubating)'
>> > >> >>>>>>>>      version { base = '3.3.0-incubating'; pkg = base;
>> release =
>> > >> >>>> 1 }
>> > >> >>>>>>>>      tarball { source      =
>> > >> >>>>>>> "$name-${version.base}-source-release.tar.gz"
>> > >> >>>>>>>>                destination = "$name-${version.base}.tar.gz"
>> }
>> > >> >>>>>>>>      url     { download_path =
>> > >> >>>> "/incubator/$name/v${version.base}/"
>> > >> >>>>>>>>                site =
>> "${apache.APACHE_MIRROR}/${download_path}"
>> > >> >>>>>>>>                archive =
>> > >> >>>> "${apache.APACHE_ARCHIVE}/${download_path}" }
>> > >> >>>>>>>> }
>> > >> >>>>>>>>
>> > >> >>>>>>>> 3. After installation of package (deb+rpm), the 
>> > >> >>>>>>>> directory
>> > >> >>>> structure
>> > >> >>>>>>> related
>> > >> >>>>>>>> to apex will look like following:
>> > >> >>>>>>>>
>> > >> >>>>>>>> /
>> > >> >>>>>>>> `-- usr
>> > >> >>>>>>>>    |-- bin
>> > >> >>>>>>>>    |   |-- apex -> ../lib/apex/bin/apex
>> > >> >>>>>>>>    |   `-- dtcli -> ../lib/apex/bin/apex
>> > >> >>>>>>>>    |-- lib
>> > >> >>>>>>>>    |   `-- apex
>> > >> >>>>>>>>    |       |-- bin
>> > >> >>>>>>>>    |       |   |-- apex
>> > >> >>>>>>>>    |       |   `-- dtcli -> apex
>> > >> >>>>>>>>    |       |-- conf
>> > >> >>>>>>>>    |       |   `-- dt-env.sh
>> > >> >>>>>>>>    |       `-- lib
>> > >> >>>>>>>>    |           |-- apex-api-3.3.0-incubating.jar
>> > >> >>>>>>>>    |           |-- apex-bufferserver-3.3.0-incubating.jar
>> > >> >>>>>>>>    |           |-- apex-common-3.3.0-incubating.jar
>> > >> >>>>>>>>    |           |-- apex-engine.jar
>> > >> >>>>>>>>    |           `-- <other dependency jars>
>> > >> >>>>>>>>    `-- share
>> > >> >>>>>>>>        `-- doc
>> > >> >>>>>>>>            `-- apex
>> > >> >>>>>>>>                |-- changelog.Debian.gz
>> > >> >>>>>>>>                |-- copyright
>> > >> >>>>>>>>                `-- license
>> > >> >>>>>>>>                    `-- LICENSE.gz
>> > >> >>>>>>>>
>> > >> >>>>>>>>
>> > >> >>>>>>>> Please share your thoughts on above approach.
>> > >> >>>>>>>>
>> > >> >>>>>>>>
>> > >> >>>>>>>> Thanks,
>> > >> >>>>>>>> Chinmay.
>> > >> >>>>>>>>
>> > >> >>>>>>>>
>> > >> >>>>>>>>
>> > >> >>>>>>>>
>> > >> >>>>>>>> On Sat, Feb 13, 2016 at 8:38 AM, Thomas Weise <
>> thw@apache.org>
>> > >> >>>> wrote:
>> > >> >>>>>>>>
>> > >> >>>>>>>>> Chinmay,
>> > >> >>>>>>>>>
>> > >> >>>>>>>>> Can you share some details regarding the packaging?
>> > >> >>>>>>>>>
>> > >> >>>>>>>>> Thanks
>> > >> >>>>>>>>>
>> > >> >>>>>>>>> On Thu, Feb 11, 2016 at 12:35 AM, Chinmay Kolhatkar < 
>> > >> >>>>>>>>> chinmay@datatorrent.com
>> > >> >>>>>>>>>> wrote:
>> > >> >>>>>>>>>
>> > >> >>>>>>>>>> Dear Community,
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>> Here is the Jira that is created for adding Apex into
>> Bigtop.
>> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/BIGTOP-2313
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>> I've started working on creating packages for Apex.
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>> Please share your thoughts on approach as well as on
>> questions
>> > >> >>>>>>> related to
>> > >> >>>>>>>>>> packaging in my previous mail.
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>> Thanks,
>> > >> >>>>>>>>>> Chinmay.
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>> On Wed, Feb 10, 2016 at 9:55 PM, Chinmay Kolhatkar < 
>> > >> >>>>>>>>>> chinmay@datatorrent.com>
>> > >> >>>>>>>>>> wrote:
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>>> Hello Bigtop Community,
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> Continuing with the packing effort, here is how 
>> > >> >>>>>>>>>>> we're
>> > >> >>>> planning to
>> > >> >>>>>>>>> package
>> > >> >>>>>>>>>>> Apache Apex.
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> Tentative package directory structure for apex would 
>> > >> >>>>>>>>>>> look
>> > >> >>>> like
>> > >> >>>>>>>>> following:
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli 
>> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-api-3.3.0-incubating.jar
>> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-engine-3.3.0-incubating.jar
>> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-bufferserver-3.3.0-incubating
>> > >> >>>>>>>>>>> .jar 
>> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-common-3.3.0-incubating.jar
>> > >> >>>>>>>>>>> /usr/lib/apex/lib/<other dependent jars> 
>> > >> >>>>>>>>>>> /usr/lib/apex/conf/dt-env.sh /usr/bin/dtcli -> 
>> > >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli 
>> > >> >>>>>>>>>>> /usr/share/doc/man/man1/dtcli.1.gz
>> > >> >>>>>>>>>>> /usr/share/doc/apex/license/LICENSE.txt
>> > >> >>>>>>>>>>> /usr/share/doc/apex/license/<package>-LICENSE.txt
>> > >> >>>>>>>>>>> /usr/share/doc/apex/CHANGELOG 
>> > >> >>>>>>>>>>> /usr/share/doc/apex/NOTICE
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> Addition contents to bigtop.bom:
>> > >> >>>>>>>>>>> 1. Dependency of apex will be on "hadoop"
>> > >> >>>>>>>>>>> 2. Component addition would looks like this:
>> > >> >>>>>>>>>>> 'apex' {
>> > >> >>>>>>>>>>>      name    = 'apex'
>> > >> >>>>>>>>>>>      relNotes = 'Apache Apex (incubating)'
>> > >> >>>>>>>>>>>      version { base = '3.3.0-incubating'; pkg = 
>> > >> >>>>>>>>>>> base;
>> > >> >>>> release = 1
>> > >> >>>>>>> }
>> > >> >>>>>>>>>>>      tarball { source      =
>> > >> >>>>>>>>>> "$name-${version.base}-source-release.tar.gz"
>> > >> >>>>>>>>>>>                destination =
>> "$name-${version.base}.tar.gz"
>> > >> >>>> }
>> > >> >>>>>>>>>>>      url     { download_path =
>> > >> >>>>>>> "/incubator/$name/v${version.base}/"
>> > >> >>>>>>>>>>>                site =
>> > >> >>>> "${apache.APACHE_MIRROR}/${download_path}"
>> > >> >>>>>>>>>>>                archive =
>> > >> >>>>>>> "${apache.APACHE_ARCHIVE}/${download_path}" }
>> > >> >>>>>>>>>>> }
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> I have couple of questions related to packaging:
>> > >> >>>>>>>>>>> 1. What is the best possible way to test the package
>> locally?
>> > >> >>>>>>>>>>> 2. Do you have any specific size limitation on size 
>> > >> >>>>>>>>>>> of
>> the
>> > >> >>>> package
>> > >> >>>>>>> (rpm
>> > >> >>>>>>>>>> OR
>> > >> >>>>>>>>>>> deb)?
>> > >> >>>>>>>>>>> 3. Is above proposed packaging information inline 
>> > >> >>>>>>>>>>> with
>> bigtop
>> > >> >>>>>>>>> standards?
>> > >> >>>>>>>>>>> 4. Am I missing anything there?
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> Thanks,
>> > >> >>>>>>>>>>> Chinmay.
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> On Fri, Feb 5, 2016 at 12:01 AM, Chinmay Kolhatkar < 
>> > >> >>>>>>>>>>> chinmay@datatorrent.com> wrote:
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>>> Hello Bigtop community!
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> My name is Chinmay Kolhatkar and I am a committer 
>> > >> >>>>>>>>>>>> in
>> Apache
>> > >> >>>> Apex
>> > >> >>>>>>>>>>>> (incubating) project.
>> > >> >>>>>>>>>>>> Our website is at: 
>> > >> >>>>>>>>>>>> http://apex.incubator.apache.org/
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> We want to explore and integrate Apache Apex into 
>> > >> >>>>>>>>>>>> the
>> > >> >>>> Bigtop as a
>> > >> >>>>>>>>>>>> component.
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> Please treat this mail as a continuation of a mail
>> thread:
>> > >> >>>>
>> > >>
>> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201601.mbox/%3CCA
>> %2B5xAo20CAHuYtdxw3mtH2X9whREKd7fLdz4xmuTMqkaTiiskg%40mail.gmail.com%
>> 3E
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> We're currently confirming with requirements for
>> components
>> > >> >>>> and
>> > >> >>>>>>> making
>> > >> >>>>>>>>>>>> sure that we meet the expectations:
>> > >> >>>>
>> > >>
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+ad
>> ding+a+new+component+to+Bigtop+distribution
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> Once requirements are through, we're planning to 
>> > >> >>>>>>>>>>>> add
>> > >> >>>> component
>> > >> >>>>>>> into
>> > >> >>>>>>>>>>>> Apache Bigtop in following phases (as separate
>> patches/pull
>> > >> >>>>>>> requests):
>> > >> >>>>>>>>>>>> 1. Packaging
>> > >> >>>>>>>>>>>>    - Creating of package i.e. rpm & deb files.
>> > >> >>>>>>>>>>>>    - documentations/READMEs, LICENSE, DISCLAMER, 
>> > >> >>>>>>>>>>>> NOTES
>> etc
>> > >> >>>> if any
>> > >> >>>>>>>>>> needed.
>> > >> >>>>>>>>>>>>    - Any documentation that need to be added to
>> > >> >>>> distribution of
>> > >> >>>>>>> our
>> > >> >>>>>>>>>>>> software.
>> > >> >>>>>>>>>>>>    - Any license information of dependencies 
>> > >> >>>>>>>>>>>> required
>> to be
>> > >> >>>>>>> added to
>> > >> >>>>>>>>>>>> package
>> > >> >>>>>>>>>>>> 2. Smoke tests
>> > >> >>>>>>>>>>>>    - Adding smoke test for packaging.
>> > >> >>>>>>>>>>>> 3. Puppet recipes for automatic deployment and
>> configuration
>> > >> >>>>>>>>>>>>    - Add puppet recipes for automatic deployment 4. 
>> > >> >>>>>>>>>>>> license clearance:
>> > >> >>>>>>>>>>>>    - Run 'gradle rat' to make sure all new stuff is
>> > >> >>>> compliant
>> > >> >>>>>>> with
>> > >> >>>>>>>>> ASF
>> > >> >>>>>>>>>>>> license requirements.
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> Please share your thoughts on above approach.
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> If the approach seems fine, we can create a Jira 
>> > >> >>>>>>>>>>>> items
>> and
>> > >> >>>>>>>>> corresponding
>> > >> >>>>>>>>>>>> subtasks to start working.
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> Thanks,
>> > >> >>>>>>>>>>>> Chinmay.
>> > >> >>
>> > >>
>> > >
>> > >
>>
>
>

Re: Apache Apex as component in Apache Bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Hello Bigtop Community,

The last PR for adding apex as component is bigtop is merged. This
completes the Jira BIGTOP-2313.

A huge thanks to everyone for accepting the patches and adding apex as a
component in bigtop github repository.

We're excited to share "yum/apt-get install apex" commands with users of
apex.
Any idea on when will be the next release of bigtop and when yum/apt-get
repositories will get updated?

Thanks,
Chinmay.



On Fri, Mar 4, 2016 at 12:24 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> ​Hello Bigtop Community,
>
> As a part of second phase of integration, I've created a pull request for
> adding smoke test of Apex package to Bigtop that I would like to get
> reviewed and committed to Bigtop repository:
> https://github.com/apache/bigtop/pull/94
>
> The pull request contains 2 basic tests which validates correctness of the
> installed apex package.
>
> If you have some time, could you please review the pull request and
> provide the feedback so that we can get these tests merged in?
>
> Thanks a ton!!
>
> - Chinmay.
>
>
>
> On Fri, Feb 26, 2016 at 9:22 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
>
>> On Fri, Feb 26, 2016 at 02:26PM, Chinmay Kolhatkar wrote:
>> > Thank you Bigtop community for efficiently consuming the first patch
>> > (packing of apex).
>>
>> yeah, you don't want to see the regurgitation process ;)
>>
>> > For the second phase, I'll soon be creating a PR for Smoke tests for
>> apex
>> > package.
>>
>> Keep 'em coming and thanks for the contributions!
>>   Cos
>>
>> > On Thu, Feb 25, 2016 at 12:30 PM, Chinmay Kolhatkar <chinmay@apache.org
>> >
>> > wrote:
>> >
>> > > Thanks Jay.
>> > > There are already some comments from YoungWoo Kim which are taken
>> care of
>> > > in that PR.
>> > >
>> > > On Tue, Feb 23, 2016 at 6:05 PM, Jay Vyas <
>> jayunit100.apache@gmail.com>
>> > > wrote:
>> > >
>> > >> Thanks again for this , don't apologize for the email.
>> > >>
>> > >> We actually are excited about apex, thanks for this! We will get it
>> > >> reviewed within the next couple of days I'm sure (I'll look some
>> time after
>> > >> work if nobody else gets to it first).
>> > >>
>> > >> > On Feb 23, 2016, at 6:09 AM, Chinmay Kolhatkar <chinmay@apache.org
>> >
>> > >> wrote:
>> > >> >
>> > >> > Sorry for another mail. Just want to mention how I tested the patch
>> > >> before
>> > >> > creating PR:
>> > >> >
>> > >> > Apex package was built using with following docker images:
>> > >> > 1) Ubuntu trusty:
>> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
>> > >> bigtop/slaves:trunk-ubuntu-14.04
>> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
>> > >> >
>> > >> > 2) CentOS 7
>> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
>> bigtop/slaves:trunk-centos-7
>> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
>> > >> >
>> > >> > 3) CentOS 6
>> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
>> bigtop/slaves:trunk-centos-6
>> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg'
>> > >> >
>> > >> > 4) OpenSUSE 13.2
>> > >> > docker run --rm -v `pwd`:/ws --workdir /ws
>> > >> > bigtop/slaves:trunk-opensuse-13.2 -l -c './gradlew allclean ;
>> ./gradlew
>> > >> > apex-pkg'
>> > >> >
>> > >> >
>> > >> > Apex package was installed on following docker containers and
>> verified
>> > >> for
>> > >> > sanity:
>> > >> > 1) bigtop/slaves:trunk-ubuntu-14.04
>> > >> > 2) bigtop/slaves:trunk-centos-6
>> > >> > 3) bigtop/slaves:trunk-centos-7
>> > >> > 4) bigtop/slaves:trunk-opensuse-13.2
>> > >> >
>> > >> > When you have some time, please provide the feedback on the pull
>> > >> request (
>> > >> > https://github.com/apache/bigtop/pull/92)
>> > >> >
>> > >> > Thank you,
>> > >> > Chinmay.
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Mon, Feb 22, 2016 at 9:43 PM, Chinmay Kolhatkar <
>> chinmay@apache.org>
>> > >> > wrote:
>> > >> >
>> > >> >> Hello Bigtop Community,
>> > >> >>
>> > >> >> I've created a pull request for packing of Apache Apex into
>> bigtop that
>> > >> >> I'd like to get reviewd and committed to Bigtop repository:
>> > >> >> https://github.com/apache/bigtop/pull/92
>> > >> >>
>> > >> >> The pull request contains code for packing of apex for both deb
>> and rpm
>> > >> >> packing.
>> > >> >> This does not include smoke tests which will be the very next
>> steps
>> > >> once
>> > >> >> this gets accepted.
>> > >> >>
>> > >> >> If you have some time, could you please review this pull request
>> and
>> > >> >> provide feedback so that we can get these merged in?
>> > >> >>
>> > >> >> Also, there is a question related to what should be included in
>> package
>> > >> >> with respect to packing of runtime dependent jars.
>> > >> >> Currently, I've included license files of non-ASF based licenses
>> in
>> > >> >> package.
>> > >> >> Is there any specific guidelines from bigtop that integrating
>> projects
>> > >> >> needs to follow on that front?
>> > >> >>
>> > >> >> In the meantime, I would like to get some feedback on pull
>> request for
>> > >> if
>> > >> >> there are any code change required.
>> > >> >>
>> > >> >> Thank you in advance!!
>> > >> >>
>> > >> >> Chinmay.
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> On Wed, Feb 17, 2016 at 11:10 AM, Chinmay Kolhatkar <
>> > >> chinmay@apache.org>
>> > >> >> wrote:
>> > >> >>
>> > >> >>> Cos,
>> > >> >>> Thank for a quick reply. We (apex community) has decided to
>> follow
>> > >> your
>> > >> >>> recommendation i.e. drop "-incubating".
>> > >> >>>
>> > >> >>> Roman,
>> > >> >>> Apologies, I did not understand your last question earlier.
>> > >> >>> Yes, I can certainly provide the prototype for rpm+deb packing
>> of Apex
>> > >> >>> with bigtop.
>> > >> >>> Here is the link to GH where development is WIP:
>> > >> >>> https://github.com/chinmaykolhatkar/bigtop/tree/BIGTOP-2314
>> > >> >>>
>> > >> >>> Thanks,
>> > >> >>> Chinmay.
>> > >> >>>
>> > >> >>>
>> > >> >>> On Tue, Feb 16, 2016 at 9:15 PM, Konstantin Boudnik <
>> cos@apache.org>
>> > >> >>> wrote:
>> > >> >>>
>> > >> >>>> Dude to known limitations of RPMs "-" aren't allowed for sure.
>> Deb
>> > >> >>>> doesn't
>> > >> >>>> have such idiosyncrasy, but there are ppl who are using RPM,
>> which is
>> > >> >>>> beyond
>> > >> >>>> me. Anyway...
>> > >> >>>>
>> > >> >>>> This is a long standing issue for us as well, as we have a few
>> > >> incubating
>> > >> >>>> projects in the stack. My recommendation would be to drop
>> > >> "-incubating"
>> > >> >>>> part
>> > >> >>>> from the package version. Check out how it is done for zeppelin.
>> > >> >>>>
>> > >> >>>> Cos
>> > >> >>>>
>> > >> >>>>> On Tue, Feb 16, 2016 at 09:04PM, Chinmay Kolhatkar wrote:
>> > >> >>>>> Hello Everyone,
>> > >> >>>>>
>> > >> >>>>> I'm facing an issue while setting base version for rpm package.
>> > >> >>>>> We want to have version of Apache Apex as "3.3.0-incubating".
>> > >> >>>>> But when I try to set version in bigtop.bom file, I get error
>> while
>> > >> >>>>> generating rpm package as rpm does not allow hyphen ('-') in
>> > >> version.
>> > >> >>>>>
>> > >> >>>>> I also tried to set 'incubating' as a value as release:
>> version {
>> > >> base
>> > >> >>>> =
>> > >> >>>>> '3.3.0'; pkg = base; release = 'incubating' }
>> > >> >>>>> But that doesn't seem to take any effect.
>> > >> >>>>>
>> > >> >>>>> Can you please suggest how can we achieve this?
>> > >> >>>>>
>> > >> >>>>> Thanks,
>> > >> >>>>> Chinmay.
>> > >> >>>>>
>> > >> >>>>>
>> > >> >>>>> On Mon, Feb 15, 2016 at 11:38 AM, Chinmay Kolhatkar <
>> > >> >>>> chinmay@apache.org>
>> > >> >>>>> wrote:
>> > >> >>>>>
>> > >> >>>>>> Hi Roman,
>> > >> >>>>>>
>> > >> >>>>>> Thanks for the feedback.
>> > >> >>>>>> Here are the answers to the question in the same order:
>> > >> >>>>>> 1. Apex is a YARN-native platform. Hence apex needs YARN to be
>> > >> there.
>> > >> >>>>>> 2. conf/ directory recently got removed. This is not needed
>> anymore
>> > >> >>>> in the
>> > >> >>>>>> package.
>> > >> >>>>>> 3. Here is the link to GH repo for apex platform:
>> > >> >>>>>>     https://github.com/apache/incubator-apex-core
>> > >> >>>>>>   The source as of now does not output any package.
>> do-component
>> > >> >>>> build
>> > >> >>>>>> would compile the apex platform and generate a rpm/deb package
>> > >> using
>> > >> >>>> files
>> > >> >>>>>> present in source code. Hence there is no prototype of the
>> package.
>> > >> >>>>>>   Is that the prototype you're looking for?
>> > >> >>>>>>
>> > >> >>>>>> Thanks,
>> > >> >>>>>> Chinmay.
>> > >> >>>>>>
>> > >> >>>>>>
>> > >> >>>>>> On Mon, Feb 15, 2016 at 11:11 AM, Roman Shaposhnik <
>> > >> >>>> roman@shaposhnik.org>
>> > >> >>>>>> wrote:
>> > >> >>>>>>
>> > >> >>>>>>> Looks good to me. Three questions:
>> > >> >>>>>>>   1. Does apex provide standalone daemon or does only run on
>> YARN
>> > >> ?
>> > >> >>>>>>>   2. What's the intended use for content under
>> /usr/lib/apex/conf
>> > >> ?
>> > >> >>>>>>>   3. Any chance you can provide a pointer to GH repo with a
>> > >> >>>> prototype?
>> > >> >>>>>>>
>> > >> >>>>>>> Thanks,
>> > >> >>>>>>> Roman.
>> > >> >>>>>>>
>> > >> >>>>>>>
>> > >> >>>>>>> On Sun, Feb 14, 2016 at 8:36 PM, Chinmay Kolhatkar <
>> > >> >>>> chinmay@apache.org>
>> > >> >>>>>>> wrote:
>> > >> >>>>>>>> Hi Everyone,
>> > >> >>>>>>>>
>> > >> >>>>>>>> Here is some details regarding packaging apex component into
>> > >> >>>> bigtop:
>> > >> >>>>>>>> 1. Apex component will dependent on hadoop. This will be
>> > >> >>>> mentioned in
>> > >> >>>>>>>> bigtop.bom file.
>> > >> >>>>>>>> 2. The component added to bom file will look like following:
>> > >> >>>>>>>>
>> > >> >>>>>>>> 'apex' {
>> > >> >>>>>>>>      name    = 'apex'
>> > >> >>>>>>>>      relNotes = 'Apache Apex (incubating)'
>> > >> >>>>>>>>      version { base = '3.3.0-incubating'; pkg = base;
>> release =
>> > >> >>>> 1 }
>> > >> >>>>>>>>      tarball { source      =
>> > >> >>>>>>> "$name-${version.base}-source-release.tar.gz"
>> > >> >>>>>>>>                destination = "$name-${version.base}.tar.gz"
>> }
>> > >> >>>>>>>>      url     { download_path =
>> > >> >>>> "/incubator/$name/v${version.base}/"
>> > >> >>>>>>>>                site =
>> "${apache.APACHE_MIRROR}/${download_path}"
>> > >> >>>>>>>>                archive =
>> > >> >>>> "${apache.APACHE_ARCHIVE}/${download_path}" }
>> > >> >>>>>>>> }
>> > >> >>>>>>>>
>> > >> >>>>>>>> 3. After installation of package (deb+rpm), the directory
>> > >> >>>> structure
>> > >> >>>>>>> related
>> > >> >>>>>>>> to apex will look like following:
>> > >> >>>>>>>>
>> > >> >>>>>>>> /
>> > >> >>>>>>>> `-- usr
>> > >> >>>>>>>>    |-- bin
>> > >> >>>>>>>>    |   |-- apex -> ../lib/apex/bin/apex
>> > >> >>>>>>>>    |   `-- dtcli -> ../lib/apex/bin/apex
>> > >> >>>>>>>>    |-- lib
>> > >> >>>>>>>>    |   `-- apex
>> > >> >>>>>>>>    |       |-- bin
>> > >> >>>>>>>>    |       |   |-- apex
>> > >> >>>>>>>>    |       |   `-- dtcli -> apex
>> > >> >>>>>>>>    |       |-- conf
>> > >> >>>>>>>>    |       |   `-- dt-env.sh
>> > >> >>>>>>>>    |       `-- lib
>> > >> >>>>>>>>    |           |-- apex-api-3.3.0-incubating.jar
>> > >> >>>>>>>>    |           |-- apex-bufferserver-3.3.0-incubating.jar
>> > >> >>>>>>>>    |           |-- apex-common-3.3.0-incubating.jar
>> > >> >>>>>>>>    |           |-- apex-engine.jar
>> > >> >>>>>>>>    |           `-- <other dependency jars>
>> > >> >>>>>>>>    `-- share
>> > >> >>>>>>>>        `-- doc
>> > >> >>>>>>>>            `-- apex
>> > >> >>>>>>>>                |-- changelog.Debian.gz
>> > >> >>>>>>>>                |-- copyright
>> > >> >>>>>>>>                `-- license
>> > >> >>>>>>>>                    `-- LICENSE.gz
>> > >> >>>>>>>>
>> > >> >>>>>>>>
>> > >> >>>>>>>> Please share your thoughts on above approach.
>> > >> >>>>>>>>
>> > >> >>>>>>>>
>> > >> >>>>>>>> Thanks,
>> > >> >>>>>>>> Chinmay.
>> > >> >>>>>>>>
>> > >> >>>>>>>>
>> > >> >>>>>>>>
>> > >> >>>>>>>>
>> > >> >>>>>>>> On Sat, Feb 13, 2016 at 8:38 AM, Thomas Weise <
>> thw@apache.org>
>> > >> >>>> wrote:
>> > >> >>>>>>>>
>> > >> >>>>>>>>> Chinmay,
>> > >> >>>>>>>>>
>> > >> >>>>>>>>> Can you share some details regarding the packaging?
>> > >> >>>>>>>>>
>> > >> >>>>>>>>> Thanks
>> > >> >>>>>>>>>
>> > >> >>>>>>>>> On Thu, Feb 11, 2016 at 12:35 AM, Chinmay Kolhatkar <
>> > >> >>>>>>>>> chinmay@datatorrent.com
>> > >> >>>>>>>>>> wrote:
>> > >> >>>>>>>>>
>> > >> >>>>>>>>>> Dear Community,
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>> Here is the Jira that is created for adding Apex into
>> Bigtop.
>> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/BIGTOP-2313
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>> I've started working on creating packages for Apex.
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>> Please share your thoughts on approach as well as on
>> questions
>> > >> >>>>>>> related to
>> > >> >>>>>>>>>> packaging in my previous mail.
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>> Thanks,
>> > >> >>>>>>>>>> Chinmay.
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>> On Wed, Feb 10, 2016 at 9:55 PM, Chinmay Kolhatkar <
>> > >> >>>>>>>>>> chinmay@datatorrent.com>
>> > >> >>>>>>>>>> wrote:
>> > >> >>>>>>>>>>
>> > >> >>>>>>>>>>> Hello Bigtop Community,
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> Continuing with the packing effort, here is how we're
>> > >> >>>> planning to
>> > >> >>>>>>>>> package
>> > >> >>>>>>>>>>> Apache Apex.
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> Tentative package directory structure for apex would look
>> > >> >>>> like
>> > >> >>>>>>>>> following:
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli
>> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-api-3.3.0-incubating.jar
>> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-engine-3.3.0-incubating.jar
>> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-bufferserver-3.3.0-incubating.jar
>> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-common-3.3.0-incubating.jar
>> > >> >>>>>>>>>>> /usr/lib/apex/lib/<other dependent jars>
>> > >> >>>>>>>>>>> /usr/lib/apex/conf/dt-env.sh
>> > >> >>>>>>>>>>> /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
>> > >> >>>>>>>>>>> /usr/share/doc/man/man1/dtcli.1.gz
>> > >> >>>>>>>>>>> /usr/share/doc/apex/license/LICENSE.txt
>> > >> >>>>>>>>>>> /usr/share/doc/apex/license/<package>-LICENSE.txt
>> > >> >>>>>>>>>>> /usr/share/doc/apex/CHANGELOG
>> > >> >>>>>>>>>>> /usr/share/doc/apex/NOTICE
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> Addition contents to bigtop.bom:
>> > >> >>>>>>>>>>> 1. Dependency of apex will be on "hadoop"
>> > >> >>>>>>>>>>> 2. Component addition would looks like this:
>> > >> >>>>>>>>>>> 'apex' {
>> > >> >>>>>>>>>>>      name    = 'apex'
>> > >> >>>>>>>>>>>      relNotes = 'Apache Apex (incubating)'
>> > >> >>>>>>>>>>>      version { base = '3.3.0-incubating'; pkg = base;
>> > >> >>>> release = 1
>> > >> >>>>>>> }
>> > >> >>>>>>>>>>>      tarball { source      =
>> > >> >>>>>>>>>> "$name-${version.base}-source-release.tar.gz"
>> > >> >>>>>>>>>>>                destination =
>> "$name-${version.base}.tar.gz"
>> > >> >>>> }
>> > >> >>>>>>>>>>>      url     { download_path =
>> > >> >>>>>>> "/incubator/$name/v${version.base}/"
>> > >> >>>>>>>>>>>                site =
>> > >> >>>> "${apache.APACHE_MIRROR}/${download_path}"
>> > >> >>>>>>>>>>>                archive =
>> > >> >>>>>>> "${apache.APACHE_ARCHIVE}/${download_path}" }
>> > >> >>>>>>>>>>> }
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> I have couple of questions related to packaging:
>> > >> >>>>>>>>>>> 1. What is the best possible way to test the package
>> locally?
>> > >> >>>>>>>>>>> 2. Do you have any specific size limitation on size of
>> the
>> > >> >>>> package
>> > >> >>>>>>> (rpm
>> > >> >>>>>>>>>> OR
>> > >> >>>>>>>>>>> deb)?
>> > >> >>>>>>>>>>> 3. Is above proposed packaging information inline with
>> bigtop
>> > >> >>>>>>>>> standards?
>> > >> >>>>>>>>>>> 4. Am I missing anything there?
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> Thanks,
>> > >> >>>>>>>>>>> Chinmay.
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>> On Fri, Feb 5, 2016 at 12:01 AM, Chinmay Kolhatkar <
>> > >> >>>>>>>>>>> chinmay@datatorrent.com> wrote:
>> > >> >>>>>>>>>>>
>> > >> >>>>>>>>>>>> Hello Bigtop community!
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> My name is Chinmay Kolhatkar and I am a committer in
>> Apache
>> > >> >>>> Apex
>> > >> >>>>>>>>>>>> (incubating) project.
>> > >> >>>>>>>>>>>> Our website is at: http://apex.incubator.apache.org/
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> We want to explore and integrate Apache Apex into the
>> > >> >>>> Bigtop as a
>> > >> >>>>>>>>>>>> component.
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> Please treat this mail as a continuation of a mail
>> thread:
>> > >> >>>>
>> > >>
>> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201601.mbox/%3CCA%2B5xAo20CAHuYtdxw3mtH2X9whREKd7fLdz4xmuTMqkaTiiskg%40mail.gmail.com%3E
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> We're currently confirming with requirements for
>> components
>> > >> >>>> and
>> > >> >>>>>>> making
>> > >> >>>>>>>>>>>> sure that we meet the expectations:
>> > >> >>>>
>> > >>
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> Once requirements are through, we're planning to add
>> > >> >>>> component
>> > >> >>>>>>> into
>> > >> >>>>>>>>>>>> Apache Bigtop in following phases (as separate
>> patches/pull
>> > >> >>>>>>> requests):
>> > >> >>>>>>>>>>>> 1. Packaging
>> > >> >>>>>>>>>>>>    - Creating of package i.e. rpm & deb files.
>> > >> >>>>>>>>>>>>    - documentations/READMEs, LICENSE, DISCLAMER, NOTES
>> etc
>> > >> >>>> if any
>> > >> >>>>>>>>>> needed.
>> > >> >>>>>>>>>>>>    - Any documentation that need to be added to
>> > >> >>>> distribution of
>> > >> >>>>>>> our
>> > >> >>>>>>>>>>>> software.
>> > >> >>>>>>>>>>>>    - Any license information of dependencies required
>> to be
>> > >> >>>>>>> added to
>> > >> >>>>>>>>>>>> package
>> > >> >>>>>>>>>>>> 2. Smoke tests
>> > >> >>>>>>>>>>>>    - Adding smoke test for packaging.
>> > >> >>>>>>>>>>>> 3. Puppet recipes for automatic deployment and
>> configuration
>> > >> >>>>>>>>>>>>    - Add puppet recipes for automatic deployment
>> > >> >>>>>>>>>>>> 4. license clearance:
>> > >> >>>>>>>>>>>>    - Run 'gradle rat' to make sure all new stuff is
>> > >> >>>> compliant
>> > >> >>>>>>> with
>> > >> >>>>>>>>> ASF
>> > >> >>>>>>>>>>>> license requirements.
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> Please share your thoughts on above approach.
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> If the approach seems fine, we can create a Jira items
>> and
>> > >> >>>>>>>>> corresponding
>> > >> >>>>>>>>>>>> subtasks to start working.
>> > >> >>>>>>>>>>>>
>> > >> >>>>>>>>>>>> Thanks,
>> > >> >>>>>>>>>>>> Chinmay.
>> > >> >>
>> > >>
>> > >
>> > >
>>
>
>