Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@datatorrent.com>]

Hi All,

We're planning a work on adding Apache Apex as a component to Apache Bigtop.
Bigtop is the packaging system for the Apache big data ecosystem. Several
Hadoop distros use it, most recently EMR.

Here is the tracking Jira task in APEXCORE for the same:
https://issues.apache.org/jira/browse/APEXCORE-331

Proposed plan of execution is as follows:
*Step 1) Handle prerequisites*
Apache bigtop has some hard and soft expectation for new components to get
integrated into Bigtop.
Here is the list of it:
https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution

Most of them seems to be standard ASF based requirements, but few need to
be checked for:
1. Software projects are expected to be Licensed under Apache License,
Version 2.0 (and their dependencies are expected to be compatible with this
license)
    - Apex is under ASL 2.0 but need to check if dependencies of Apex are
compatible with ASL 2.0. This I guess would be a verification check.
2. Software projects are expected to be compatible with all of the
supported platforms that Bigtop distribution is targeting
    - This needs verifying whether our software runs fine in centos-6
centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
3. What smoke tests that should be added for deployment.
4. Identifying the test artifacts which goes beyond smoke test
    - These are basically the integration tests for verification after the
deployment. This is a soft requirement, but aim is to achieve this as well
or at least have explanation why not to include.

If there are any from the link which explicitly needs to be checked other
than above 4, please let us know.

*Step 2) Adding Apex as component to Bigtop*
>From one of the mail archive of Bigtop, it was learnt that the bigtop
community want to see the addition of new components in phases. Here are
the phases:
1. Packaging
    - This needs creating of package i.e. rpm & deb files.
    - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if any needed.
    - Any documentation that need to be added to distribution of our
software.
    - Any license information of dependencies required to be added to
package
2. Smoke tests (at very least)
    - Adding smoke test for packaging.
3. Puppet recipes for automatic deployment and configuration
    - Add puppet recipes for automatic deployment
4. Integration tests
    - For verification of deployments.
5. license clearance:
    Run 'gradle rat' to make sure all new stuff is compliant with ASF
license requirements. If you add code licenses under different licenses,
those would need to be listed in the NOTICE.

Please share your thoughts on the approach.
We'll start corresponding communication on bigtop mailing list as well.

We have some specific questions/suggestions related to what should be the
content of the package and what should be the smoke tests, but in the
interest of not having too much content here, we'll put the questions as a
separate mail in this mailthread.

Thanks,
Chinmay.

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

AFAIK these files are created on demand. If there is nothing that needs to
be there by default then no need for the files in the package.

On Wed, Feb 10, 2016 at 9:03 AM, Aniruddha Thombare <
aniruddha@datatorrent.com> wrote:

> AFI recall...
>
> Clirc
> Clirc history
> Etc
>
> Default site.xml and env can be kept too.
>
>
>
>
> Sent from handheld
>
> On Wed, 10 Feb 2016 10:25 pm Chinmay Kolhatkar <ch...@datatorrent.com>
> wrote:
>
> > Ok. Is there any content that we should add in .dt ?
> > On 10 Feb 2016 22:10, "Aniruddha Thombare" <an...@datatorrent.com>
> > wrote:
> >
> > > Hi,
> > >
> > > .dt directory.
> > > In future, it may  become .apex
> > > On 10 Feb 2016 9:53 pm, "Chinmay Kolhatkar" <ch...@datatorrent.com>
> > > wrote:
> > >
> > > > Hi Aniruddha,
> > > >
> > > > I like the idea of skeleton directory. But I don't see any use of it,
> > > > unless we've anything in default.
> > > > If you have something in your mind which should be default, can you
> > > please
> > > > share it?
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > > On Wed, Feb 10, 2016 at 9:49 PM, Aniruddha Thombare <
> > > > aniruddha@datatorrent.com> wrote:
> > > >
> > > > > On point 2:
> > > > > Skeleton directory is for default home dir structure for future
> > users.
> > > > > By avoiding that, we can't inconvenience users and admins.
> > > > > Defaults in that configuration can be specified or evaluated at
> > > install.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <
> > chinmay@datatorrent.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Really good points Pradeep and Aniruddha.
> > > > > >
> > > > > > 1. I believe we won't need to change the dtcli considering it
> works
> > > > with
> > > > > DT
> > > > > > Community edition. We can keep the directory structure similar to
> > > that.
> > > > > > dt-env.sh has variables which contains information required for
> > dtcli
> > > > to
> > > > > > launch.
> > > > > >
> > > > > > 2. Let me check with bigtop community that whether they
> facilitate
> > > the
> > > > > > installation of rpms/debs before user is created. In either case,
> > > > current
> > > > > > dtcli creates a .dt folder in home directory. Also before putting
> > > > > anything
> > > > > > in /etc/skel we need to define what are the default contents that
> > > > should
> > > > > go
> > > > > > to ~/.dt/ folder. If there is no defaults, probably we should not
> > > > > > explicitly add it in /etc/skel.
> > > > > >
> > > > > > 3.  /etc/profile.d approach looks nice. This way contents of
> > > dt-env.sh
> > > > > are
> > > > > > present as env variables. But I see a catch there. Adding
> dt-env.sh
> > > to
> > > > > > /etc/profile.d would make all the variables available at runtime
> > all
> > > > the
> > > > > > time. I feel a little skeptical about that. Maybe a possible
> > > collision
> > > > > can
> > > > > > occur with other application vars.
> > > > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > > > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > > > > >
> > > > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > > > > Jira/repository.
> > > > > > Packaging related code usually goes there. (That's what all the
> > > > > components
> > > > > > in bigtop does).
> > > > > > Having said that, once a PR is created, I'll be sharing the link
> of
> > > PR
> > > > > > here, so that apex community as well can review it.
> > > > > >
> > > > > > Thanks,
> > > > > > Chinmay.
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > > > > aniruddha@datatorrent.com> wrote:
> > > > > >
> > > > > > > +1 on suggestions and approach.
> > > > > > > We may need to iron out details about exact paths etc.
> > > > > > > Which can be done on jira / PR comments.
> > > > > > > Is that right @dev?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > > apache@pradeepdalvi.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Inline comments...
> > > > > > > >
> > > > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > > > > > chinmay@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > @Thomas, Not all the jar present in DT community edition
> will
> > > be
> > > > > > > included
> > > > > > > > > there.
> > > > > > > > >
> > > > > > > > > Here is the list of jars I found from common dependencies
> > > between
> > > > > of
> > > > > > > apex
> > > > > > > > > and DT Community edition:
> > > > > > > > > netlet-1.2.0.jar
> > > > > > > > > kryo-2.24.0.jar
> > > > > > > > > jackson-core-asl-1.9.2.jar
> > > > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > > > > > async-http-client-1.7.20.jar
> > > > > > > > > netty-3.6.6.Final.jar
> > > > > > > > > validation-api-1.1.0.Final.jar
> > > > > > > > > bval-jsr303-0.5.jar
> > > > > > > > > bval-core-0.5.jar
> > > > > > > > > commons-lang3-3.1.jar
> > > > > > > > > commons-beanutils-1.8.3.jar
> > > > > > > > > httpclient-4.3.5.jar
> > > > > > > > > commons-codec-1.10.jar
> > > > > > > > > zip4j-1.3.2.jar
> > > > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > > > > jersey-apache-client4-1.9.jar
> > > > > > > > > jline-2.11.jar
> > > > > > > > > ant-1.9.2.jar
> > > > > > > > > ant-launcher-1.9.2.jar
> > > > > > > > > mbassador-1.1.9.jar
> > > > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > > > > jackson-xc-1.9.2.jar
> > > > > > > > > hadoop-common-2.2.0-tests.jar
> > > > > > > > >
> > > > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> > works
> > > > > > properly
> > > > > > > > for
> > > > > > > > > launch+shutdown+kill of apps with only these libraries
> > present
> > > in
> > > > > > > > isolation
> > > > > > > > > without dependency on local m2.
> > > > > > > > > I'm believe that there are unwanted jars which are used for
> > > > compile
> > > > > > > time
> > > > > > > > > dependency and not runtime in above list which I can drop
> to
> > > keep
> > > > > > > package
> > > > > > > > > size minimal.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > @Anniruddha,
> > > > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> > > required.
> > > > > > Only
> > > > > > > > > required values I see in it are classpath. and dtcli is
> > capable
> > > > of
> > > > > > > > building
> > > > > > > > > it on the fly.
> > > > > > > > >
> > > > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find
> > the
> > > > jar
> > > > > > > path,
> > > > > > > > > dt-env.sh and all such conf files. If we need a change in
> > > dtcli,
> > > > > then
> > > > > > > > > community's opinion is required for whether dtcli should
> > change
> > > > in
> > > > > > our
> > > > > > > > repo
> > > > > > > > > or a copy of that with required changes exist in bigtop
> repo
> > > > until
> > > > > we
> > > > > > > > make
> > > > > > > > > the dtcli generic enough.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Can we have these file paths set using environment variables
> > with
> > > > > some
> > > > > > > > default values in dtcli?
> > > > > > > > Then we can set such params in dt-env.sh.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 3. dt-sited.xml -> How is this file different from
> > dt-site.xml?
> > > > Not
> > > > > > > sure
> > > > > > > > if
> > > > > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel
> > are
> > > > > > copied
> > > > > > > to
> > > > > > > > > home of new user when useradd program is called. But for
> > > existing
> > > > > > users
> > > > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on
> > the
> > > > fly
> > > > > > if
> > > > > > > > not
> > > > > > > > > encountered for the first time. Again correct me if I'm
> > wrong.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yes, you are right. However this would probably be necessary
> > step
> > > > for
> > > > > > > > rpm/deb, as they may also get installed during OS install and
> > > > before
> > > > > > user
> > > > > > > > accounts were created.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
> > best
> > > > > idea.
> > > > > > > > We'll
> > > > > > > > > anyway put dtcli in location which is by default present in
> > > path
> > > > > i.e.
> > > > > > > > > /usr/bin.
> > > > > > > >
> > > > > > > > For other env variables specific to apex, I'll prefer to use
> > > > > dt-env.sh
> > > > > > > and
> > > > > > > > > source it in dtcli rather than changing bashrc etc...
> > > > > > > >
> > > > > > > >
> > > > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> > cleaner
> > > > way
> > > > > > to
> > > > > > > > achieve the same.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > > Chinmay.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > > > > > aniruddha@datatorrent.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > @Chinmay,
> > > > > > > > > >
> > > > > > > > > > We need to consider following:
> > > > > > > > > > System wide default config files can be located at
> > following
> > > > > > > locations:
> > > > > > > > > >
> > > > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> behaviour
> > > on
> > > > > how
> > > > > > it
> > > > > > > > > finds
> > > > > > > > > > those locations)
> > > > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> > behaviour
> > > on
> > > > > how
> > > > > > > it
> > > > > > > > > > finds those locations)
> > > > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users
> > > that
> > > > > will
> > > > > > > be
> > > > > > > > > > created in system in future.)
> > > > > > > > > >
> > > > > > > > > > We may also have to modify bashrc / bashprofile for
> > > population
> > > > > any
> > > > > > > > > > variables that are required.
> > > > > > > > > >
> > > > > > > > > > @dev, please put in your comments / suggestion.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Aniruddha
> > > > > > > > > >
> > > > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > > > > > thomas@datatorrent.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > -->
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > > > > > chinmay@datatorrent.com
> > > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello Everyone!!
> > > > > > > > > > > >
> > > > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
> > here
> > > > are
> > > > > > some
> > > > > > > > > > > proposals
> > > > > > > > > > > > about package structure etc..
> > > > > > > > > > > > Before posting it on bbigtop mailing list, I have
> some
> > > > > question
> > > > > > > for
> > > > > > > > > > apex
> > > > > > > > > > > > community.
> > > > > > > > > > > >
> > > > > > > > > > > > Proposed Directory structure of apex package for both
> > > deb &
> > > > > > rpm:
> > > > > > > > > > > >
> > > > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > > > > > >
> > > > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > > > > > <other dependent jars> = All the 3rd party jars which
> > are
> > > > > > > required
> > > > > > > > > for
> > > > > > > > > > > apex
> > > > > > > > > > > > to run. Usually the dependencies are packaged as part
> > of
> > > > > > rpm/deb
> > > > > > > by
> > > > > > > > > any
> > > > > > > > > > > > software in bigtop.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Can you specify what those jars are in Bigtop context?
> > Same
> > > > as
> > > > > > > > shipped
> > > > > > > > > > with
> > > > > > > > > > > DT community addition under lib/ ?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
> > > party
> > > > > jars
> > > > > > > > which
> > > > > > > > > > > needs
> > > > > > > > > > > > to included while packaging.
> > > > > > > > > > > >
> > > > > > > > > > > > Questions related to this:
> > > > > > > > > > > > 1. Should we call the cli of apex as "apex" instead
> of
> > > > > "dtcli"
> > > > > > in
> > > > > > > > > > bigtop
> > > > > > > > > > > > package?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I think we should keep the name until we are able to
> > change
> > > > it
> > > > > in
> > > > > > > > Apex.
> > > > > > > > > > > Otherwise this may get confusing.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > 2. I see that all softwares in bigtop have man page
> for
> > > > their
> > > > > > > > > > > executables.
> > > > > > > > > > > > I think we should have it too for dtcli. Is there any
> > > > > > > documentation
> > > > > > > > > > > which I
> > > > > > > > > > > > can convert to man page? or can I use output of
> "dtcli
> > > > > --help"?
> > > > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> > 3.3.0
> > > OR
> > > > > > > > > > > > 3.3.0-incubating?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Has to be -incubating
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
> > 2.7.1
> > > > > > > version
> > > > > > > > of
> > > > > > > > > > > > bigtop hadoop? Any problems that we see with this
> > > > dependency?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Should work without changed. I thought we certified
> > against
> > > > > > 2.7.0?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > 5. Following is the apache mirror from which bigtop
> > will
> > > > pick
> > > > > > the
> > > > > > > > > apex
> > > > > > > > > > > > source for compilation and packaging. Please correct
> if
> > > > > > > incorrect:
> > > > > > > > > > > >
> > > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
> all
> > > the
> > > > > > > > softwares
> > > > > > > > > in
> > > > > > > > > > > > bigtop.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Please share your opinion.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Chinmay.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > > > > > > chinmay@datatorrent.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've already started discussion on bigtop dev
> mailing
> > > > list
> > > > > > for
> > > > > > > > > > > > > integration. Also created a Jira for the same. For
> > this
> > > > > > > > communities
> > > > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > > > > > > thomas@datatorrent.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Chinmay,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> I don't see anything under prerequisites that
> would
> > > be a
> > > > > > > > problem.
> > > > > > > > > We
> > > > > > > > > > > > >> looked
> > > > > > > > > > > > >> at the ASF licencing compatibility as part of
> > becoming
> > > > an
> > > > > > > > > incubator
> > > > > > > > > > > > >> project.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Please focus on the packaging during the next
> weeks.
> > > > Since
> > > > > > the
> > > > > > > > > work
> > > > > > > > > > > will
> > > > > > > > > > > > >> be
> > > > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> should
> > > > also
> > > > > be
> > > > > > > > > there.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Would be good to have the packaging in place by
> end
> > > Feb.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >> Thomas
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> Kolhatkar <
> > > > > > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > We're planning a work on adding Apache Apex as a
> > > > > component
> > > > > > > to
> > > > > > > > > > Apache
> > > > > > > > > > > > >> > Bigtop.
> > > > > > > > > > > > >> > Bigtop is the packaging system for the Apache
> big
> > > data
> > > > > > > > > ecosystem.
> > > > > > > > > > > > >> Several
> > > > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for
> the
> > > > same:
> > > > > > > > > > > > >> >
> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > > > > > >> > Apache bigtop has some hard and soft expectation
> > for
> > > > new
> > > > > > > > > > components
> > > > > > > > > > > to
> > > > > > > > > > > > >> get
> > > > > > > > > > > > >> > integrated into Bigtop.
> > > > > > > > > > > > >> > Here is the list of it:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Most of them seems to be standard ASF based
> > > > > requirements,
> > > > > > > but
> > > > > > > > > few
> > > > > > > > > > > need
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> > be checked for:
> > > > > > > > > > > > >> > 1. Software projects are expected to be Licensed
> > > under
> > > > > > > Apache
> > > > > > > > > > > License,
> > > > > > > > > > > > >> > Version 2.0 (and their dependencies are expected
> > to
> > > be
> > > > > > > > > compatible
> > > > > > > > > > > with
> > > > > > > > > > > > >> this
> > > > > > > > > > > > >> > license)
> > > > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > > > > > > dependencies
> > > > > > > > of
> > > > > > > > > > > Apex
> > > > > > > > > > > > >> are
> > > > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > > > > > > verification
> > > > > > > > > > check.
> > > > > > > > > > > > >> > 2. Software projects are expected to be
> compatible
> > > > with
> > > > > > all
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > >> > supported platforms that Bigtop distribution is
> > > > > targeting
> > > > > > > > > > > > >> >     - This needs verifying whether our software
> > runs
> > > > > fine
> > > > > > in
> > > > > > > > > > > centos-6
> > > > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > > > opensuse-13.2.
> > > > > > > > > > > > >> > 3. What smoke tests that should be added for
> > > > deployment.
> > > > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> > beyond
> > > > > smoke
> > > > > > > test
> > > > > > > > > > > > >> >     - These are basically the integration tests
> > for
> > > > > > > > verification
> > > > > > > > > > > after
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> > deployment. This is a soft requirement, but aim
> is
> > > to
> > > > > > > achieve
> > > > > > > > > this
> > > > > > > > > > > as
> > > > > > > > > > > > >> well
> > > > > > > > > > > > >> > or at least have explanation why not to include.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > If there are any from the link which explicitly
> > > needs
> > > > to
> > > > > > be
> > > > > > > > > > checked
> > > > > > > > > > > > >> other
> > > > > > > > > > > > >> > than above 4, please let us know.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
> > > learnt
> > > > > that
> > > > > > > the
> > > > > > > > > > > bigtop
> > > > > > > > > > > > >> > community want to see the addition of new
> > components
> > > > in
> > > > > > > > phases.
> > > > > > > > > > Here
> > > > > > > > > > > > are
> > > > > > > > > > > > >> > the phases:
> > > > > > > > > > > > >> > 1. Packaging
> > > > > > > > > > > > >> >     - This needs creating of package i.e. rpm &
> > deb
> > > > > files.
> > > > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> DISCLAMER,
> > > > NOTES
> > > > > > etc
> > > > > > > if
> > > > > > > > > any
> > > > > > > > > > > > >> needed.
> > > > > > > > > > > > >> >     - Any documentation that need to be added to
> > > > > > > distribution
> > > > > > > > of
> > > > > > > > > > our
> > > > > > > > > > > > >> > software.
> > > > > > > > > > > > >> >     - Any license information of dependencies
> > > required
> > > > > to
> > > > > > be
> > > > > > > > > added
> > > > > > > > > > > to
> > > > > > > > > > > > >> > package
> > > > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > > > > > configuration
> > > > > > > > > > > > >> >     - Add puppet recipes for automatic
> deployment
> > > > > > > > > > > > >> > 4. Integration tests
> > > > > > > > > > > > >> >     - For verification of deployments.
> > > > > > > > > > > > >> > 5. license clearance:
> > > > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff
> is
> > > > > > compliant
> > > > > > > > > with
> > > > > > > > > > > ASF
> > > > > > > > > > > > >> > license requirements. If you add code licenses
> > under
> > > > > > > different
> > > > > > > > > > > > licenses,
> > > > > > > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > > > > > >> > We'll start corresponding communication on
> bigtop
> > > > > mailing
> > > > > > > list
> > > > > > > > > as
> > > > > > > > > > > > well.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > We have some specific questions/suggestions
> > related
> > > to
> > > > > > what
> > > > > > > > > should
> > > > > > > > > > > be
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> > content of the package and what should be the
> > smoke
> > > > > tests,
> > > > > > > but
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > >> > interest of not having too much content here,
> > we'll
> > > > put
> > > > > > the
> > > > > > > > > > > questions
> > > > > > > > > > > > >> as a
> > > > > > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Thanks,
> > > > > > > > > > > > >> > Chinmay.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Aniruddha Thombare <an...@datatorrent.com>]

+1 On that! It will ease few things.

Thanks,


Aniruddha

On Thu, Feb 11, 2016 at 12:02 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:

> Hi Everyone,
>
> I just wanted to mention a learning while packing apex for bigtop
> (Something we can think about in future).
> We should probably have a mvn profile at top level pom which can called
> optionally be invoked and which will package the apex and all its
> dependency in a single tarball. This tarball can then just be extracted and
> one is readily use apex.
>
> I don't know whether this is inline with ASF guidelines, but this I think
> this can be a nice addition.
> Something to think about in future.
>
> Thanks,
> Chinmay.
>
>
> On Wed, Feb 10, 2016 at 10:36 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com
> > wrote:
>
> > Ok... Then make sense to use skeleton.
> > Let's wait for rest of the community as well to provide their opinion.
> >
> > - Chinmay.
> > On 10 Feb 2016 22:34, "Aniruddha Thombare" <an...@datatorrent.com>
> > wrote:
> >
> >> AFI recall...
> >>
> >> Clirc
> >> Clirc history
> >> Etc
> >>
> >> Default site.xml and env can be kept too.
> >>
> >>
> >>
> >>
> >> Sent from handheld
> >>
> >> On Wed, 10 Feb 2016 10:25 pm Chinmay Kolhatkar <chinmay@datatorrent.com
> >
> >> wrote:
> >>
> >> > Ok. Is there any content that we should add in .dt ?
> >> > On 10 Feb 2016 22:10, "Aniruddha Thombare" <aniruddha@datatorrent.com
> >
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > .dt directory.
> >> > > In future, it may  become .apex
> >> > > On 10 Feb 2016 9:53 pm, "Chinmay Kolhatkar" <
> chinmay@datatorrent.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Aniruddha,
> >> > > >
> >> > > > I like the idea of skeleton directory. But I don't see any use of
> >> it,
> >> > > > unless we've anything in default.
> >> > > > If you have something in your mind which should be default, can
> you
> >> > > please
> >> > > > share it?
> >> > > >
> >> > > > Thanks,
> >> > > > Chinmay.
> >> > > >
> >> > > >
> >> > > > On Wed, Feb 10, 2016 at 9:49 PM, Aniruddha Thombare <
> >> > > > aniruddha@datatorrent.com> wrote:
> >> > > >
> >> > > > > On point 2:
> >> > > > > Skeleton directory is for default home dir structure for future
> >> > users.
> >> > > > > By avoiding that, we can't inconvenience users and admins.
> >> > > > > Defaults in that configuration can be specified or evaluated at
> >> > > install.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <
> >> > chinmay@datatorrent.com
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Really good points Pradeep and Aniruddha.
> >> > > > > >
> >> > > > > > 1. I believe we won't need to change the dtcli considering it
> >> works
> >> > > > with
> >> > > > > DT
> >> > > > > > Community edition. We can keep the directory structure similar
> >> to
> >> > > that.
> >> > > > > > dt-env.sh has variables which contains information required
> for
> >> > dtcli
> >> > > > to
> >> > > > > > launch.
> >> > > > > >
> >> > > > > > 2. Let me check with bigtop community that whether they
> >> facilitate
> >> > > the
> >> > > > > > installation of rpms/debs before user is created. In either
> >> case,
> >> > > > current
> >> > > > > > dtcli creates a .dt folder in home directory. Also before
> >> putting
> >> > > > > anything
> >> > > > > > in /etc/skel we need to define what are the default contents
> >> that
> >> > > > should
> >> > > > > go
> >> > > > > > to ~/.dt/ folder. If there is no defaults, probably we should
> >> not
> >> > > > > > explicitly add it in /etc/skel.
> >> > > > > >
> >> > > > > > 3.  /etc/profile.d approach looks nice. This way contents of
> >> > > dt-env.sh
> >> > > > > are
> >> > > > > > present as env variables. But I see a catch there. Adding
> >> dt-env.sh
> >> > > to
> >> > > > > > /etc/profile.d would make all the variables available at
> runtime
> >> > all
> >> > > > the
> >> > > > > > time. I feel a little skeptical about that. Maybe a possible
> >> > > collision
> >> > > > > can
> >> > > > > > occur with other application vars.
> >> > > > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> >> > > > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> >> > > > > >
> >> > > > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> >> > > > > Jira/repository.
> >> > > > > > Packaging related code usually goes there. (That's what all
> the
> >> > > > > components
> >> > > > > > in bigtop does).
> >> > > > > > Having said that, once a PR is created, I'll be sharing the
> >> link of
> >> > > PR
> >> > > > > > here, so that apex community as well can review it.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Chinmay.
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> >> > > > > > aniruddha@datatorrent.com> wrote:
> >> > > > > >
> >> > > > > > > +1 on suggestions and approach.
> >> > > > > > > We may need to iron out details about exact paths etc.
> >> > > > > > > Which can be done on jira / PR comments.
> >> > > > > > > Is that right @dev?
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> >> > > > apache@pradeepdalvi.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Inline comments...
> >> > > > > > > >
> >> > > > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> >> > > > > > > > chinmay@datatorrent.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > @Thomas, Not all the jar present in DT community edition
> >> will
> >> > > be
> >> > > > > > > included
> >> > > > > > > > > there.
> >> > > > > > > > >
> >> > > > > > > > > Here is the list of jars I found from common
> dependencies
> >> > > between
> >> > > > > of
> >> > > > > > > apex
> >> > > > > > > > > and DT Community edition:
> >> > > > > > > > > netlet-1.2.0.jar
> >> > > > > > > > > kryo-2.24.0.jar
> >> > > > > > > > > jackson-core-asl-1.9.2.jar
> >> > > > > > > > > jackson-mapper-asl-1.9.2.jar
> >> > > > > > > > > async-http-client-1.7.20.jar
> >> > > > > > > > > netty-3.6.6.Final.jar
> >> > > > > > > > > validation-api-1.1.0.Final.jar
> >> > > > > > > > > bval-jsr303-0.5.jar
> >> > > > > > > > > bval-core-0.5.jar
> >> > > > > > > > > commons-lang3-3.1.jar
> >> > > > > > > > > commons-beanutils-1.8.3.jar
> >> > > > > > > > > httpclient-4.3.5.jar
> >> > > > > > > > > commons-codec-1.10.jar
> >> > > > > > > > > zip4j-1.3.2.jar
> >> > > > > > > > > jetty-websocket-8.1.10.v20130312.jar
> >> > > > > > > > > xbean-asm5-shaded-4.3.jar
> >> > > > > > > > > jersey-apache-client4-1.9.jar
> >> > > > > > > > > jline-2.11.jar
> >> > > > > > > > > ant-1.9.2.jar
> >> > > > > > > > > ant-launcher-1.9.2.jar
> >> > > > > > > > > mbassador-1.1.9.jar
> >> > > > > > > > > jackson-jaxrs-1.9.2.jar
> >> > > > > > > > > jackson-xc-1.9.2.jar
> >> > > > > > > > > hadoop-common-2.2.0-tests.jar
> >> > > > > > > > >
> >> > > > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> >> > works
> >> > > > > > properly
> >> > > > > > > > for
> >> > > > > > > > > launch+shutdown+kill of apps with only these libraries
> >> > present
> >> > > in
> >> > > > > > > > isolation
> >> > > > > > > > > without dependency on local m2.
> >> > > > > > > > > I'm believe that there are unwanted jars which are used
> >> for
> >> > > > compile
> >> > > > > > > time
> >> > > > > > > > > dependency and not runtime in above list which I can
> drop
> >> to
> >> > > keep
> >> > > > > > > package
> >> > > > > > > > > size minimal.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > @Anniruddha,
> >> > > > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> >> > > required.
> >> > > > > > Only
> >> > > > > > > > > required values I see in it are classpath. and dtcli is
> >> > capable
> >> > > > of
> >> > > > > > > > building
> >> > > > > > > > > it on the fly.
> >> > > > > > > > >
> >> > > > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can
> >> find
> >> > the
> >> > > > jar
> >> > > > > > > path,
> >> > > > > > > > > dt-env.sh and all such conf files. If we need a change
> in
> >> > > dtcli,
> >> > > > > then
> >> > > > > > > > > community's opinion is required for whether dtcli should
> >> > change
> >> > > > in
> >> > > > > > our
> >> > > > > > > > repo
> >> > > > > > > > > or a copy of that with required changes exist in bigtop
> >> repo
> >> > > > until
> >> > > > > we
> >> > > > > > > > make
> >> > > > > > > > > the dtcli generic enough.
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Can we have these file paths set using environment
> variables
> >> > with
> >> > > > > some
> >> > > > > > > > default values in dtcli?
> >> > > > > > > > Then we can set such params in dt-env.sh.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > > 3. dt-sited.xml -> How is this file different from
> >> > dt-site.xml?
> >> > > > Not
> >> > > > > > > sure
> >> > > > > > > > if
> >> > > > > > > > > adding in /etc/skel would help. Files & Dirs from
> >> /etc/skel
> >> > are
> >> > > > > > copied
> >> > > > > > > to
> >> > > > > > > > > home of new user when useradd program is called. But for
> >> > > existing
> >> > > > > > users
> >> > > > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/
> >> on
> >> > the
> >> > > > fly
> >> > > > > > if
> >> > > > > > > > not
> >> > > > > > > > > encountered for the first time. Again correct me if I'm
> >> > wrong.
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Yes, you are right. However this would probably be
> necessary
> >> > step
> >> > > > for
> >> > > > > > > > rpm/deb, as they may also get installed during OS install
> >> and
> >> > > > before
> >> > > > > > user
> >> > > > > > > > accounts were created.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its
> the
> >> > best
> >> > > > > idea.
> >> > > > > > > > We'll
> >> > > > > > > > > anyway put dtcli in location which is by default present
> >> in
> >> > > path
> >> > > > > i.e.
> >> > > > > > > > > /usr/bin.
> >> > > > > > > >
> >> > > > > > > > For other env variables specific to apex, I'll prefer to
> use
> >> > > > > dt-env.sh
> >> > > > > > > and
> >> > > > > > > > > source it in dtcli rather than changing bashrc etc...
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> >> > cleaner
> >> > > > way
> >> > > > > > to
> >> > > > > > > > achieve the same.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > > Chinmay.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> >> > > > > > > > > aniruddha@datatorrent.com> wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi,
> >> > > > > > > > > >
> >> > > > > > > > > > @Chinmay,
> >> > > > > > > > > >
> >> > > > > > > > > > We need to consider following:
> >> > > > > > > > > > System wide default config files can be located at
> >> > following
> >> > > > > > > locations:
> >> > > > > > > > > >
> >> > > > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> >> behaviour
> >> > > on
> >> > > > > how
> >> > > > > > it
> >> > > > > > > > > finds
> >> > > > > > > > > > those locations)
> >> > > > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> >> > behaviour
> >> > > on
> >> > > > > how
> >> > > > > > > it
> >> > > > > > > > > > finds those locations)
> >> > > > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new
> >> users
> >> > > that
> >> > > > > will
> >> > > > > > > be
> >> > > > > > > > > > created in system in future.)
> >> > > > > > > > > >
> >> > > > > > > > > > We may also have to modify bashrc / bashprofile for
> >> > > population
> >> > > > > any
> >> > > > > > > > > > variables that are required.
> >> > > > > > > > > >
> >> > > > > > > > > > @dev, please put in your comments / suggestion.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Aniruddha
> >> > > > > > > > > >
> >> > > > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> >> > > > > > > thomas@datatorrent.com>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > -->
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> >> > > > > > > > > > chinmay@datatorrent.com
> >> > > > > > > > > > > >
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Hello Everyone!!
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Continuing with packaging effort (rpm+deb) of
> apex,
> >> > here
> >> > > > are
> >> > > > > > some
> >> > > > > > > > > > > proposals
> >> > > > > > > > > > > > about package structure etc..
> >> > > > > > > > > > > > Before posting it on bbigtop mailing list, I have
> >> some
> >> > > > > question
> >> > > > > > > for
> >> > > > > > > > > > apex
> >> > > > > > > > > > > > community.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Proposed Directory structure of apex package for
> >> both
> >> > > deb &
> >> > > > > > rpm:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > /usr/lib/apex/bin/dtcli
> >> > > > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> >> > > > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> >> > > > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >> > > > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> >> > > > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> >> > > > > > > > > > >
> >> > > > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> >> > > > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> >> > > > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> >> > > > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > > > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> >> > > > > > > > > > > > /usr/share/doc/apex/NOTICE
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > <version> = 3.3.0-incubating.
> >> > > > > > > > > > > > <other dependent jars> = All the 3rd party jars
> >> which
> >> > are
> >> > > > > > > required
> >> > > > > > > > > for
> >> > > > > > > > > > > apex
> >> > > > > > > > > > > > to run. Usually the dependencies are packaged as
> >> part
> >> > of
> >> > > > > > rpm/deb
> >> > > > > > > by
> >> > > > > > > > > any
> >> > > > > > > > > > > > software in bigtop.
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > Can you specify what those jars are in Bigtop
> context?
> >> > Same
> >> > > > as
> >> > > > > > > > shipped
> >> > > > > > > > > > with
> >> > > > > > > > > > > DT community addition under lib/ ?
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > > <package LICENSE> = Licenses of corresponding of
> 3rd
> >> > > party
> >> > > > > jars
> >> > > > > > > > which
> >> > > > > > > > > > > needs
> >> > > > > > > > > > > > to included while packaging.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Questions related to this:
> >> > > > > > > > > > > > 1. Should we call the cli of apex as "apex"
> instead
> >> of
> >> > > > > "dtcli"
> >> > > > > > in
> >> > > > > > > > > > bigtop
> >> > > > > > > > > > > > package?
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > I think we should keep the name until we are able to
> >> > change
> >> > > > it
> >> > > > > in
> >> > > > > > > > Apex.
> >> > > > > > > > > > > Otherwise this may get confusing.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > > 2. I see that all softwares in bigtop have man
> page
> >> for
> >> > > > their
> >> > > > > > > > > > > executables.
> >> > > > > > > > > > > > I think we should have it too for dtcli. Is there
> >> any
> >> > > > > > > documentation
> >> > > > > > > > > > > which I
> >> > > > > > > > > > > > can convert to man page? or can I use output of
> >> "dtcli
> >> > > > > --help"?
> >> > > > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> >> > 3.3.0
> >> > > OR
> >> > > > > > > > > > > > 3.3.0-incubating?
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > Has to be -incubating
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > > 4. Is it ok for apex package of bigtop to depend
> on
> >> > 2.7.1
> >> > > > > > > version
> >> > > > > > > > of
> >> > > > > > > > > > > > bigtop hadoop? Any problems that we see with this
> >> > > > dependency?
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > Should work without changed. I thought we certified
> >> > against
> >> > > > > > 2.7.0?
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > > 5. Following is the apache mirror from which
> bigtop
> >> > will
> >> > > > pick
> >> > > > > > the
> >> > > > > > > > > apex
> >> > > > > > > > > > > > source for compilation and packaging. Please
> >> correct if
> >> > > > > > > incorrect:
> >> > > > > > > > > > > >
> >> > > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >> > > > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
> >> all
> >> > > the
> >> > > > > > > > softwares
> >> > > > > > > > > in
> >> > > > > > > > > > > > bigtop.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Please share your opinion.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > Chinmay.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> Kolhatkar <
> >> > > > > > > > > > > > chinmay@datatorrent.com>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Yes.. Starting to work on the packaging.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > I've already started discussion on bigtop dev
> >> mailing
> >> > > > list
> >> > > > > > for
> >> > > > > > > > > > > > > integration. Also created a Jira for the same.
> For
> >> > this
> >> > > > > > > > communities
> >> > > > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> >> > > > > > > > > thomas@datatorrent.com
> >> > > > > > > > > > >
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >> Chinmay,
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> I don't see anything under prerequisites that
> >> would
> >> > > be a
> >> > > > > > > > problem.
> >> > > > > > > > > We
> >> > > > > > > > > > > > >> looked
> >> > > > > > > > > > > > >> at the ASF licencing compatibility as part of
> >> > becoming
> >> > > > an
> >> > > > > > > > > incubator
> >> > > > > > > > > > > > >> project.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> Please focus on the packaging during the next
> >> weeks.
> >> > > > Since
> >> > > > > > the
> >> > > > > > > > > work
> >> > > > > > > > > > > will
> >> > > > > > > > > > > > >> be
> >> > > > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> >> should
> >> > > > also
> >> > > > > be
> >> > > > > > > > > there.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> Would be good to have the packaging in place by
> >> end
> >> > > Feb.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> Thanks,
> >> > > > > > > > > > > > >> Thomas
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> >> Kolhatkar <
> >> > > > > > > > > > > > >> chinmay@datatorrent.com>
> >> > > > > > > > > > > > >> wrote:
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> > Hi All,
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > We're planning a work on adding Apache Apex
> as
> >> a
> >> > > > > component
> >> > > > > > > to
> >> > > > > > > > > > Apache
> >> > > > > > > > > > > > >> > Bigtop.
> >> > > > > > > > > > > > >> > Bigtop is the packaging system for the Apache
> >> big
> >> > > data
> >> > > > > > > > > ecosystem.
> >> > > > > > > > > > > > >> Several
> >> > > > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE
> for
> >> the
> >> > > > same:
> >> > > > > > > > > > > > >> >
> >> > https://issues.apache.org/jira/browse/APEXCORE-331
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Proposed plan of execution is as follows:
> >> > > > > > > > > > > > >> > *Step 1) Handle prerequisites*
> >> > > > > > > > > > > > >> > Apache bigtop has some hard and soft
> >> expectation
> >> > for
> >> > > > new
> >> > > > > > > > > > components
> >> > > > > > > > > > > to
> >> > > > > > > > > > > > >> get
> >> > > > > > > > > > > > >> > integrated into Bigtop.
> >> > > > > > > > > > > > >> > Here is the list of it:
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Most of them seems to be standard ASF based
> >> > > > > requirements,
> >> > > > > > > but
> >> > > > > > > > > few
> >> > > > > > > > > > > need
> >> > > > > > > > > > > > >> to
> >> > > > > > > > > > > > >> > be checked for:
> >> > > > > > > > > > > > >> > 1. Software projects are expected to be
> >> Licensed
> >> > > under
> >> > > > > > > Apache
> >> > > > > > > > > > > License,
> >> > > > > > > > > > > > >> > Version 2.0 (and their dependencies are
> >> expected
> >> > to
> >> > > be
> >> > > > > > > > > compatible
> >> > > > > > > > > > > with
> >> > > > > > > > > > > > >> this
> >> > > > > > > > > > > > >> > license)
> >> > > > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check
> >> if
> >> > > > > > > dependencies
> >> > > > > > > > of
> >> > > > > > > > > > > Apex
> >> > > > > > > > > > > > >> are
> >> > > > > > > > > > > > >> > compatible with ASL 2.0. This I guess would
> be
> >> a
> >> > > > > > > verification
> >> > > > > > > > > > check.
> >> > > > > > > > > > > > >> > 2. Software projects are expected to be
> >> compatible
> >> > > > with
> >> > > > > > all
> >> > > > > > > of
> >> > > > > > > > > the
> >> > > > > > > > > > > > >> > supported platforms that Bigtop distribution
> is
> >> > > > > targeting
> >> > > > > > > > > > > > >> >     - This needs verifying whether our
> software
> >> > runs
> >> > > > > fine
> >> > > > > > in
> >> > > > > > > > > > > centos-6
> >> > > > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> >> > > > opensuse-13.2.
> >> > > > > > > > > > > > >> > 3. What smoke tests that should be added for
> >> > > > deployment.
> >> > > > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> >> > beyond
> >> > > > > smoke
> >> > > > > > > test
> >> > > > > > > > > > > > >> >     - These are basically the integration
> tests
> >> > for
> >> > > > > > > > verification
> >> > > > > > > > > > > after
> >> > > > > > > > > > > > >> the
> >> > > > > > > > > > > > >> > deployment. This is a soft requirement, but
> >> aim is
> >> > > to
> >> > > > > > > achieve
> >> > > > > > > > > this
> >> > > > > > > > > > > as
> >> > > > > > > > > > > > >> well
> >> > > > > > > > > > > > >> > or at least have explanation why not to
> >> include.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > If there are any from the link which
> explicitly
> >> > > needs
> >> > > > to
> >> > > > > > be
> >> > > > > > > > > > checked
> >> > > > > > > > > > > > >> other
> >> > > > > > > > > > > > >> > than above 4, please let us know.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> >> > > > > > > > > > > > >> > From one of the mail archive of Bigtop, it
> was
> >> > > learnt
> >> > > > > that
> >> > > > > > > the
> >> > > > > > > > > > > bigtop
> >> > > > > > > > > > > > >> > community want to see the addition of new
> >> > components
> >> > > > in
> >> > > > > > > > phases.
> >> > > > > > > > > > Here
> >> > > > > > > > > > > > are
> >> > > > > > > > > > > > >> > the phases:
> >> > > > > > > > > > > > >> > 1. Packaging
> >> > > > > > > > > > > > >> >     - This needs creating of package i.e.
> rpm &
> >> > deb
> >> > > > > files.
> >> > > > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> >> DISCLAMER,
> >> > > > NOTES
> >> > > > > > etc
> >> > > > > > > if
> >> > > > > > > > > any
> >> > > > > > > > > > > > >> needed.
> >> > > > > > > > > > > > >> >     - Any documentation that need to be added
> >> to
> >> > > > > > > distribution
> >> > > > > > > > of
> >> > > > > > > > > > our
> >> > > > > > > > > > > > >> > software.
> >> > > > > > > > > > > > >> >     - Any license information of dependencies
> >> > > required
> >> > > > > to
> >> > > > > > be
> >> > > > > > > > > added
> >> > > > > > > > > > > to
> >> > > > > > > > > > > > >> > package
> >> > > > > > > > > > > > >> > 2. Smoke tests (at very least)
> >> > > > > > > > > > > > >> >     - Adding smoke test for packaging.
> >> > > > > > > > > > > > >> > 3. Puppet recipes for automatic deployment
> and
> >> > > > > > configuration
> >> > > > > > > > > > > > >> >     - Add puppet recipes for automatic
> >> deployment
> >> > > > > > > > > > > > >> > 4. Integration tests
> >> > > > > > > > > > > > >> >     - For verification of deployments.
> >> > > > > > > > > > > > >> > 5. license clearance:
> >> > > > > > > > > > > > >> >     Run 'gradle rat' to make sure all new
> >> stuff is
> >> > > > > > compliant
> >> > > > > > > > > with
> >> > > > > > > > > > > ASF
> >> > > > > > > > > > > > >> > license requirements. If you add code
> licenses
> >> > under
> >> > > > > > > different
> >> > > > > > > > > > > > licenses,
> >> > > > > > > > > > > > >> > those would need to be listed in the NOTICE.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Please share your thoughts on the approach.
> >> > > > > > > > > > > > >> > We'll start corresponding communication on
> >> bigtop
> >> > > > > mailing
> >> > > > > > > list
> >> > > > > > > > > as
> >> > > > > > > > > > > > well.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > We have some specific questions/suggestions
> >> > related
> >> > > to
> >> > > > > > what
> >> > > > > > > > > should
> >> > > > > > > > > > > be
> >> > > > > > > > > > > > >> the
> >> > > > > > > > > > > > >> > content of the package and what should be the
> >> > smoke
> >> > > > > tests,
> >> > > > > > > but
> >> > > > > > > > > in
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > >> > interest of not having too much content here,
> >> > we'll
> >> > > > put
> >> > > > > > the
> >> > > > > > > > > > > questions
> >> > > > > > > > > > > > >> as a
> >> > > > > > > > > > > > >> > separate mail in this mailthread.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Thanks,
> >> > > > > > > > > > > > >> > Chinmay.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

What is the use case for a tarball? Has anyone expressed interest for it?
IMO this should be demand driven.


On Wed, Feb 10, 2016 at 10:32 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:

> Hi Everyone,
>
> I just wanted to mention a learning while packing apex for bigtop
> (Something we can think about in future).
> We should probably have a mvn profile at top level pom which can called
> optionally be invoked and which will package the apex and all its
> dependency in a single tarball. This tarball can then just be extracted and
> one is readily use apex.
>
> I don't know whether this is inline with ASF guidelines, but this I think
> this can be a nice addition.
> Something to think about in future.
>
> Thanks,
> Chinmay.
>
>
> On Wed, Feb 10, 2016 at 10:36 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com
> > wrote:
>
> > Ok... Then make sense to use skeleton.
> > Let's wait for rest of the community as well to provide their opinion.
> >
> > - Chinmay.
> > On 10 Feb 2016 22:34, "Aniruddha Thombare" <an...@datatorrent.com>
> > wrote:
> >
> >> AFI recall...
> >>
> >> Clirc
> >> Clirc history
> >> Etc
> >>
> >> Default site.xml and env can be kept too.
> >>
> >>
> >>
> >>
> >> Sent from handheld
> >>
> >> On Wed, 10 Feb 2016 10:25 pm Chinmay Kolhatkar <chinmay@datatorrent.com
> >
> >> wrote:
> >>
> >> > Ok. Is there any content that we should add in .dt ?
> >> > On 10 Feb 2016 22:10, "Aniruddha Thombare" <aniruddha@datatorrent.com
> >
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > .dt directory.
> >> > > In future, it may  become .apex
> >> > > On 10 Feb 2016 9:53 pm, "Chinmay Kolhatkar" <
> chinmay@datatorrent.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Aniruddha,
> >> > > >
> >> > > > I like the idea of skeleton directory. But I don't see any use of
> >> it,
> >> > > > unless we've anything in default.
> >> > > > If you have something in your mind which should be default, can
> you
> >> > > please
> >> > > > share it?
> >> > > >
> >> > > > Thanks,
> >> > > > Chinmay.
> >> > > >
> >> > > >
> >> > > > On Wed, Feb 10, 2016 at 9:49 PM, Aniruddha Thombare <
> >> > > > aniruddha@datatorrent.com> wrote:
> >> > > >
> >> > > > > On point 2:
> >> > > > > Skeleton directory is for default home dir structure for future
> >> > users.
> >> > > > > By avoiding that, we can't inconvenience users and admins.
> >> > > > > Defaults in that configuration can be specified or evaluated at
> >> > > install.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <
> >> > chinmay@datatorrent.com
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Really good points Pradeep and Aniruddha.
> >> > > > > >
> >> > > > > > 1. I believe we won't need to change the dtcli considering it
> >> works
> >> > > > with
> >> > > > > DT
> >> > > > > > Community edition. We can keep the directory structure similar
> >> to
> >> > > that.
> >> > > > > > dt-env.sh has variables which contains information required
> for
> >> > dtcli
> >> > > > to
> >> > > > > > launch.
> >> > > > > >
> >> > > > > > 2. Let me check with bigtop community that whether they
> >> facilitate
> >> > > the
> >> > > > > > installation of rpms/debs before user is created. In either
> >> case,
> >> > > > current
> >> > > > > > dtcli creates a .dt folder in home directory. Also before
> >> putting
> >> > > > > anything
> >> > > > > > in /etc/skel we need to define what are the default contents
> >> that
> >> > > > should
> >> > > > > go
> >> > > > > > to ~/.dt/ folder. If there is no defaults, probably we should
> >> not
> >> > > > > > explicitly add it in /etc/skel.
> >> > > > > >
> >> > > > > > 3.  /etc/profile.d approach looks nice. This way contents of
> >> > > dt-env.sh
> >> > > > > are
> >> > > > > > present as env variables. But I see a catch there. Adding
> >> dt-env.sh
> >> > > to
> >> > > > > > /etc/profile.d would make all the variables available at
> runtime
> >> > all
> >> > > > the
> >> > > > > > time. I feel a little skeptical about that. Maybe a possible
> >> > > collision
> >> > > > > can
> >> > > > > > occur with other application vars.
> >> > > > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> >> > > > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> >> > > > > >
> >> > > > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> >> > > > > Jira/repository.
> >> > > > > > Packaging related code usually goes there. (That's what all
> the
> >> > > > > components
> >> > > > > > in bigtop does).
> >> > > > > > Having said that, once a PR is created, I'll be sharing the
> >> link of
> >> > > PR
> >> > > > > > here, so that apex community as well can review it.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Chinmay.
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> >> > > > > > aniruddha@datatorrent.com> wrote:
> >> > > > > >
> >> > > > > > > +1 on suggestions and approach.
> >> > > > > > > We may need to iron out details about exact paths etc.
> >> > > > > > > Which can be done on jira / PR comments.
> >> > > > > > > Is that right @dev?
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> >> > > > apache@pradeepdalvi.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Inline comments...
> >> > > > > > > >
> >> > > > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> >> > > > > > > > chinmay@datatorrent.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > @Thomas, Not all the jar present in DT community edition
> >> will
> >> > > be
> >> > > > > > > included
> >> > > > > > > > > there.
> >> > > > > > > > >
> >> > > > > > > > > Here is the list of jars I found from common
> dependencies
> >> > > between
> >> > > > > of
> >> > > > > > > apex
> >> > > > > > > > > and DT Community edition:
> >> > > > > > > > > netlet-1.2.0.jar
> >> > > > > > > > > kryo-2.24.0.jar
> >> > > > > > > > > jackson-core-asl-1.9.2.jar
> >> > > > > > > > > jackson-mapper-asl-1.9.2.jar
> >> > > > > > > > > async-http-client-1.7.20.jar
> >> > > > > > > > > netty-3.6.6.Final.jar
> >> > > > > > > > > validation-api-1.1.0.Final.jar
> >> > > > > > > > > bval-jsr303-0.5.jar
> >> > > > > > > > > bval-core-0.5.jar
> >> > > > > > > > > commons-lang3-3.1.jar
> >> > > > > > > > > commons-beanutils-1.8.3.jar
> >> > > > > > > > > httpclient-4.3.5.jar
> >> > > > > > > > > commons-codec-1.10.jar
> >> > > > > > > > > zip4j-1.3.2.jar
> >> > > > > > > > > jetty-websocket-8.1.10.v20130312.jar
> >> > > > > > > > > xbean-asm5-shaded-4.3.jar
> >> > > > > > > > > jersey-apache-client4-1.9.jar
> >> > > > > > > > > jline-2.11.jar
> >> > > > > > > > > ant-1.9.2.jar
> >> > > > > > > > > ant-launcher-1.9.2.jar
> >> > > > > > > > > mbassador-1.1.9.jar
> >> > > > > > > > > jackson-jaxrs-1.9.2.jar
> >> > > > > > > > > jackson-xc-1.9.2.jar
> >> > > > > > > > > hadoop-common-2.2.0-tests.jar
> >> > > > > > > > >
> >> > > > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> >> > works
> >> > > > > > properly
> >> > > > > > > > for
> >> > > > > > > > > launch+shutdown+kill of apps with only these libraries
> >> > present
> >> > > in
> >> > > > > > > > isolation
> >> > > > > > > > > without dependency on local m2.
> >> > > > > > > > > I'm believe that there are unwanted jars which are used
> >> for
> >> > > > compile
> >> > > > > > > time
> >> > > > > > > > > dependency and not runtime in above list which I can
> drop
> >> to
> >> > > keep
> >> > > > > > > package
> >> > > > > > > > > size minimal.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > @Anniruddha,
> >> > > > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> >> > > required.
> >> > > > > > Only
> >> > > > > > > > > required values I see in it are classpath. and dtcli is
> >> > capable
> >> > > > of
> >> > > > > > > > building
> >> > > > > > > > > it on the fly.
> >> > > > > > > > >
> >> > > > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can
> >> find
> >> > the
> >> > > > jar
> >> > > > > > > path,
> >> > > > > > > > > dt-env.sh and all such conf files. If we need a change
> in
> >> > > dtcli,
> >> > > > > then
> >> > > > > > > > > community's opinion is required for whether dtcli should
> >> > change
> >> > > > in
> >> > > > > > our
> >> > > > > > > > repo
> >> > > > > > > > > or a copy of that with required changes exist in bigtop
> >> repo
> >> > > > until
> >> > > > > we
> >> > > > > > > > make
> >> > > > > > > > > the dtcli generic enough.
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Can we have these file paths set using environment
> variables
> >> > with
> >> > > > > some
> >> > > > > > > > default values in dtcli?
> >> > > > > > > > Then we can set such params in dt-env.sh.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > > 3. dt-sited.xml -> How is this file different from
> >> > dt-site.xml?
> >> > > > Not
> >> > > > > > > sure
> >> > > > > > > > if
> >> > > > > > > > > adding in /etc/skel would help. Files & Dirs from
> >> /etc/skel
> >> > are
> >> > > > > > copied
> >> > > > > > > to
> >> > > > > > > > > home of new user when useradd program is called. But for
> >> > > existing
> >> > > > > > users
> >> > > > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/
> >> on
> >> > the
> >> > > > fly
> >> > > > > > if
> >> > > > > > > > not
> >> > > > > > > > > encountered for the first time. Again correct me if I'm
> >> > wrong.
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Yes, you are right. However this would probably be
> necessary
> >> > step
> >> > > > for
> >> > > > > > > > rpm/deb, as they may also get installed during OS install
> >> and
> >> > > > before
> >> > > > > > user
> >> > > > > > > > accounts were created.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its
> the
> >> > best
> >> > > > > idea.
> >> > > > > > > > We'll
> >> > > > > > > > > anyway put dtcli in location which is by default present
> >> in
> >> > > path
> >> > > > > i.e.
> >> > > > > > > > > /usr/bin.
> >> > > > > > > >
> >> > > > > > > > For other env variables specific to apex, I'll prefer to
> use
> >> > > > > dt-env.sh
> >> > > > > > > and
> >> > > > > > > > > source it in dtcli rather than changing bashrc etc...
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> >> > cleaner
> >> > > > way
> >> > > > > > to
> >> > > > > > > > achieve the same.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > > Chinmay.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> >> > > > > > > > > aniruddha@datatorrent.com> wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi,
> >> > > > > > > > > >
> >> > > > > > > > > > @Chinmay,
> >> > > > > > > > > >
> >> > > > > > > > > > We need to consider following:
> >> > > > > > > > > > System wide default config files can be located at
> >> > following
> >> > > > > > > locations:
> >> > > > > > > > > >
> >> > > > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> >> behaviour
> >> > > on
> >> > > > > how
> >> > > > > > it
> >> > > > > > > > > finds
> >> > > > > > > > > > those locations)
> >> > > > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> >> > behaviour
> >> > > on
> >> > > > > how
> >> > > > > > > it
> >> > > > > > > > > > finds those locations)
> >> > > > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new
> >> users
> >> > > that
> >> > > > > will
> >> > > > > > > be
> >> > > > > > > > > > created in system in future.)
> >> > > > > > > > > >
> >> > > > > > > > > > We may also have to modify bashrc / bashprofile for
> >> > > population
> >> > > > > any
> >> > > > > > > > > > variables that are required.
> >> > > > > > > > > >
> >> > > > > > > > > > @dev, please put in your comments / suggestion.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Aniruddha
> >> > > > > > > > > >
> >> > > > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> >> > > > > > > thomas@datatorrent.com>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > -->
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> >> > > > > > > > > > chinmay@datatorrent.com
> >> > > > > > > > > > > >
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Hello Everyone!!
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Continuing with packaging effort (rpm+deb) of
> apex,
> >> > here
> >> > > > are
> >> > > > > > some
> >> > > > > > > > > > > proposals
> >> > > > > > > > > > > > about package structure etc..
> >> > > > > > > > > > > > Before posting it on bbigtop mailing list, I have
> >> some
> >> > > > > question
> >> > > > > > > for
> >> > > > > > > > > > apex
> >> > > > > > > > > > > > community.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Proposed Directory structure of apex package for
> >> both
> >> > > deb &
> >> > > > > > rpm:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > /usr/lib/apex/bin/dtcli
> >> > > > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> >> > > > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> >> > > > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >> > > > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> >> > > > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> >> > > > > > > > > > >
> >> > > > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> >> > > > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> >> > > > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> >> > > > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > > > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> >> > > > > > > > > > > > /usr/share/doc/apex/NOTICE
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > <version> = 3.3.0-incubating.
> >> > > > > > > > > > > > <other dependent jars> = All the 3rd party jars
> >> which
> >> > are
> >> > > > > > > required
> >> > > > > > > > > for
> >> > > > > > > > > > > apex
> >> > > > > > > > > > > > to run. Usually the dependencies are packaged as
> >> part
> >> > of
> >> > > > > > rpm/deb
> >> > > > > > > by
> >> > > > > > > > > any
> >> > > > > > > > > > > > software in bigtop.
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > Can you specify what those jars are in Bigtop
> context?
> >> > Same
> >> > > > as
> >> > > > > > > > shipped
> >> > > > > > > > > > with
> >> > > > > > > > > > > DT community addition under lib/ ?
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > > <package LICENSE> = Licenses of corresponding of
> 3rd
> >> > > party
> >> > > > > jars
> >> > > > > > > > which
> >> > > > > > > > > > > needs
> >> > > > > > > > > > > > to included while packaging.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Questions related to this:
> >> > > > > > > > > > > > 1. Should we call the cli of apex as "apex"
> instead
> >> of
> >> > > > > "dtcli"
> >> > > > > > in
> >> > > > > > > > > > bigtop
> >> > > > > > > > > > > > package?
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > I think we should keep the name until we are able to
> >> > change
> >> > > > it
> >> > > > > in
> >> > > > > > > > Apex.
> >> > > > > > > > > > > Otherwise this may get confusing.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > > 2. I see that all softwares in bigtop have man
> page
> >> for
> >> > > > their
> >> > > > > > > > > > > executables.
> >> > > > > > > > > > > > I think we should have it too for dtcli. Is there
> >> any
> >> > > > > > > documentation
> >> > > > > > > > > > > which I
> >> > > > > > > > > > > > can convert to man page? or can I use output of
> >> "dtcli
> >> > > > > --help"?
> >> > > > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> >> > 3.3.0
> >> > > OR
> >> > > > > > > > > > > > 3.3.0-incubating?
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > Has to be -incubating
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > > 4. Is it ok for apex package of bigtop to depend
> on
> >> > 2.7.1
> >> > > > > > > version
> >> > > > > > > > of
> >> > > > > > > > > > > > bigtop hadoop? Any problems that we see with this
> >> > > > dependency?
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > Should work without changed. I thought we certified
> >> > against
> >> > > > > > 2.7.0?
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > > 5. Following is the apache mirror from which
> bigtop
> >> > will
> >> > > > pick
> >> > > > > > the
> >> > > > > > > > > apex
> >> > > > > > > > > > > > source for compilation and packaging. Please
> >> correct if
> >> > > > > > > incorrect:
> >> > > > > > > > > > > >
> >> > > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >> > > > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
> >> all
> >> > > the
> >> > > > > > > > softwares
> >> > > > > > > > > in
> >> > > > > > > > > > > > bigtop.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Please share your opinion.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > Chinmay.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> Kolhatkar <
> >> > > > > > > > > > > > chinmay@datatorrent.com>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Yes.. Starting to work on the packaging.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > I've already started discussion on bigtop dev
> >> mailing
> >> > > > list
> >> > > > > > for
> >> > > > > > > > > > > > > integration. Also created a Jira for the same.
> For
> >> > this
> >> > > > > > > > communities
> >> > > > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> >> > > > > > > > > thomas@datatorrent.com
> >> > > > > > > > > > >
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >> Chinmay,
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> I don't see anything under prerequisites that
> >> would
> >> > > be a
> >> > > > > > > > problem.
> >> > > > > > > > > We
> >> > > > > > > > > > > > >> looked
> >> > > > > > > > > > > > >> at the ASF licencing compatibility as part of
> >> > becoming
> >> > > > an
> >> > > > > > > > > incubator
> >> > > > > > > > > > > > >> project.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> Please focus on the packaging during the next
> >> weeks.
> >> > > > Since
> >> > > > > > the
> >> > > > > > > > > work
> >> > > > > > > > > > > will
> >> > > > > > > > > > > > >> be
> >> > > > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> >> should
> >> > > > also
> >> > > > > be
> >> > > > > > > > > there.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> Would be good to have the packaging in place by
> >> end
> >> > > Feb.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> Thanks,
> >> > > > > > > > > > > > >> Thomas
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> >> Kolhatkar <
> >> > > > > > > > > > > > >> chinmay@datatorrent.com>
> >> > > > > > > > > > > > >> wrote:
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> > Hi All,
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > We're planning a work on adding Apache Apex
> as
> >> a
> >> > > > > component
> >> > > > > > > to
> >> > > > > > > > > > Apache
> >> > > > > > > > > > > > >> > Bigtop.
> >> > > > > > > > > > > > >> > Bigtop is the packaging system for the Apache
> >> big
> >> > > data
> >> > > > > > > > > ecosystem.
> >> > > > > > > > > > > > >> Several
> >> > > > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE
> for
> >> the
> >> > > > same:
> >> > > > > > > > > > > > >> >
> >> > https://issues.apache.org/jira/browse/APEXCORE-331
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Proposed plan of execution is as follows:
> >> > > > > > > > > > > > >> > *Step 1) Handle prerequisites*
> >> > > > > > > > > > > > >> > Apache bigtop has some hard and soft
> >> expectation
> >> > for
> >> > > > new
> >> > > > > > > > > > components
> >> > > > > > > > > > > to
> >> > > > > > > > > > > > >> get
> >> > > > > > > > > > > > >> > integrated into Bigtop.
> >> > > > > > > > > > > > >> > Here is the list of it:
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Most of them seems to be standard ASF based
> >> > > > > requirements,
> >> > > > > > > but
> >> > > > > > > > > few
> >> > > > > > > > > > > need
> >> > > > > > > > > > > > >> to
> >> > > > > > > > > > > > >> > be checked for:
> >> > > > > > > > > > > > >> > 1. Software projects are expected to be
> >> Licensed
> >> > > under
> >> > > > > > > Apache
> >> > > > > > > > > > > License,
> >> > > > > > > > > > > > >> > Version 2.0 (and their dependencies are
> >> expected
> >> > to
> >> > > be
> >> > > > > > > > > compatible
> >> > > > > > > > > > > with
> >> > > > > > > > > > > > >> this
> >> > > > > > > > > > > > >> > license)
> >> > > > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check
> >> if
> >> > > > > > > dependencies
> >> > > > > > > > of
> >> > > > > > > > > > > Apex
> >> > > > > > > > > > > > >> are
> >> > > > > > > > > > > > >> > compatible with ASL 2.0. This I guess would
> be
> >> a
> >> > > > > > > verification
> >> > > > > > > > > > check.
> >> > > > > > > > > > > > >> > 2. Software projects are expected to be
> >> compatible
> >> > > > with
> >> > > > > > all
> >> > > > > > > of
> >> > > > > > > > > the
> >> > > > > > > > > > > > >> > supported platforms that Bigtop distribution
> is
> >> > > > > targeting
> >> > > > > > > > > > > > >> >     - This needs verifying whether our
> software
> >> > runs
> >> > > > > fine
> >> > > > > > in
> >> > > > > > > > > > > centos-6
> >> > > > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> >> > > > opensuse-13.2.
> >> > > > > > > > > > > > >> > 3. What smoke tests that should be added for
> >> > > > deployment.
> >> > > > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> >> > beyond
> >> > > > > smoke
> >> > > > > > > test
> >> > > > > > > > > > > > >> >     - These are basically the integration
> tests
> >> > for
> >> > > > > > > > verification
> >> > > > > > > > > > > after
> >> > > > > > > > > > > > >> the
> >> > > > > > > > > > > > >> > deployment. This is a soft requirement, but
> >> aim is
> >> > > to
> >> > > > > > > achieve
> >> > > > > > > > > this
> >> > > > > > > > > > > as
> >> > > > > > > > > > > > >> well
> >> > > > > > > > > > > > >> > or at least have explanation why not to
> >> include.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > If there are any from the link which
> explicitly
> >> > > needs
> >> > > > to
> >> > > > > > be
> >> > > > > > > > > > checked
> >> > > > > > > > > > > > >> other
> >> > > > > > > > > > > > >> > than above 4, please let us know.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> >> > > > > > > > > > > > >> > From one of the mail archive of Bigtop, it
> was
> >> > > learnt
> >> > > > > that
> >> > > > > > > the
> >> > > > > > > > > > > bigtop
> >> > > > > > > > > > > > >> > community want to see the addition of new
> >> > components
> >> > > > in
> >> > > > > > > > phases.
> >> > > > > > > > > > Here
> >> > > > > > > > > > > > are
> >> > > > > > > > > > > > >> > the phases:
> >> > > > > > > > > > > > >> > 1. Packaging
> >> > > > > > > > > > > > >> >     - This needs creating of package i.e.
> rpm &
> >> > deb
> >> > > > > files.
> >> > > > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> >> DISCLAMER,
> >> > > > NOTES
> >> > > > > > etc
> >> > > > > > > if
> >> > > > > > > > > any
> >> > > > > > > > > > > > >> needed.
> >> > > > > > > > > > > > >> >     - Any documentation that need to be added
> >> to
> >> > > > > > > distribution
> >> > > > > > > > of
> >> > > > > > > > > > our
> >> > > > > > > > > > > > >> > software.
> >> > > > > > > > > > > > >> >     - Any license information of dependencies
> >> > > required
> >> > > > > to
> >> > > > > > be
> >> > > > > > > > > added
> >> > > > > > > > > > > to
> >> > > > > > > > > > > > >> > package
> >> > > > > > > > > > > > >> > 2. Smoke tests (at very least)
> >> > > > > > > > > > > > >> >     - Adding smoke test for packaging.
> >> > > > > > > > > > > > >> > 3. Puppet recipes for automatic deployment
> and
> >> > > > > > configuration
> >> > > > > > > > > > > > >> >     - Add puppet recipes for automatic
> >> deployment
> >> > > > > > > > > > > > >> > 4. Integration tests
> >> > > > > > > > > > > > >> >     - For verification of deployments.
> >> > > > > > > > > > > > >> > 5. license clearance:
> >> > > > > > > > > > > > >> >     Run 'gradle rat' to make sure all new
> >> stuff is
> >> > > > > > compliant
> >> > > > > > > > > with
> >> > > > > > > > > > > ASF
> >> > > > > > > > > > > > >> > license requirements. If you add code
> licenses
> >> > under
> >> > > > > > > different
> >> > > > > > > > > > > > licenses,
> >> > > > > > > > > > > > >> > those would need to be listed in the NOTICE.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Please share your thoughts on the approach.
> >> > > > > > > > > > > > >> > We'll start corresponding communication on
> >> bigtop
> >> > > > > mailing
> >> > > > > > > list
> >> > > > > > > > > as
> >> > > > > > > > > > > > well.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > We have some specific questions/suggestions
> >> > related
> >> > > to
> >> > > > > > what
> >> > > > > > > > > should
> >> > > > > > > > > > > be
> >> > > > > > > > > > > > >> the
> >> > > > > > > > > > > > >> > content of the package and what should be the
> >> > smoke
> >> > > > > tests,
> >> > > > > > > but
> >> > > > > > > > > in
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > >> > interest of not having too much content here,
> >> > we'll
> >> > > > put
> >> > > > > > the
> >> > > > > > > > > > > questions
> >> > > > > > > > > > > > >> as a
> >> > > > > > > > > > > > >> > separate mail in this mailthread.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Thanks,
> >> > > > > > > > > > > > >> > Chinmay.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@datatorrent.com>]

Hi Everyone,

I just wanted to mention a learning while packing apex for bigtop
(Something we can think about in future).
We should probably have a mvn profile at top level pom which can called
optionally be invoked and which will package the apex and all its
dependency in a single tarball. This tarball can then just be extracted and
one is readily use apex.

I don't know whether this is inline with ASF guidelines, but this I think
this can be a nice addition.
Something to think about in future.

Thanks,
Chinmay.


On Wed, Feb 10, 2016 at 10:36 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:

> Ok... Then make sense to use skeleton.
> Let's wait for rest of the community as well to provide their opinion.
>
> - Chinmay.
> On 10 Feb 2016 22:34, "Aniruddha Thombare" <an...@datatorrent.com>
> wrote:
>
>> AFI recall...
>>
>> Clirc
>> Clirc history
>> Etc
>>
>> Default site.xml and env can be kept too.
>>
>>
>>
>>
>> Sent from handheld
>>
>> On Wed, 10 Feb 2016 10:25 pm Chinmay Kolhatkar <ch...@datatorrent.com>
>> wrote:
>>
>> > Ok. Is there any content that we should add in .dt ?
>> > On 10 Feb 2016 22:10, "Aniruddha Thombare" <an...@datatorrent.com>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > .dt directory.
>> > > In future, it may  become .apex
>> > > On 10 Feb 2016 9:53 pm, "Chinmay Kolhatkar" <ch...@datatorrent.com>
>> > > wrote:
>> > >
>> > > > Hi Aniruddha,
>> > > >
>> > > > I like the idea of skeleton directory. But I don't see any use of
>> it,
>> > > > unless we've anything in default.
>> > > > If you have something in your mind which should be default, can you
>> > > please
>> > > > share it?
>> > > >
>> > > > Thanks,
>> > > > Chinmay.
>> > > >
>> > > >
>> > > > On Wed, Feb 10, 2016 at 9:49 PM, Aniruddha Thombare <
>> > > > aniruddha@datatorrent.com> wrote:
>> > > >
>> > > > > On point 2:
>> > > > > Skeleton directory is for default home dir structure for future
>> > users.
>> > > > > By avoiding that, we can't inconvenience users and admins.
>> > > > > Defaults in that configuration can be specified or evaluated at
>> > > install.
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <
>> > chinmay@datatorrent.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Really good points Pradeep and Aniruddha.
>> > > > > >
>> > > > > > 1. I believe we won't need to change the dtcli considering it
>> works
>> > > > with
>> > > > > DT
>> > > > > > Community edition. We can keep the directory structure similar
>> to
>> > > that.
>> > > > > > dt-env.sh has variables which contains information required for
>> > dtcli
>> > > > to
>> > > > > > launch.
>> > > > > >
>> > > > > > 2. Let me check with bigtop community that whether they
>> facilitate
>> > > the
>> > > > > > installation of rpms/debs before user is created. In either
>> case,
>> > > > current
>> > > > > > dtcli creates a .dt folder in home directory. Also before
>> putting
>> > > > > anything
>> > > > > > in /etc/skel we need to define what are the default contents
>> that
>> > > > should
>> > > > > go
>> > > > > > to ~/.dt/ folder. If there is no defaults, probably we should
>> not
>> > > > > > explicitly add it in /etc/skel.
>> > > > > >
>> > > > > > 3.  /etc/profile.d approach looks nice. This way contents of
>> > > dt-env.sh
>> > > > > are
>> > > > > > present as env variables. But I see a catch there. Adding
>> dt-env.sh
>> > > to
>> > > > > > /etc/profile.d would make all the variables available at runtime
>> > all
>> > > > the
>> > > > > > time. I feel a little skeptical about that. Maybe a possible
>> > > collision
>> > > > > can
>> > > > > > occur with other application vars.
>> > > > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
>> > > > > > "~/.dt/dt-env.sh". So those variables are anyway available.
>> > > > > >
>> > > > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
>> > > > > Jira/repository.
>> > > > > > Packaging related code usually goes there. (That's what all the
>> > > > > components
>> > > > > > in bigtop does).
>> > > > > > Having said that, once a PR is created, I'll be sharing the
>> link of
>> > > PR
>> > > > > > here, so that apex community as well can review it.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Chinmay.
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
>> > > > > > aniruddha@datatorrent.com> wrote:
>> > > > > >
>> > > > > > > +1 on suggestions and approach.
>> > > > > > > We may need to iron out details about exact paths etc.
>> > > > > > > Which can be done on jira / PR comments.
>> > > > > > > Is that right @dev?
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
>> > > > apache@pradeepdalvi.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Inline comments...
>> > > > > > > >
>> > > > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
>> > > > > > > > chinmay@datatorrent.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > @Thomas, Not all the jar present in DT community edition
>> will
>> > > be
>> > > > > > > included
>> > > > > > > > > there.
>> > > > > > > > >
>> > > > > > > > > Here is the list of jars I found from common dependencies
>> > > between
>> > > > > of
>> > > > > > > apex
>> > > > > > > > > and DT Community edition:
>> > > > > > > > > netlet-1.2.0.jar
>> > > > > > > > > kryo-2.24.0.jar
>> > > > > > > > > jackson-core-asl-1.9.2.jar
>> > > > > > > > > jackson-mapper-asl-1.9.2.jar
>> > > > > > > > > async-http-client-1.7.20.jar
>> > > > > > > > > netty-3.6.6.Final.jar
>> > > > > > > > > validation-api-1.1.0.Final.jar
>> > > > > > > > > bval-jsr303-0.5.jar
>> > > > > > > > > bval-core-0.5.jar
>> > > > > > > > > commons-lang3-3.1.jar
>> > > > > > > > > commons-beanutils-1.8.3.jar
>> > > > > > > > > httpclient-4.3.5.jar
>> > > > > > > > > commons-codec-1.10.jar
>> > > > > > > > > zip4j-1.3.2.jar
>> > > > > > > > > jetty-websocket-8.1.10.v20130312.jar
>> > > > > > > > > xbean-asm5-shaded-4.3.jar
>> > > > > > > > > jersey-apache-client4-1.9.jar
>> > > > > > > > > jline-2.11.jar
>> > > > > > > > > ant-1.9.2.jar
>> > > > > > > > > ant-launcher-1.9.2.jar
>> > > > > > > > > mbassador-1.1.9.jar
>> > > > > > > > > jackson-jaxrs-1.9.2.jar
>> > > > > > > > > jackson-xc-1.9.2.jar
>> > > > > > > > > hadoop-common-2.2.0-tests.jar
>> > > > > > > > >
>> > > > > > > > > Ofcourse, I'll be running some tests do check that dtcli
>> > works
>> > > > > > properly
>> > > > > > > > for
>> > > > > > > > > launch+shutdown+kill of apps with only these libraries
>> > present
>> > > in
>> > > > > > > > isolation
>> > > > > > > > > without dependency on local m2.
>> > > > > > > > > I'm believe that there are unwanted jars which are used
>> for
>> > > > compile
>> > > > > > > time
>> > > > > > > > > dependency and not runtime in above list which I can drop
>> to
>> > > keep
>> > > > > > > package
>> > > > > > > > > size minimal.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > @Anniruddha,
>> > > > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
>> > > required.
>> > > > > > Only
>> > > > > > > > > required values I see in it are classpath. and dtcli is
>> > capable
>> > > > of
>> > > > > > > > building
>> > > > > > > > > it on the fly.
>> > > > > > > > >
>> > > > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can
>> find
>> > the
>> > > > jar
>> > > > > > > path,
>> > > > > > > > > dt-env.sh and all such conf files. If we need a change in
>> > > dtcli,
>> > > > > then
>> > > > > > > > > community's opinion is required for whether dtcli should
>> > change
>> > > > in
>> > > > > > our
>> > > > > > > > repo
>> > > > > > > > > or a copy of that with required changes exist in bigtop
>> repo
>> > > > until
>> > > > > we
>> > > > > > > > make
>> > > > > > > > > the dtcli generic enough.
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > > Can we have these file paths set using environment variables
>> > with
>> > > > > some
>> > > > > > > > default values in dtcli?
>> > > > > > > > Then we can set such params in dt-env.sh.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > > 3. dt-sited.xml -> How is this file different from
>> > dt-site.xml?
>> > > > Not
>> > > > > > > sure
>> > > > > > > > if
>> > > > > > > > > adding in /etc/skel would help. Files & Dirs from
>> /etc/skel
>> > are
>> > > > > > copied
>> > > > > > > to
>> > > > > > > > > home of new user when useradd program is called. But for
>> > > existing
>> > > > > > users
>> > > > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/
>> on
>> > the
>> > > > fly
>> > > > > > if
>> > > > > > > > not
>> > > > > > > > > encountered for the first time. Again correct me if I'm
>> > wrong.
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > > Yes, you are right. However this would probably be necessary
>> > step
>> > > > for
>> > > > > > > > rpm/deb, as they may also get installed during OS install
>> and
>> > > > before
>> > > > > > user
>> > > > > > > > accounts were created.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
>> > best
>> > > > > idea.
>> > > > > > > > We'll
>> > > > > > > > > anyway put dtcli in location which is by default present
>> in
>> > > path
>> > > > > i.e.
>> > > > > > > > > /usr/bin.
>> > > > > > > >
>> > > > > > > > For other env variables specific to apex, I'll prefer to use
>> > > > > dt-env.sh
>> > > > > > > and
>> > > > > > > > > source it in dtcli rather than changing bashrc etc...
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
>> > cleaner
>> > > > way
>> > > > > > to
>> > > > > > > > achieve the same.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > > Chinmay.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
>> > > > > > > > > aniruddha@datatorrent.com> wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi,
>> > > > > > > > > >
>> > > > > > > > > > @Chinmay,
>> > > > > > > > > >
>> > > > > > > > > > We need to consider following:
>> > > > > > > > > > System wide default config files can be located at
>> > following
>> > > > > > > locations:
>> > > > > > > > > >
>> > > > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
>> behaviour
>> > > on
>> > > > > how
>> > > > > > it
>> > > > > > > > > finds
>> > > > > > > > > > those locations)
>> > > > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
>> > behaviour
>> > > on
>> > > > > how
>> > > > > > > it
>> > > > > > > > > > finds those locations)
>> > > > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new
>> users
>> > > that
>> > > > > will
>> > > > > > > be
>> > > > > > > > > > created in system in future.)
>> > > > > > > > > >
>> > > > > > > > > > We may also have to modify bashrc / bashprofile for
>> > > population
>> > > > > any
>> > > > > > > > > > variables that are required.
>> > > > > > > > > >
>> > > > > > > > > > @dev, please put in your comments / suggestion.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Aniruddha
>> > > > > > > > > >
>> > > > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
>> > > > > > > thomas@datatorrent.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > -->
>> > > > > > > > > > >
>> > > > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
>> > > > > > > > > > chinmay@datatorrent.com
>> > > > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hello Everyone!!
>> > > > > > > > > > > >
>> > > > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
>> > here
>> > > > are
>> > > > > > some
>> > > > > > > > > > > proposals
>> > > > > > > > > > > > about package structure etc..
>> > > > > > > > > > > > Before posting it on bbigtop mailing list, I have
>> some
>> > > > > question
>> > > > > > > for
>> > > > > > > > > > apex
>> > > > > > > > > > > > community.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Proposed Directory structure of apex package for
>> both
>> > > deb &
>> > > > > > rpm:
>> > > > > > > > > > > >
>> > > > > > > > > > > > /usr/lib/apex/bin/dtcli
>> > > > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
>> > > > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
>> > > > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
>> > > > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
>> > > > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
>> > > > > > > > > > >
>> > > > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
>> > > > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
>> > > > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
>> > > > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
>> > > > > > > > > > > > /usr/share/doc/apex/CHANGELOG
>> > > > > > > > > > > > /usr/share/doc/apex/NOTICE
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > <version> = 3.3.0-incubating.
>> > > > > > > > > > > > <other dependent jars> = All the 3rd party jars
>> which
>> > are
>> > > > > > > required
>> > > > > > > > > for
>> > > > > > > > > > > apex
>> > > > > > > > > > > > to run. Usually the dependencies are packaged as
>> part
>> > of
>> > > > > > rpm/deb
>> > > > > > > by
>> > > > > > > > > any
>> > > > > > > > > > > > software in bigtop.
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > Can you specify what those jars are in Bigtop context?
>> > Same
>> > > > as
>> > > > > > > > shipped
>> > > > > > > > > > with
>> > > > > > > > > > > DT community addition under lib/ ?
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
>> > > party
>> > > > > jars
>> > > > > > > > which
>> > > > > > > > > > > needs
>> > > > > > > > > > > > to included while packaging.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Questions related to this:
>> > > > > > > > > > > > 1. Should we call the cli of apex as "apex" instead
>> of
>> > > > > "dtcli"
>> > > > > > in
>> > > > > > > > > > bigtop
>> > > > > > > > > > > > package?
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > I think we should keep the name until we are able to
>> > change
>> > > > it
>> > > > > in
>> > > > > > > > Apex.
>> > > > > > > > > > > Otherwise this may get confusing.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > > 2. I see that all softwares in bigtop have man page
>> for
>> > > > their
>> > > > > > > > > > > executables.
>> > > > > > > > > > > > I think we should have it too for dtcli. Is there
>> any
>> > > > > > > documentation
>> > > > > > > > > > > which I
>> > > > > > > > > > > > can convert to man page? or can I use output of
>> "dtcli
>> > > > > --help"?
>> > > > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
>> > 3.3.0
>> > > OR
>> > > > > > > > > > > > 3.3.0-incubating?
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > Has to be -incubating
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
>> > 2.7.1
>> > > > > > > version
>> > > > > > > > of
>> > > > > > > > > > > > bigtop hadoop? Any problems that we see with this
>> > > > dependency?
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > Should work without changed. I thought we certified
>> > against
>> > > > > > 2.7.0?
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > > 5. Following is the apache mirror from which bigtop
>> > will
>> > > > pick
>> > > > > > the
>> > > > > > > > > apex
>> > > > > > > > > > > > source for compilation and packaging. Please
>> correct if
>> > > > > > > incorrect:
>> > > > > > > > > > > >
>> > > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>> > > > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
>> all
>> > > the
>> > > > > > > > softwares
>> > > > > > > > > in
>> > > > > > > > > > > > bigtop.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Please share your opinion.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > Chinmay.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
>> > > > > > > > > > > > chinmay@datatorrent.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Yes.. Starting to work on the packaging.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I've already started discussion on bigtop dev
>> mailing
>> > > > list
>> > > > > > for
>> > > > > > > > > > > > > integration. Also created a Jira for the same. For
>> > this
>> > > > > > > > communities
>> > > > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
>> > > > > > > > > thomas@datatorrent.com
>> > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >> Chinmay,
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> I don't see anything under prerequisites that
>> would
>> > > be a
>> > > > > > > > problem.
>> > > > > > > > > We
>> > > > > > > > > > > > >> looked
>> > > > > > > > > > > > >> at the ASF licencing compatibility as part of
>> > becoming
>> > > > an
>> > > > > > > > > incubator
>> > > > > > > > > > > > >> project.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> Please focus on the packaging during the next
>> weeks.
>> > > > Since
>> > > > > > the
>> > > > > > > > > work
>> > > > > > > > > > > will
>> > > > > > > > > > > > >> be
>> > > > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
>> should
>> > > > also
>> > > > > be
>> > > > > > > > > there.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> Would be good to have the packaging in place by
>> end
>> > > Feb.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> Thanks,
>> > > > > > > > > > > > >> Thomas
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
>> Kolhatkar <
>> > > > > > > > > > > > >> chinmay@datatorrent.com>
>> > > > > > > > > > > > >> wrote:
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> > Hi All,
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > We're planning a work on adding Apache Apex as
>> a
>> > > > > component
>> > > > > > > to
>> > > > > > > > > > Apache
>> > > > > > > > > > > > >> > Bigtop.
>> > > > > > > > > > > > >> > Bigtop is the packaging system for the Apache
>> big
>> > > data
>> > > > > > > > > ecosystem.
>> > > > > > > > > > > > >> Several
>> > > > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for
>> the
>> > > > same:
>> > > > > > > > > > > > >> >
>> > https://issues.apache.org/jira/browse/APEXCORE-331
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > Proposed plan of execution is as follows:
>> > > > > > > > > > > > >> > *Step 1) Handle prerequisites*
>> > > > > > > > > > > > >> > Apache bigtop has some hard and soft
>> expectation
>> > for
>> > > > new
>> > > > > > > > > > components
>> > > > > > > > > > > to
>> > > > > > > > > > > > >> get
>> > > > > > > > > > > > >> > integrated into Bigtop.
>> > > > > > > > > > > > >> > Here is the list of it:
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > Most of them seems to be standard ASF based
>> > > > > requirements,
>> > > > > > > but
>> > > > > > > > > few
>> > > > > > > > > > > need
>> > > > > > > > > > > > >> to
>> > > > > > > > > > > > >> > be checked for:
>> > > > > > > > > > > > >> > 1. Software projects are expected to be
>> Licensed
>> > > under
>> > > > > > > Apache
>> > > > > > > > > > > License,
>> > > > > > > > > > > > >> > Version 2.0 (and their dependencies are
>> expected
>> > to
>> > > be
>> > > > > > > > > compatible
>> > > > > > > > > > > with
>> > > > > > > > > > > > >> this
>> > > > > > > > > > > > >> > license)
>> > > > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check
>> if
>> > > > > > > dependencies
>> > > > > > > > of
>> > > > > > > > > > > Apex
>> > > > > > > > > > > > >> are
>> > > > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be
>> a
>> > > > > > > verification
>> > > > > > > > > > check.
>> > > > > > > > > > > > >> > 2. Software projects are expected to be
>> compatible
>> > > > with
>> > > > > > all
>> > > > > > > of
>> > > > > > > > > the
>> > > > > > > > > > > > >> > supported platforms that Bigtop distribution is
>> > > > > targeting
>> > > > > > > > > > > > >> >     - This needs verifying whether our software
>> > runs
>> > > > > fine
>> > > > > > in
>> > > > > > > > > > > centos-6
>> > > > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
>> > > > opensuse-13.2.
>> > > > > > > > > > > > >> > 3. What smoke tests that should be added for
>> > > > deployment.
>> > > > > > > > > > > > >> > 4. Identifying the test artifacts which goes
>> > beyond
>> > > > > smoke
>> > > > > > > test
>> > > > > > > > > > > > >> >     - These are basically the integration tests
>> > for
>> > > > > > > > verification
>> > > > > > > > > > > after
>> > > > > > > > > > > > >> the
>> > > > > > > > > > > > >> > deployment. This is a soft requirement, but
>> aim is
>> > > to
>> > > > > > > achieve
>> > > > > > > > > this
>> > > > > > > > > > > as
>> > > > > > > > > > > > >> well
>> > > > > > > > > > > > >> > or at least have explanation why not to
>> include.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > If there are any from the link which explicitly
>> > > needs
>> > > > to
>> > > > > > be
>> > > > > > > > > > checked
>> > > > > > > > > > > > >> other
>> > > > > > > > > > > > >> > than above 4, please let us know.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
>> > > > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
>> > > learnt
>> > > > > that
>> > > > > > > the
>> > > > > > > > > > > bigtop
>> > > > > > > > > > > > >> > community want to see the addition of new
>> > components
>> > > > in
>> > > > > > > > phases.
>> > > > > > > > > > Here
>> > > > > > > > > > > > are
>> > > > > > > > > > > > >> > the phases:
>> > > > > > > > > > > > >> > 1. Packaging
>> > > > > > > > > > > > >> >     - This needs creating of package i.e. rpm &
>> > deb
>> > > > > files.
>> > > > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
>> DISCLAMER,
>> > > > NOTES
>> > > > > > etc
>> > > > > > > if
>> > > > > > > > > any
>> > > > > > > > > > > > >> needed.
>> > > > > > > > > > > > >> >     - Any documentation that need to be added
>> to
>> > > > > > > distribution
>> > > > > > > > of
>> > > > > > > > > > our
>> > > > > > > > > > > > >> > software.
>> > > > > > > > > > > > >> >     - Any license information of dependencies
>> > > required
>> > > > > to
>> > > > > > be
>> > > > > > > > > added
>> > > > > > > > > > > to
>> > > > > > > > > > > > >> > package
>> > > > > > > > > > > > >> > 2. Smoke tests (at very least)
>> > > > > > > > > > > > >> >     - Adding smoke test for packaging.
>> > > > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
>> > > > > > configuration
>> > > > > > > > > > > > >> >     - Add puppet recipes for automatic
>> deployment
>> > > > > > > > > > > > >> > 4. Integration tests
>> > > > > > > > > > > > >> >     - For verification of deployments.
>> > > > > > > > > > > > >> > 5. license clearance:
>> > > > > > > > > > > > >> >     Run 'gradle rat' to make sure all new
>> stuff is
>> > > > > > compliant
>> > > > > > > > > with
>> > > > > > > > > > > ASF
>> > > > > > > > > > > > >> > license requirements. If you add code licenses
>> > under
>> > > > > > > different
>> > > > > > > > > > > > licenses,
>> > > > > > > > > > > > >> > those would need to be listed in the NOTICE.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > Please share your thoughts on the approach.
>> > > > > > > > > > > > >> > We'll start corresponding communication on
>> bigtop
>> > > > > mailing
>> > > > > > > list
>> > > > > > > > > as
>> > > > > > > > > > > > well.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > We have some specific questions/suggestions
>> > related
>> > > to
>> > > > > > what
>> > > > > > > > > should
>> > > > > > > > > > > be
>> > > > > > > > > > > > >> the
>> > > > > > > > > > > > >> > content of the package and what should be the
>> > smoke
>> > > > > tests,
>> > > > > > > but
>> > > > > > > > > in
>> > > > > > > > > > > the
>> > > > > > > > > > > > >> > interest of not having too much content here,
>> > we'll
>> > > > put
>> > > > > > the
>> > > > > > > > > > > questions
>> > > > > > > > > > > > >> as a
>> > > > > > > > > > > > >> > separate mail in this mailthread.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > Thanks,
>> > > > > > > > > > > > >> > Chinmay.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@datatorrent.com>]

Ok... Then make sense to use skeleton.
Let's wait for rest of the community as well to provide their opinion.

- Chinmay.
On 10 Feb 2016 22:34, "Aniruddha Thombare" <an...@datatorrent.com>
wrote:

> AFI recall...
>
> Clirc
> Clirc history
> Etc
>
> Default site.xml and env can be kept too.
>
>
>
>
> Sent from handheld
>
> On Wed, 10 Feb 2016 10:25 pm Chinmay Kolhatkar <ch...@datatorrent.com>
> wrote:
>
> > Ok. Is there any content that we should add in .dt ?
> > On 10 Feb 2016 22:10, "Aniruddha Thombare" <an...@datatorrent.com>
> > wrote:
> >
> > > Hi,
> > >
> > > .dt directory.
> > > In future, it may  become .apex
> > > On 10 Feb 2016 9:53 pm, "Chinmay Kolhatkar" <ch...@datatorrent.com>
> > > wrote:
> > >
> > > > Hi Aniruddha,
> > > >
> > > > I like the idea of skeleton directory. But I don't see any use of it,
> > > > unless we've anything in default.
> > > > If you have something in your mind which should be default, can you
> > > please
> > > > share it?
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > > On Wed, Feb 10, 2016 at 9:49 PM, Aniruddha Thombare <
> > > > aniruddha@datatorrent.com> wrote:
> > > >
> > > > > On point 2:
> > > > > Skeleton directory is for default home dir structure for future
> > users.
> > > > > By avoiding that, we can't inconvenience users and admins.
> > > > > Defaults in that configuration can be specified or evaluated at
> > > install.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <
> > chinmay@datatorrent.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Really good points Pradeep and Aniruddha.
> > > > > >
> > > > > > 1. I believe we won't need to change the dtcli considering it
> works
> > > > with
> > > > > DT
> > > > > > Community edition. We can keep the directory structure similar to
> > > that.
> > > > > > dt-env.sh has variables which contains information required for
> > dtcli
> > > > to
> > > > > > launch.
> > > > > >
> > > > > > 2. Let me check with bigtop community that whether they
> facilitate
> > > the
> > > > > > installation of rpms/debs before user is created. In either case,
> > > > current
> > > > > > dtcli creates a .dt folder in home directory. Also before putting
> > > > > anything
> > > > > > in /etc/skel we need to define what are the default contents that
> > > > should
> > > > > go
> > > > > > to ~/.dt/ folder. If there is no defaults, probably we should not
> > > > > > explicitly add it in /etc/skel.
> > > > > >
> > > > > > 3.  /etc/profile.d approach looks nice. This way contents of
> > > dt-env.sh
> > > > > are
> > > > > > present as env variables. But I see a catch there. Adding
> dt-env.sh
> > > to
> > > > > > /etc/profile.d would make all the variables available at runtime
> > all
> > > > the
> > > > > > time. I feel a little skeptical about that. Maybe a possible
> > > collision
> > > > > can
> > > > > > occur with other application vars.
> > > > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > > > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > > > > >
> > > > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > > > > Jira/repository.
> > > > > > Packaging related code usually goes there. (That's what all the
> > > > > components
> > > > > > in bigtop does).
> > > > > > Having said that, once a PR is created, I'll be sharing the link
> of
> > > PR
> > > > > > here, so that apex community as well can review it.
> > > > > >
> > > > > > Thanks,
> > > > > > Chinmay.
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > > > > aniruddha@datatorrent.com> wrote:
> > > > > >
> > > > > > > +1 on suggestions and approach.
> > > > > > > We may need to iron out details about exact paths etc.
> > > > > > > Which can be done on jira / PR comments.
> > > > > > > Is that right @dev?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > > apache@pradeepdalvi.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Inline comments...
> > > > > > > >
> > > > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > > > > > chinmay@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > @Thomas, Not all the jar present in DT community edition
> will
> > > be
> > > > > > > included
> > > > > > > > > there.
> > > > > > > > >
> > > > > > > > > Here is the list of jars I found from common dependencies
> > > between
> > > > > of
> > > > > > > apex
> > > > > > > > > and DT Community edition:
> > > > > > > > > netlet-1.2.0.jar
> > > > > > > > > kryo-2.24.0.jar
> > > > > > > > > jackson-core-asl-1.9.2.jar
> > > > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > > > > > async-http-client-1.7.20.jar
> > > > > > > > > netty-3.6.6.Final.jar
> > > > > > > > > validation-api-1.1.0.Final.jar
> > > > > > > > > bval-jsr303-0.5.jar
> > > > > > > > > bval-core-0.5.jar
> > > > > > > > > commons-lang3-3.1.jar
> > > > > > > > > commons-beanutils-1.8.3.jar
> > > > > > > > > httpclient-4.3.5.jar
> > > > > > > > > commons-codec-1.10.jar
> > > > > > > > > zip4j-1.3.2.jar
> > > > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > > > > jersey-apache-client4-1.9.jar
> > > > > > > > > jline-2.11.jar
> > > > > > > > > ant-1.9.2.jar
> > > > > > > > > ant-launcher-1.9.2.jar
> > > > > > > > > mbassador-1.1.9.jar
> > > > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > > > > jackson-xc-1.9.2.jar
> > > > > > > > > hadoop-common-2.2.0-tests.jar
> > > > > > > > >
> > > > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> > works
> > > > > > properly
> > > > > > > > for
> > > > > > > > > launch+shutdown+kill of apps with only these libraries
> > present
> > > in
> > > > > > > > isolation
> > > > > > > > > without dependency on local m2.
> > > > > > > > > I'm believe that there are unwanted jars which are used for
> > > > compile
> > > > > > > time
> > > > > > > > > dependency and not runtime in above list which I can drop
> to
> > > keep
> > > > > > > package
> > > > > > > > > size minimal.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > @Anniruddha,
> > > > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> > > required.
> > > > > > Only
> > > > > > > > > required values I see in it are classpath. and dtcli is
> > capable
> > > > of
> > > > > > > > building
> > > > > > > > > it on the fly.
> > > > > > > > >
> > > > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find
> > the
> > > > jar
> > > > > > > path,
> > > > > > > > > dt-env.sh and all such conf files. If we need a change in
> > > dtcli,
> > > > > then
> > > > > > > > > community's opinion is required for whether dtcli should
> > change
> > > > in
> > > > > > our
> > > > > > > > repo
> > > > > > > > > or a copy of that with required changes exist in bigtop
> repo
> > > > until
> > > > > we
> > > > > > > > make
> > > > > > > > > the dtcli generic enough.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Can we have these file paths set using environment variables
> > with
> > > > > some
> > > > > > > > default values in dtcli?
> > > > > > > > Then we can set such params in dt-env.sh.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 3. dt-sited.xml -> How is this file different from
> > dt-site.xml?
> > > > Not
> > > > > > > sure
> > > > > > > > if
> > > > > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel
> > are
> > > > > > copied
> > > > > > > to
> > > > > > > > > home of new user when useradd program is called. But for
> > > existing
> > > > > > users
> > > > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on
> > the
> > > > fly
> > > > > > if
> > > > > > > > not
> > > > > > > > > encountered for the first time. Again correct me if I'm
> > wrong.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yes, you are right. However this would probably be necessary
> > step
> > > > for
> > > > > > > > rpm/deb, as they may also get installed during OS install and
> > > > before
> > > > > > user
> > > > > > > > accounts were created.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
> > best
> > > > > idea.
> > > > > > > > We'll
> > > > > > > > > anyway put dtcli in location which is by default present in
> > > path
> > > > > i.e.
> > > > > > > > > /usr/bin.
> > > > > > > >
> > > > > > > > For other env variables specific to apex, I'll prefer to use
> > > > > dt-env.sh
> > > > > > > and
> > > > > > > > > source it in dtcli rather than changing bashrc etc...
> > > > > > > >
> > > > > > > >
> > > > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> > cleaner
> > > > way
> > > > > > to
> > > > > > > > achieve the same.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > > Chinmay.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > > > > > aniruddha@datatorrent.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > @Chinmay,
> > > > > > > > > >
> > > > > > > > > > We need to consider following:
> > > > > > > > > > System wide default config files can be located at
> > following
> > > > > > > locations:
> > > > > > > > > >
> > > > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> behaviour
> > > on
> > > > > how
> > > > > > it
> > > > > > > > > finds
> > > > > > > > > > those locations)
> > > > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> > behaviour
> > > on
> > > > > how
> > > > > > > it
> > > > > > > > > > finds those locations)
> > > > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users
> > > that
> > > > > will
> > > > > > > be
> > > > > > > > > > created in system in future.)
> > > > > > > > > >
> > > > > > > > > > We may also have to modify bashrc / bashprofile for
> > > population
> > > > > any
> > > > > > > > > > variables that are required.
> > > > > > > > > >
> > > > > > > > > > @dev, please put in your comments / suggestion.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Aniruddha
> > > > > > > > > >
> > > > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > > > > > thomas@datatorrent.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > -->
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > > > > > chinmay@datatorrent.com
> > > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello Everyone!!
> > > > > > > > > > > >
> > > > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
> > here
> > > > are
> > > > > > some
> > > > > > > > > > > proposals
> > > > > > > > > > > > about package structure etc..
> > > > > > > > > > > > Before posting it on bbigtop mailing list, I have
> some
> > > > > question
> > > > > > > for
> > > > > > > > > > apex
> > > > > > > > > > > > community.
> > > > > > > > > > > >
> > > > > > > > > > > > Proposed Directory structure of apex package for both
> > > deb &
> > > > > > rpm:
> > > > > > > > > > > >
> > > > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > > > > > >
> > > > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > > > > > <other dependent jars> = All the 3rd party jars which
> > are
> > > > > > > required
> > > > > > > > > for
> > > > > > > > > > > apex
> > > > > > > > > > > > to run. Usually the dependencies are packaged as part
> > of
> > > > > > rpm/deb
> > > > > > > by
> > > > > > > > > any
> > > > > > > > > > > > software in bigtop.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Can you specify what those jars are in Bigtop context?
> > Same
> > > > as
> > > > > > > > shipped
> > > > > > > > > > with
> > > > > > > > > > > DT community addition under lib/ ?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
> > > party
> > > > > jars
> > > > > > > > which
> > > > > > > > > > > needs
> > > > > > > > > > > > to included while packaging.
> > > > > > > > > > > >
> > > > > > > > > > > > Questions related to this:
> > > > > > > > > > > > 1. Should we call the cli of apex as "apex" instead
> of
> > > > > "dtcli"
> > > > > > in
> > > > > > > > > > bigtop
> > > > > > > > > > > > package?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I think we should keep the name until we are able to
> > change
> > > > it
> > > > > in
> > > > > > > > Apex.
> > > > > > > > > > > Otherwise this may get confusing.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > 2. I see that all softwares in bigtop have man page
> for
> > > > their
> > > > > > > > > > > executables.
> > > > > > > > > > > > I think we should have it too for dtcli. Is there any
> > > > > > > documentation
> > > > > > > > > > > which I
> > > > > > > > > > > > can convert to man page? or can I use output of
> "dtcli
> > > > > --help"?
> > > > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> > 3.3.0
> > > OR
> > > > > > > > > > > > 3.3.0-incubating?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Has to be -incubating
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
> > 2.7.1
> > > > > > > version
> > > > > > > > of
> > > > > > > > > > > > bigtop hadoop? Any problems that we see with this
> > > > dependency?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Should work without changed. I thought we certified
> > against
> > > > > > 2.7.0?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > 5. Following is the apache mirror from which bigtop
> > will
> > > > pick
> > > > > > the
> > > > > > > > > apex
> > > > > > > > > > > > source for compilation and packaging. Please correct
> if
> > > > > > > incorrect:
> > > > > > > > > > > >
> > > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
> all
> > > the
> > > > > > > > softwares
> > > > > > > > > in
> > > > > > > > > > > > bigtop.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Please share your opinion.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Chinmay.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > > > > > > chinmay@datatorrent.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've already started discussion on bigtop dev
> mailing
> > > > list
> > > > > > for
> > > > > > > > > > > > > integration. Also created a Jira for the same. For
> > this
> > > > > > > > communities
> > > > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > > > > > > thomas@datatorrent.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Chinmay,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> I don't see anything under prerequisites that
> would
> > > be a
> > > > > > > > problem.
> > > > > > > > > We
> > > > > > > > > > > > >> looked
> > > > > > > > > > > > >> at the ASF licencing compatibility as part of
> > becoming
> > > > an
> > > > > > > > > incubator
> > > > > > > > > > > > >> project.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Please focus on the packaging during the next
> weeks.
> > > > Since
> > > > > > the
> > > > > > > > > work
> > > > > > > > > > > will
> > > > > > > > > > > > >> be
> > > > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> should
> > > > also
> > > > > be
> > > > > > > > > there.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Would be good to have the packaging in place by
> end
> > > Feb.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >> Thomas
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> Kolhatkar <
> > > > > > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > We're planning a work on adding Apache Apex as a
> > > > > component
> > > > > > > to
> > > > > > > > > > Apache
> > > > > > > > > > > > >> > Bigtop.
> > > > > > > > > > > > >> > Bigtop is the packaging system for the Apache
> big
> > > data
> > > > > > > > > ecosystem.
> > > > > > > > > > > > >> Several
> > > > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for
> the
> > > > same:
> > > > > > > > > > > > >> >
> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > > > > > >> > Apache bigtop has some hard and soft expectation
> > for
> > > > new
> > > > > > > > > > components
> > > > > > > > > > > to
> > > > > > > > > > > > >> get
> > > > > > > > > > > > >> > integrated into Bigtop.
> > > > > > > > > > > > >> > Here is the list of it:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Most of them seems to be standard ASF based
> > > > > requirements,
> > > > > > > but
> > > > > > > > > few
> > > > > > > > > > > need
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> > be checked for:
> > > > > > > > > > > > >> > 1. Software projects are expected to be Licensed
> > > under
> > > > > > > Apache
> > > > > > > > > > > License,
> > > > > > > > > > > > >> > Version 2.0 (and their dependencies are expected
> > to
> > > be
> > > > > > > > > compatible
> > > > > > > > > > > with
> > > > > > > > > > > > >> this
> > > > > > > > > > > > >> > license)
> > > > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > > > > > > dependencies
> > > > > > > > of
> > > > > > > > > > > Apex
> > > > > > > > > > > > >> are
> > > > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > > > > > > verification
> > > > > > > > > > check.
> > > > > > > > > > > > >> > 2. Software projects are expected to be
> compatible
> > > > with
> > > > > > all
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > >> > supported platforms that Bigtop distribution is
> > > > > targeting
> > > > > > > > > > > > >> >     - This needs verifying whether our software
> > runs
> > > > > fine
> > > > > > in
> > > > > > > > > > > centos-6
> > > > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > > > opensuse-13.2.
> > > > > > > > > > > > >> > 3. What smoke tests that should be added for
> > > > deployment.
> > > > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> > beyond
> > > > > smoke
> > > > > > > test
> > > > > > > > > > > > >> >     - These are basically the integration tests
> > for
> > > > > > > > verification
> > > > > > > > > > > after
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> > deployment. This is a soft requirement, but aim
> is
> > > to
> > > > > > > achieve
> > > > > > > > > this
> > > > > > > > > > > as
> > > > > > > > > > > > >> well
> > > > > > > > > > > > >> > or at least have explanation why not to include.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > If there are any from the link which explicitly
> > > needs
> > > > to
> > > > > > be
> > > > > > > > > > checked
> > > > > > > > > > > > >> other
> > > > > > > > > > > > >> > than above 4, please let us know.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
> > > learnt
> > > > > that
> > > > > > > the
> > > > > > > > > > > bigtop
> > > > > > > > > > > > >> > community want to see the addition of new
> > components
> > > > in
> > > > > > > > phases.
> > > > > > > > > > Here
> > > > > > > > > > > > are
> > > > > > > > > > > > >> > the phases:
> > > > > > > > > > > > >> > 1. Packaging
> > > > > > > > > > > > >> >     - This needs creating of package i.e. rpm &
> > deb
> > > > > files.
> > > > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> DISCLAMER,
> > > > NOTES
> > > > > > etc
> > > > > > > if
> > > > > > > > > any
> > > > > > > > > > > > >> needed.
> > > > > > > > > > > > >> >     - Any documentation that need to be added to
> > > > > > > distribution
> > > > > > > > of
> > > > > > > > > > our
> > > > > > > > > > > > >> > software.
> > > > > > > > > > > > >> >     - Any license information of dependencies
> > > required
> > > > > to
> > > > > > be
> > > > > > > > > added
> > > > > > > > > > > to
> > > > > > > > > > > > >> > package
> > > > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > > > > > configuration
> > > > > > > > > > > > >> >     - Add puppet recipes for automatic
> deployment
> > > > > > > > > > > > >> > 4. Integration tests
> > > > > > > > > > > > >> >     - For verification of deployments.
> > > > > > > > > > > > >> > 5. license clearance:
> > > > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff
> is
> > > > > > compliant
> > > > > > > > > with
> > > > > > > > > > > ASF
> > > > > > > > > > > > >> > license requirements. If you add code licenses
> > under
> > > > > > > different
> > > > > > > > > > > > licenses,
> > > > > > > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > > > > > >> > We'll start corresponding communication on
> bigtop
> > > > > mailing
> > > > > > > list
> > > > > > > > > as
> > > > > > > > > > > > well.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > We have some specific questions/suggestions
> > related
> > > to
> > > > > > what
> > > > > > > > > should
> > > > > > > > > > > be
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> > content of the package and what should be the
> > smoke
> > > > > tests,
> > > > > > > but
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > >> > interest of not having too much content here,
> > we'll
> > > > put
> > > > > > the
> > > > > > > > > > > questions
> > > > > > > > > > > > >> as a
> > > > > > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Thanks,
> > > > > > > > > > > > >> > Chinmay.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Aniruddha Thombare <an...@datatorrent.com>]

AFI recall...

Clirc
Clirc history
Etc

Default site.xml and env can be kept too.




Sent from handheld

On Wed, 10 Feb 2016 10:25 pm Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> Ok. Is there any content that we should add in .dt ?
> On 10 Feb 2016 22:10, "Aniruddha Thombare" <an...@datatorrent.com>
> wrote:
>
> > Hi,
> >
> > .dt directory.
> > In future, it may  become .apex
> > On 10 Feb 2016 9:53 pm, "Chinmay Kolhatkar" <ch...@datatorrent.com>
> > wrote:
> >
> > > Hi Aniruddha,
> > >
> > > I like the idea of skeleton directory. But I don't see any use of it,
> > > unless we've anything in default.
> > > If you have something in your mind which should be default, can you
> > please
> > > share it?
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > > On Wed, Feb 10, 2016 at 9:49 PM, Aniruddha Thombare <
> > > aniruddha@datatorrent.com> wrote:
> > >
> > > > On point 2:
> > > > Skeleton directory is for default home dir structure for future
> users.
> > > > By avoiding that, we can't inconvenience users and admins.
> > > > Defaults in that configuration can be specified or evaluated at
> > install.
> > > >
> > > >
> > > >
> > > > On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <
> chinmay@datatorrent.com
> > >
> > > > wrote:
> > > >
> > > > > Really good points Pradeep and Aniruddha.
> > > > >
> > > > > 1. I believe we won't need to change the dtcli considering it works
> > > with
> > > > DT
> > > > > Community edition. We can keep the directory structure similar to
> > that.
> > > > > dt-env.sh has variables which contains information required for
> dtcli
> > > to
> > > > > launch.
> > > > >
> > > > > 2. Let me check with bigtop community that whether they facilitate
> > the
> > > > > installation of rpms/debs before user is created. In either case,
> > > current
> > > > > dtcli creates a .dt folder in home directory. Also before putting
> > > > anything
> > > > > in /etc/skel we need to define what are the default contents that
> > > should
> > > > go
> > > > > to ~/.dt/ folder. If there is no defaults, probably we should not
> > > > > explicitly add it in /etc/skel.
> > > > >
> > > > > 3.  /etc/profile.d approach looks nice. This way contents of
> > dt-env.sh
> > > > are
> > > > > present as env variables. But I see a catch there. Adding dt-env.sh
> > to
> > > > > /etc/profile.d would make all the variables available at runtime
> all
> > > the
> > > > > time. I feel a little skeptical about that. Maybe a possible
> > collision
> > > > can
> > > > > occur with other application vars.
> > > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > > > >
> > > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > > > Jira/repository.
> > > > > Packaging related code usually goes there. (That's what all the
> > > > components
> > > > > in bigtop does).
> > > > > Having said that, once a PR is created, I'll be sharing the link of
> > PR
> > > > > here, so that apex community as well can review it.
> > > > >
> > > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > > > aniruddha@datatorrent.com> wrote:
> > > > >
> > > > > > +1 on suggestions and approach.
> > > > > > We may need to iron out details about exact paths etc.
> > > > > > Which can be done on jira / PR comments.
> > > > > > Is that right @dev?
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > apache@pradeepdalvi.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Inline comments...
> > > > > > >
> > > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > > > > chinmay@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > @Thomas, Not all the jar present in DT community edition will
> > be
> > > > > > included
> > > > > > > > there.
> > > > > > > >
> > > > > > > > Here is the list of jars I found from common dependencies
> > between
> > > > of
> > > > > > apex
> > > > > > > > and DT Community edition:
> > > > > > > > netlet-1.2.0.jar
> > > > > > > > kryo-2.24.0.jar
> > > > > > > > jackson-core-asl-1.9.2.jar
> > > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > > > > async-http-client-1.7.20.jar
> > > > > > > > netty-3.6.6.Final.jar
> > > > > > > > validation-api-1.1.0.Final.jar
> > > > > > > > bval-jsr303-0.5.jar
> > > > > > > > bval-core-0.5.jar
> > > > > > > > commons-lang3-3.1.jar
> > > > > > > > commons-beanutils-1.8.3.jar
> > > > > > > > httpclient-4.3.5.jar
> > > > > > > > commons-codec-1.10.jar
> > > > > > > > zip4j-1.3.2.jar
> > > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > > > jersey-apache-client4-1.9.jar
> > > > > > > > jline-2.11.jar
> > > > > > > > ant-1.9.2.jar
> > > > > > > > ant-launcher-1.9.2.jar
> > > > > > > > mbassador-1.1.9.jar
> > > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > > > jackson-xc-1.9.2.jar
> > > > > > > > hadoop-common-2.2.0-tests.jar
> > > > > > > >
> > > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> works
> > > > > properly
> > > > > > > for
> > > > > > > > launch+shutdown+kill of apps with only these libraries
> present
> > in
> > > > > > > isolation
> > > > > > > > without dependency on local m2.
> > > > > > > > I'm believe that there are unwanted jars which are used for
> > > compile
> > > > > > time
> > > > > > > > dependency and not runtime in above list which I can drop to
> > keep
> > > > > > package
> > > > > > > > size minimal.
> > > > > > > >
> > > > > > > >
> > > > > > > > @Anniruddha,
> > > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> > required.
> > > > > Only
> > > > > > > > required values I see in it are classpath. and dtcli is
> capable
> > > of
> > > > > > > building
> > > > > > > > it on the fly.
> > > > > > > >
> > > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find
> the
> > > jar
> > > > > > path,
> > > > > > > > dt-env.sh and all such conf files. If we need a change in
> > dtcli,
> > > > then
> > > > > > > > community's opinion is required for whether dtcli should
> change
> > > in
> > > > > our
> > > > > > > repo
> > > > > > > > or a copy of that with required changes exist in bigtop repo
> > > until
> > > > we
> > > > > > > make
> > > > > > > > the dtcli generic enough.
> > > > > > > >
> > > > > > >
> > > > > > > Can we have these file paths set using environment variables
> with
> > > > some
> > > > > > > default values in dtcli?
> > > > > > > Then we can set such params in dt-env.sh.
> > > > > > >
> > > > > > >
> > > > > > > > 3. dt-sited.xml -> How is this file different from
> dt-site.xml?
> > > Not
> > > > > > sure
> > > > > > > if
> > > > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel
> are
> > > > > copied
> > > > > > to
> > > > > > > > home of new user when useradd program is called. But for
> > existing
> > > > > users
> > > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on
> the
> > > fly
> > > > > if
> > > > > > > not
> > > > > > > > encountered for the first time. Again correct me if I'm
> wrong.
> > > > > > > >
> > > > > > >
> > > > > > > Yes, you are right. However this would probably be necessary
> step
> > > for
> > > > > > > rpm/deb, as they may also get installed during OS install and
> > > before
> > > > > user
> > > > > > > accounts were created.
> > > > > > >
> > > > > > >
> > > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
> best
> > > > idea.
> > > > > > > We'll
> > > > > > > > anyway put dtcli in location which is by default present in
> > path
> > > > i.e.
> > > > > > > > /usr/bin.
> > > > > > >
> > > > > > > For other env variables specific to apex, I'll prefer to use
> > > > dt-env.sh
> > > > > > and
> > > > > > > > source it in dtcli rather than changing bashrc etc...
> > > > > > >
> > > > > > >
> > > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> cleaner
> > > way
> > > > > to
> > > > > > > achieve the same.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > Thanks,
> > > > > > > > Chinmay.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > > > > aniruddha@datatorrent.com> wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > @Chinmay,
> > > > > > > > >
> > > > > > > > > We need to consider following:
> > > > > > > > > System wide default config files can be located at
> following
> > > > > > locations:
> > > > > > > > >
> > > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour
> > on
> > > > how
> > > > > it
> > > > > > > > finds
> > > > > > > > > those locations)
> > > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> behaviour
> > on
> > > > how
> > > > > > it
> > > > > > > > > finds those locations)
> > > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users
> > that
> > > > will
> > > > > > be
> > > > > > > > > created in system in future.)
> > > > > > > > >
> > > > > > > > > We may also have to modify bashrc / bashprofile for
> > population
> > > > any
> > > > > > > > > variables that are required.
> > > > > > > > >
> > > > > > > > > @dev, please put in your comments / suggestion.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Aniruddha
> > > > > > > > >
> > > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > > > > thomas@datatorrent.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > -->
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > > > > chinmay@datatorrent.com
> > > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello Everyone!!
> > > > > > > > > > >
> > > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
> here
> > > are
> > > > > some
> > > > > > > > > > proposals
> > > > > > > > > > > about package structure etc..
> > > > > > > > > > > Before posting it on bbigtop mailing list, I have some
> > > > question
> > > > > > for
> > > > > > > > > apex
> > > > > > > > > > > community.
> > > > > > > > > > >
> > > > > > > > > > > Proposed Directory structure of apex package for both
> > deb &
> > > > > rpm:
> > > > > > > > > > >
> > > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > > > > >
> > > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > > > > <other dependent jars> = All the 3rd party jars which
> are
> > > > > > required
> > > > > > > > for
> > > > > > > > > > apex
> > > > > > > > > > > to run. Usually the dependencies are packaged as part
> of
> > > > > rpm/deb
> > > > > > by
> > > > > > > > any
> > > > > > > > > > > software in bigtop.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Can you specify what those jars are in Bigtop context?
> Same
> > > as
> > > > > > > shipped
> > > > > > > > > with
> > > > > > > > > > DT community addition under lib/ ?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
> > party
> > > > jars
> > > > > > > which
> > > > > > > > > > needs
> > > > > > > > > > > to included while packaging.
> > > > > > > > > > >
> > > > > > > > > > > Questions related to this:
> > > > > > > > > > > 1. Should we call the cli of apex as "apex" instead of
> > > > "dtcli"
> > > > > in
> > > > > > > > > bigtop
> > > > > > > > > > > package?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I think we should keep the name until we are able to
> change
> > > it
> > > > in
> > > > > > > Apex.
> > > > > > > > > > Otherwise this may get confusing.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > 2. I see that all softwares in bigtop have man page for
> > > their
> > > > > > > > > > executables.
> > > > > > > > > > > I think we should have it too for dtcli. Is there any
> > > > > > documentation
> > > > > > > > > > which I
> > > > > > > > > > > can convert to man page? or can I use output of "dtcli
> > > > --help"?
> > > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> 3.3.0
> > OR
> > > > > > > > > > > 3.3.0-incubating?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Has to be -incubating
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
> 2.7.1
> > > > > > version
> > > > > > > of
> > > > > > > > > > > bigtop hadoop? Any problems that we see with this
> > > dependency?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Should work without changed. I thought we certified
> against
> > > > > 2.7.0?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > 5. Following is the apache mirror from which bigtop
> will
> > > pick
> > > > > the
> > > > > > > > apex
> > > > > > > > > > > source for compilation and packaging. Please correct if
> > > > > > incorrect:
> > > > > > > > > > >
> > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by all
> > the
> > > > > > > softwares
> > > > > > > > in
> > > > > > > > > > > bigtop.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Please share your opinion.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Chinmay.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > > > > > chinmay@datatorrent.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > > > > >
> > > > > > > > > > > > I've already started discussion on bigtop dev mailing
> > > list
> > > > > for
> > > > > > > > > > > > integration. Also created a Jira for the same. For
> this
> > > > > > > communities
> > > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > > > > > thomas@datatorrent.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Chinmay,
> > > > > > > > > > > >>
> > > > > > > > > > > >> I don't see anything under prerequisites that would
> > be a
> > > > > > > problem.
> > > > > > > > We
> > > > > > > > > > > >> looked
> > > > > > > > > > > >> at the ASF licencing compatibility as part of
> becoming
> > > an
> > > > > > > > incubator
> > > > > > > > > > > >> project.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Please focus on the packaging during the next weeks.
> > > Since
> > > > > the
> > > > > > > > work
> > > > > > > > > > will
> > > > > > > > > > > >> be
> > > > > > > > > > > >> part of Bigtop, related discussions and JIRAs should
> > > also
> > > > be
> > > > > > > > there.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Would be good to have the packaging in place by end
> > Feb.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks,
> > > > > > > > > > > >> Thomas
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Hi All,
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > We're planning a work on adding Apache Apex as a
> > > > component
> > > > > > to
> > > > > > > > > Apache
> > > > > > > > > > > >> > Bigtop.
> > > > > > > > > > > >> > Bigtop is the packaging system for the Apache big
> > data
> > > > > > > > ecosystem.
> > > > > > > > > > > >> Several
> > > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for the
> > > same:
> > > > > > > > > > > >> >
> https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > > > > >> > Apache bigtop has some hard and soft expectation
> for
> > > new
> > > > > > > > > components
> > > > > > > > > > to
> > > > > > > > > > > >> get
> > > > > > > > > > > >> > integrated into Bigtop.
> > > > > > > > > > > >> > Here is the list of it:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Most of them seems to be standard ASF based
> > > > requirements,
> > > > > > but
> > > > > > > > few
> > > > > > > > > > need
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > be checked for:
> > > > > > > > > > > >> > 1. Software projects are expected to be Licensed
> > under
> > > > > > Apache
> > > > > > > > > > License,
> > > > > > > > > > > >> > Version 2.0 (and their dependencies are expected
> to
> > be
> > > > > > > > compatible
> > > > > > > > > > with
> > > > > > > > > > > >> this
> > > > > > > > > > > >> > license)
> > > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > > > > > dependencies
> > > > > > > of
> > > > > > > > > > Apex
> > > > > > > > > > > >> are
> > > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > > > > > verification
> > > > > > > > > check.
> > > > > > > > > > > >> > 2. Software projects are expected to be compatible
> > > with
> > > > > all
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > >> > supported platforms that Bigtop distribution is
> > > > targeting
> > > > > > > > > > > >> >     - This needs verifying whether our software
> runs
> > > > fine
> > > > > in
> > > > > > > > > > centos-6
> > > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > > opensuse-13.2.
> > > > > > > > > > > >> > 3. What smoke tests that should be added for
> > > deployment.
> > > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> beyond
> > > > smoke
> > > > > > test
> > > > > > > > > > > >> >     - These are basically the integration tests
> for
> > > > > > > verification
> > > > > > > > > > after
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > deployment. This is a soft requirement, but aim is
> > to
> > > > > > achieve
> > > > > > > > this
> > > > > > > > > > as
> > > > > > > > > > > >> well
> > > > > > > > > > > >> > or at least have explanation why not to include.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > If there are any from the link which explicitly
> > needs
> > > to
> > > > > be
> > > > > > > > > checked
> > > > > > > > > > > >> other
> > > > > > > > > > > >> > than above 4, please let us know.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
> > learnt
> > > > that
> > > > > > the
> > > > > > > > > > bigtop
> > > > > > > > > > > >> > community want to see the addition of new
> components
> > > in
> > > > > > > phases.
> > > > > > > > > Here
> > > > > > > > > > > are
> > > > > > > > > > > >> > the phases:
> > > > > > > > > > > >> > 1. Packaging
> > > > > > > > > > > >> >     - This needs creating of package i.e. rpm &
> deb
> > > > files.
> > > > > > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER,
> > > NOTES
> > > > > etc
> > > > > > if
> > > > > > > > any
> > > > > > > > > > > >> needed.
> > > > > > > > > > > >> >     - Any documentation that need to be added to
> > > > > > distribution
> > > > > > > of
> > > > > > > > > our
> > > > > > > > > > > >> > software.
> > > > > > > > > > > >> >     - Any license information of dependencies
> > required
> > > > to
> > > > > be
> > > > > > > > added
> > > > > > > > > > to
> > > > > > > > > > > >> > package
> > > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > > > > configuration
> > > > > > > > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > > > > > > > >> > 4. Integration tests
> > > > > > > > > > > >> >     - For verification of deployments.
> > > > > > > > > > > >> > 5. license clearance:
> > > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff is
> > > > > compliant
> > > > > > > > with
> > > > > > > > > > ASF
> > > > > > > > > > > >> > license requirements. If you add code licenses
> under
> > > > > > different
> > > > > > > > > > > licenses,
> > > > > > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > > > > >> > We'll start corresponding communication on bigtop
> > > > mailing
> > > > > > list
> > > > > > > > as
> > > > > > > > > > > well.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > We have some specific questions/suggestions
> related
> > to
> > > > > what
> > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > content of the package and what should be the
> smoke
> > > > tests,
> > > > > > but
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > >> > interest of not having too much content here,
> we'll
> > > put
> > > > > the
> > > > > > > > > > questions
> > > > > > > > > > > >> as a
> > > > > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Thanks,
> > > > > > > > > > > >> > Chinmay.
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@datatorrent.com>]

Ok. Is there any content that we should add in .dt ?
On 10 Feb 2016 22:10, "Aniruddha Thombare" <an...@datatorrent.com>
wrote:

> Hi,
>
> .dt directory.
> In future, it may  become .apex
> On 10 Feb 2016 9:53 pm, "Chinmay Kolhatkar" <ch...@datatorrent.com>
> wrote:
>
> > Hi Aniruddha,
> >
> > I like the idea of skeleton directory. But I don't see any use of it,
> > unless we've anything in default.
> > If you have something in your mind which should be default, can you
> please
> > share it?
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Wed, Feb 10, 2016 at 9:49 PM, Aniruddha Thombare <
> > aniruddha@datatorrent.com> wrote:
> >
> > > On point 2:
> > > Skeleton directory is for default home dir structure for future users.
> > > By avoiding that, we can't inconvenience users and admins.
> > > Defaults in that configuration can be specified or evaluated at
> install.
> > >
> > >
> > >
> > > On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <chinmay@datatorrent.com
> >
> > > wrote:
> > >
> > > > Really good points Pradeep and Aniruddha.
> > > >
> > > > 1. I believe we won't need to change the dtcli considering it works
> > with
> > > DT
> > > > Community edition. We can keep the directory structure similar to
> that.
> > > > dt-env.sh has variables which contains information required for dtcli
> > to
> > > > launch.
> > > >
> > > > 2. Let me check with bigtop community that whether they facilitate
> the
> > > > installation of rpms/debs before user is created. In either case,
> > current
> > > > dtcli creates a .dt folder in home directory. Also before putting
> > > anything
> > > > in /etc/skel we need to define what are the default contents that
> > should
> > > go
> > > > to ~/.dt/ folder. If there is no defaults, probably we should not
> > > > explicitly add it in /etc/skel.
> > > >
> > > > 3.  /etc/profile.d approach looks nice. This way contents of
> dt-env.sh
> > > are
> > > > present as env variables. But I see a catch there. Adding dt-env.sh
> to
> > > > /etc/profile.d would make all the variables available at runtime all
> > the
> > > > time. I feel a little skeptical about that. Maybe a possible
> collision
> > > can
> > > > occur with other application vars.
> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > > >
> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > > Jira/repository.
> > > > Packaging related code usually goes there. (That's what all the
> > > components
> > > > in bigtop does).
> > > > Having said that, once a PR is created, I'll be sharing the link of
> PR
> > > > here, so that apex community as well can review it.
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > > aniruddha@datatorrent.com> wrote:
> > > >
> > > > > +1 on suggestions and approach.
> > > > > We may need to iron out details about exact paths etc.
> > > > > Which can be done on jira / PR comments.
> > > > > Is that right @dev?
> > > > >
> > > > >
> > > > >
> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > apache@pradeepdalvi.com>
> > > > > wrote:
> > > > >
> > > > > > Inline comments...
> > > > > >
> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > > > chinmay@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > @Thomas, Not all the jar present in DT community edition will
> be
> > > > > included
> > > > > > > there.
> > > > > > >
> > > > > > > Here is the list of jars I found from common dependencies
> between
> > > of
> > > > > apex
> > > > > > > and DT Community edition:
> > > > > > > netlet-1.2.0.jar
> > > > > > > kryo-2.24.0.jar
> > > > > > > jackson-core-asl-1.9.2.jar
> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > > > async-http-client-1.7.20.jar
> > > > > > > netty-3.6.6.Final.jar
> > > > > > > validation-api-1.1.0.Final.jar
> > > > > > > bval-jsr303-0.5.jar
> > > > > > > bval-core-0.5.jar
> > > > > > > commons-lang3-3.1.jar
> > > > > > > commons-beanutils-1.8.3.jar
> > > > > > > httpclient-4.3.5.jar
> > > > > > > commons-codec-1.10.jar
> > > > > > > zip4j-1.3.2.jar
> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > > jersey-apache-client4-1.9.jar
> > > > > > > jline-2.11.jar
> > > > > > > ant-1.9.2.jar
> > > > > > > ant-launcher-1.9.2.jar
> > > > > > > mbassador-1.1.9.jar
> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > > jackson-xc-1.9.2.jar
> > > > > > > hadoop-common-2.2.0-tests.jar
> > > > > > >
> > > > > > > Ofcourse, I'll be running some tests do check that dtcli works
> > > > properly
> > > > > > for
> > > > > > > launch+shutdown+kill of apps with only these libraries present
> in
> > > > > > isolation
> > > > > > > without dependency on local m2.
> > > > > > > I'm believe that there are unwanted jars which are used for
> > compile
> > > > > time
> > > > > > > dependency and not runtime in above list which I can drop to
> keep
> > > > > package
> > > > > > > size minimal.
> > > > > > >
> > > > > > >
> > > > > > > @Anniruddha,
> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> required.
> > > > Only
> > > > > > > required values I see in it are classpath. and dtcli is capable
> > of
> > > > > > building
> > > > > > > it on the fly.
> > > > > > >
> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the
> > jar
> > > > > path,
> > > > > > > dt-env.sh and all such conf files. If we need a change in
> dtcli,
> > > then
> > > > > > > community's opinion is required for whether dtcli should change
> > in
> > > > our
> > > > > > repo
> > > > > > > or a copy of that with required changes exist in bigtop repo
> > until
> > > we
> > > > > > make
> > > > > > > the dtcli generic enough.
> > > > > > >
> > > > > >
> > > > > > Can we have these file paths set using environment variables with
> > > some
> > > > > > default values in dtcli?
> > > > > > Then we can set such params in dt-env.sh.
> > > > > >
> > > > > >
> > > > > > > 3. dt-sited.xml -> How is this file different from dt-site.xml?
> > Not
> > > > > sure
> > > > > > if
> > > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel are
> > > > copied
> > > > > to
> > > > > > > home of new user when useradd program is called. But for
> existing
> > > > users
> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the
> > fly
> > > > if
> > > > > > not
> > > > > > > encountered for the first time. Again correct me if I'm wrong.
> > > > > > >
> > > > > >
> > > > > > Yes, you are right. However this would probably be necessary step
> > for
> > > > > > rpm/deb, as they may also get installed during OS install and
> > before
> > > > user
> > > > > > accounts were created.
> > > > > >
> > > > > >
> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best
> > > idea.
> > > > > > We'll
> > > > > > > anyway put dtcli in location which is by default present in
> path
> > > i.e.
> > > > > > > /usr/bin.
> > > > > >
> > > > > > For other env variables specific to apex, I'll prefer to use
> > > dt-env.sh
> > > > > and
> > > > > > > source it in dtcli rather than changing bashrc etc...
> > > > > >
> > > > > >
> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner
> > way
> > > > to
> > > > > > achieve the same.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > Thanks,
> > > > > > > Chinmay.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > > > aniruddha@datatorrent.com> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > @Chinmay,
> > > > > > > >
> > > > > > > > We need to consider following:
> > > > > > > > System wide default config files can be located at following
> > > > > locations:
> > > > > > > >
> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour
> on
> > > how
> > > > it
> > > > > > > finds
> > > > > > > > those locations)
> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour
> on
> > > how
> > > > > it
> > > > > > > > finds those locations)
> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users
> that
> > > will
> > > > > be
> > > > > > > > created in system in future.)
> > > > > > > >
> > > > > > > > We may also have to modify bashrc / bashprofile for
> population
> > > any
> > > > > > > > variables that are required.
> > > > > > > >
> > > > > > > > @dev, please put in your comments / suggestion.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > >
> > > > > > > > Aniruddha
> > > > > > > >
> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > > > thomas@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > -->
> > > > > > > > >
> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > > > chinmay@datatorrent.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello Everyone!!
> > > > > > > > > >
> > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex, here
> > are
> > > > some
> > > > > > > > > proposals
> > > > > > > > > > about package structure etc..
> > > > > > > > > > Before posting it on bbigtop mailing list, I have some
> > > question
> > > > > for
> > > > > > > > apex
> > > > > > > > > > community.
> > > > > > > > > >
> > > > > > > > > > Proposed Directory structure of apex package for both
> deb &
> > > > rpm:
> > > > > > > > > >
> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > > > >
> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > > > <other dependent jars> = All the 3rd party jars which are
> > > > > required
> > > > > > > for
> > > > > > > > > apex
> > > > > > > > > > to run. Usually the dependencies are packaged as part of
> > > > rpm/deb
> > > > > by
> > > > > > > any
> > > > > > > > > > software in bigtop.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Can you specify what those jars are in Bigtop context? Same
> > as
> > > > > > shipped
> > > > > > > > with
> > > > > > > > > DT community addition under lib/ ?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
> party
> > > jars
> > > > > > which
> > > > > > > > > needs
> > > > > > > > > > to included while packaging.
> > > > > > > > > >
> > > > > > > > > > Questions related to this:
> > > > > > > > > > 1. Should we call the cli of apex as "apex" instead of
> > > "dtcli"
> > > > in
> > > > > > > > bigtop
> > > > > > > > > > package?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I think we should keep the name until we are able to change
> > it
> > > in
> > > > > > Apex.
> > > > > > > > > Otherwise this may get confusing.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > 2. I see that all softwares in bigtop have man page for
> > their
> > > > > > > > > executables.
> > > > > > > > > > I think we should have it too for dtcli. Is there any
> > > > > documentation
> > > > > > > > > which I
> > > > > > > > > > can convert to man page? or can I use output of "dtcli
> > > --help"?
> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0
> OR
> > > > > > > > > > 3.3.0-incubating?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Has to be -incubating
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1
> > > > > version
> > > > > > of
> > > > > > > > > > bigtop hadoop? Any problems that we see with this
> > dependency?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Should work without changed. I thought we certified against
> > > > 2.7.0?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > 5. Following is the apache mirror from which bigtop will
> > pick
> > > > the
> > > > > > > apex
> > > > > > > > > > source for compilation and packaging. Please correct if
> > > > > incorrect:
> > > > > > > > > >
> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by all
> the
> > > > > > softwares
> > > > > > > in
> > > > > > > > > > bigtop.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Please share your opinion.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Chinmay.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > > > > chinmay@datatorrent.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > > > >
> > > > > > > > > > > I've already started discussion on bigtop dev mailing
> > list
> > > > for
> > > > > > > > > > > integration. Also created a Jira for the same. For this
> > > > > > communities
> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > > > > thomas@datatorrent.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Chinmay,
> > > > > > > > > > >>
> > > > > > > > > > >> I don't see anything under prerequisites that would
> be a
> > > > > > problem.
> > > > > > > We
> > > > > > > > > > >> looked
> > > > > > > > > > >> at the ASF licencing compatibility as part of becoming
> > an
> > > > > > > incubator
> > > > > > > > > > >> project.
> > > > > > > > > > >>
> > > > > > > > > > >> Please focus on the packaging during the next weeks.
> > Since
> > > > the
> > > > > > > work
> > > > > > > > > will
> > > > > > > > > > >> be
> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs should
> > also
> > > be
> > > > > > > there.
> > > > > > > > > > >>
> > > > > > > > > > >> Would be good to have the packaging in place by end
> Feb.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> Thomas
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi All,
> > > > > > > > > > >> >
> > > > > > > > > > >> > We're planning a work on adding Apache Apex as a
> > > component
> > > > > to
> > > > > > > > Apache
> > > > > > > > > > >> > Bigtop.
> > > > > > > > > > >> > Bigtop is the packaging system for the Apache big
> data
> > > > > > > ecosystem.
> > > > > > > > > > >> Several
> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for the
> > same:
> > > > > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > > > >> >
> > > > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > > > >> > Apache bigtop has some hard and soft expectation for
> > new
> > > > > > > > components
> > > > > > > > > to
> > > > > > > > > > >> get
> > > > > > > > > > >> > integrated into Bigtop.
> > > > > > > > > > >> > Here is the list of it:
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > > > >> >
> > > > > > > > > > >> > Most of them seems to be standard ASF based
> > > requirements,
> > > > > but
> > > > > > > few
> > > > > > > > > need
> > > > > > > > > > >> to
> > > > > > > > > > >> > be checked for:
> > > > > > > > > > >> > 1. Software projects are expected to be Licensed
> under
> > > > > Apache
> > > > > > > > > License,
> > > > > > > > > > >> > Version 2.0 (and their dependencies are expected to
> be
> > > > > > > compatible
> > > > > > > > > with
> > > > > > > > > > >> this
> > > > > > > > > > >> > license)
> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > > > > dependencies
> > > > > > of
> > > > > > > > > Apex
> > > > > > > > > > >> are
> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > > > > verification
> > > > > > > > check.
> > > > > > > > > > >> > 2. Software projects are expected to be compatible
> > with
> > > > all
> > > > > of
> > > > > > > the
> > > > > > > > > > >> > supported platforms that Bigtop distribution is
> > > targeting
> > > > > > > > > > >> >     - This needs verifying whether our software runs
> > > fine
> > > > in
> > > > > > > > > centos-6
> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > opensuse-13.2.
> > > > > > > > > > >> > 3. What smoke tests that should be added for
> > deployment.
> > > > > > > > > > >> > 4. Identifying the test artifacts which goes beyond
> > > smoke
> > > > > test
> > > > > > > > > > >> >     - These are basically the integration tests for
> > > > > > verification
> > > > > > > > > after
> > > > > > > > > > >> the
> > > > > > > > > > >> > deployment. This is a soft requirement, but aim is
> to
> > > > > achieve
> > > > > > > this
> > > > > > > > > as
> > > > > > > > > > >> well
> > > > > > > > > > >> > or at least have explanation why not to include.
> > > > > > > > > > >> >
> > > > > > > > > > >> > If there are any from the link which explicitly
> needs
> > to
> > > > be
> > > > > > > > checked
> > > > > > > > > > >> other
> > > > > > > > > > >> > than above 4, please let us know.
> > > > > > > > > > >> >
> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
> learnt
> > > that
> > > > > the
> > > > > > > > > bigtop
> > > > > > > > > > >> > community want to see the addition of new components
> > in
> > > > > > phases.
> > > > > > > > Here
> > > > > > > > > > are
> > > > > > > > > > >> > the phases:
> > > > > > > > > > >> > 1. Packaging
> > > > > > > > > > >> >     - This needs creating of package i.e. rpm & deb
> > > files.
> > > > > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER,
> > NOTES
> > > > etc
> > > > > if
> > > > > > > any
> > > > > > > > > > >> needed.
> > > > > > > > > > >> >     - Any documentation that need to be added to
> > > > > distribution
> > > > > > of
> > > > > > > > our
> > > > > > > > > > >> > software.
> > > > > > > > > > >> >     - Any license information of dependencies
> required
> > > to
> > > > be
> > > > > > > added
> > > > > > > > > to
> > > > > > > > > > >> > package
> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > > > configuration
> > > > > > > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > > > > > > >> > 4. Integration tests
> > > > > > > > > > >> >     - For verification of deployments.
> > > > > > > > > > >> > 5. license clearance:
> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff is
> > > > compliant
> > > > > > > with
> > > > > > > > > ASF
> > > > > > > > > > >> > license requirements. If you add code licenses under
> > > > > different
> > > > > > > > > > licenses,
> > > > > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > > > >> > We'll start corresponding communication on bigtop
> > > mailing
> > > > > list
> > > > > > > as
> > > > > > > > > > well.
> > > > > > > > > > >> >
> > > > > > > > > > >> > We have some specific questions/suggestions related
> to
> > > > what
> > > > > > > should
> > > > > > > > > be
> > > > > > > > > > >> the
> > > > > > > > > > >> > content of the package and what should be the smoke
> > > tests,
> > > > > but
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >> > interest of not having too much content here, we'll
> > put
> > > > the
> > > > > > > > > questions
> > > > > > > > > > >> as a
> > > > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks,
> > > > > > > > > > >> > Chinmay.
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Aniruddha Thombare <an...@datatorrent.com>]

Hi,

.dt directory.
In future, it may  become .apex
On 10 Feb 2016 9:53 pm, "Chinmay Kolhatkar" <ch...@datatorrent.com> wrote:

> Hi Aniruddha,
>
> I like the idea of skeleton directory. But I don't see any use of it,
> unless we've anything in default.
> If you have something in your mind which should be default, can you please
> share it?
>
> Thanks,
> Chinmay.
>
>
> On Wed, Feb 10, 2016 at 9:49 PM, Aniruddha Thombare <
> aniruddha@datatorrent.com> wrote:
>
> > On point 2:
> > Skeleton directory is for default home dir structure for future users.
> > By avoiding that, we can't inconvenience users and admins.
> > Defaults in that configuration can be specified or evaluated at install.
> >
> >
> >
> > On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <ch...@datatorrent.com>
> > wrote:
> >
> > > Really good points Pradeep and Aniruddha.
> > >
> > > 1. I believe we won't need to change the dtcli considering it works
> with
> > DT
> > > Community edition. We can keep the directory structure similar to that.
> > > dt-env.sh has variables which contains information required for dtcli
> to
> > > launch.
> > >
> > > 2. Let me check with bigtop community that whether they facilitate the
> > > installation of rpms/debs before user is created. In either case,
> current
> > > dtcli creates a .dt folder in home directory. Also before putting
> > anything
> > > in /etc/skel we need to define what are the default contents that
> should
> > go
> > > to ~/.dt/ folder. If there is no defaults, probably we should not
> > > explicitly add it in /etc/skel.
> > >
> > > 3.  /etc/profile.d approach looks nice. This way contents of dt-env.sh
> > are
> > > present as env variables. But I see a catch there. Adding dt-env.sh to
> > > /etc/profile.d would make all the variables available at runtime all
> the
> > > time. I feel a little skeptical about that. Maybe a possible collision
> > can
> > > occur with other application vars.
> > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > >
> > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > Jira/repository.
> > > Packaging related code usually goes there. (That's what all the
> > components
> > > in bigtop does).
> > > Having said that, once a PR is created, I'll be sharing the link of PR
> > > here, so that apex community as well can review it.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > aniruddha@datatorrent.com> wrote:
> > >
> > > > +1 on suggestions and approach.
> > > > We may need to iron out details about exact paths etc.
> > > > Which can be done on jira / PR comments.
> > > > Is that right @dev?
> > > >
> > > >
> > > >
> > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> apache@pradeepdalvi.com>
> > > > wrote:
> > > >
> > > > > Inline comments...
> > > > >
> > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > > chinmay@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > @Thomas, Not all the jar present in DT community edition will be
> > > > included
> > > > > > there.
> > > > > >
> > > > > > Here is the list of jars I found from common dependencies between
> > of
> > > > apex
> > > > > > and DT Community edition:
> > > > > > netlet-1.2.0.jar
> > > > > > kryo-2.24.0.jar
> > > > > > jackson-core-asl-1.9.2.jar
> > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > > async-http-client-1.7.20.jar
> > > > > > netty-3.6.6.Final.jar
> > > > > > validation-api-1.1.0.Final.jar
> > > > > > bval-jsr303-0.5.jar
> > > > > > bval-core-0.5.jar
> > > > > > commons-lang3-3.1.jar
> > > > > > commons-beanutils-1.8.3.jar
> > > > > > httpclient-4.3.5.jar
> > > > > > commons-codec-1.10.jar
> > > > > > zip4j-1.3.2.jar
> > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > jersey-apache-client4-1.9.jar
> > > > > > jline-2.11.jar
> > > > > > ant-1.9.2.jar
> > > > > > ant-launcher-1.9.2.jar
> > > > > > mbassador-1.1.9.jar
> > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > jackson-xc-1.9.2.jar
> > > > > > hadoop-common-2.2.0-tests.jar
> > > > > >
> > > > > > Ofcourse, I'll be running some tests do check that dtcli works
> > > properly
> > > > > for
> > > > > > launch+shutdown+kill of apps with only these libraries present in
> > > > > isolation
> > > > > > without dependency on local m2.
> > > > > > I'm believe that there are unwanted jars which are used for
> compile
> > > > time
> > > > > > dependency and not runtime in above list which I can drop to keep
> > > > package
> > > > > > size minimal.
> > > > > >
> > > > > >
> > > > > > @Anniruddha,
> > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required.
> > > Only
> > > > > > required values I see in it are classpath. and dtcli is capable
> of
> > > > > building
> > > > > > it on the fly.
> > > > > >
> > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the
> jar
> > > > path,
> > > > > > dt-env.sh and all such conf files. If we need a change in dtcli,
> > then
> > > > > > community's opinion is required for whether dtcli should change
> in
> > > our
> > > > > repo
> > > > > > or a copy of that with required changes exist in bigtop repo
> until
> > we
> > > > > make
> > > > > > the dtcli generic enough.
> > > > > >
> > > > >
> > > > > Can we have these file paths set using environment variables with
> > some
> > > > > default values in dtcli?
> > > > > Then we can set such params in dt-env.sh.
> > > > >
> > > > >
> > > > > > 3. dt-sited.xml -> How is this file different from dt-site.xml?
> Not
> > > > sure
> > > > > if
> > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel are
> > > copied
> > > > to
> > > > > > home of new user when useradd program is called. But for existing
> > > users
> > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the
> fly
> > > if
> > > > > not
> > > > > > encountered for the first time. Again correct me if I'm wrong.
> > > > > >
> > > > >
> > > > > Yes, you are right. However this would probably be necessary step
> for
> > > > > rpm/deb, as they may also get installed during OS install and
> before
> > > user
> > > > > accounts were created.
> > > > >
> > > > >
> > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best
> > idea.
> > > > > We'll
> > > > > > anyway put dtcli in location which is by default present in path
> > i.e.
> > > > > > /usr/bin.
> > > > >
> > > > > For other env variables specific to apex, I'll prefer to use
> > dt-env.sh
> > > > and
> > > > > > source it in dtcli rather than changing bashrc etc...
> > > > >
> > > > >
> > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner
> way
> > > to
> > > > > achieve the same.
> > > > >
> > > > >
> > > > > >
> > > > > Thanks,
> > > > > > Chinmay.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > > aniruddha@datatorrent.com> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > @Chinmay,
> > > > > > >
> > > > > > > We need to consider following:
> > > > > > > System wide default config files can be located at following
> > > > locations:
> > > > > > >
> > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on
> > how
> > > it
> > > > > > finds
> > > > > > > those locations)
> > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on
> > how
> > > > it
> > > > > > > finds those locations)
> > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users that
> > will
> > > > be
> > > > > > > created in system in future.)
> > > > > > >
> > > > > > > We may also have to modify bashrc / bashprofile for population
> > any
> > > > > > > variables that are required.
> > > > > > >
> > > > > > > @dev, please put in your comments / suggestion.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > >
> > > > > > > Aniruddha
> > > > > > >
> > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > > thomas@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > -->
> > > > > > > >
> > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > > chinmay@datatorrent.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Everyone!!
> > > > > > > > >
> > > > > > > > > Continuing with packaging effort (rpm+deb) of apex, here
> are
> > > some
> > > > > > > > proposals
> > > > > > > > > about package structure etc..
> > > > > > > > > Before posting it on bbigtop mailing list, I have some
> > question
> > > > for
> > > > > > > apex
> > > > > > > > > community.
> > > > > > > > >
> > > > > > > > > Proposed Directory structure of apex package for both deb &
> > > rpm:
> > > > > > > > >
> > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > > >
> > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > > <other dependent jars> = All the 3rd party jars which are
> > > > required
> > > > > > for
> > > > > > > > apex
> > > > > > > > > to run. Usually the dependencies are packaged as part of
> > > rpm/deb
> > > > by
> > > > > > any
> > > > > > > > > software in bigtop.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Can you specify what those jars are in Bigtop context? Same
> as
> > > > > shipped
> > > > > > > with
> > > > > > > > DT community addition under lib/ ?
> > > > > > > >
> > > > > > > >
> > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd party
> > jars
> > > > > which
> > > > > > > > needs
> > > > > > > > > to included while packaging.
> > > > > > > > >
> > > > > > > > > Questions related to this:
> > > > > > > > > 1. Should we call the cli of apex as "apex" instead of
> > "dtcli"
> > > in
> > > > > > > bigtop
> > > > > > > > > package?
> > > > > > > > >
> > > > > > > >
> > > > > > > > I think we should keep the name until we are able to change
> it
> > in
> > > > > Apex.
> > > > > > > > Otherwise this may get confusing.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 2. I see that all softwares in bigtop have man page for
> their
> > > > > > > > executables.
> > > > > > > > > I think we should have it too for dtcli. Is there any
> > > > documentation
> > > > > > > > which I
> > > > > > > > > can convert to man page? or can I use output of "dtcli
> > --help"?
> > > > > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > > > > > > 3.3.0-incubating?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Has to be -incubating
> > > > > > > >
> > > > > > > >
> > > > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1
> > > > version
> > > > > of
> > > > > > > > > bigtop hadoop? Any problems that we see with this
> dependency?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Should work without changed. I thought we certified against
> > > 2.7.0?
> > > > > > > >
> > > > > > > >
> > > > > > > > > 5. Following is the apache mirror from which bigtop will
> pick
> > > the
> > > > > > apex
> > > > > > > > > source for compilation and packaging. Please correct if
> > > > incorrect:
> > > > > > > > >
> > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by all the
> > > > > softwares
> > > > > > in
> > > > > > > > > bigtop.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Please share your opinion.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Chinmay.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > > > chinmay@datatorrent.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > > >
> > > > > > > > > > I've already started discussion on bigtop dev mailing
> list
> > > for
> > > > > > > > > > integration. Also created a Jira for the same. For this
> > > > > communities
> > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > > > thomas@datatorrent.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Chinmay,
> > > > > > > > > >>
> > > > > > > > > >> I don't see anything under prerequisites that would be a
> > > > > problem.
> > > > > > We
> > > > > > > > > >> looked
> > > > > > > > > >> at the ASF licencing compatibility as part of becoming
> an
> > > > > > incubator
> > > > > > > > > >> project.
> > > > > > > > > >>
> > > > > > > > > >> Please focus on the packaging during the next weeks.
> Since
> > > the
> > > > > > work
> > > > > > > > will
> > > > > > > > > >> be
> > > > > > > > > >> part of Bigtop, related discussions and JIRAs should
> also
> > be
> > > > > > there.
> > > > > > > > > >>
> > > > > > > > > >> Would be good to have the packaging in place by end Feb.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Thomas
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi All,
> > > > > > > > > >> >
> > > > > > > > > >> > We're planning a work on adding Apache Apex as a
> > component
> > > > to
> > > > > > > Apache
> > > > > > > > > >> > Bigtop.
> > > > > > > > > >> > Bigtop is the packaging system for the Apache big data
> > > > > > ecosystem.
> > > > > > > > > >> Several
> > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > > > >> >
> > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for the
> same:
> > > > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > > >> >
> > > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > > >> > Apache bigtop has some hard and soft expectation for
> new
> > > > > > > components
> > > > > > > > to
> > > > > > > > > >> get
> > > > > > > > > >> > integrated into Bigtop.
> > > > > > > > > >> > Here is the list of it:
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > > >> >
> > > > > > > > > >> > Most of them seems to be standard ASF based
> > requirements,
> > > > but
> > > > > > few
> > > > > > > > need
> > > > > > > > > >> to
> > > > > > > > > >> > be checked for:
> > > > > > > > > >> > 1. Software projects are expected to be Licensed under
> > > > Apache
> > > > > > > > License,
> > > > > > > > > >> > Version 2.0 (and their dependencies are expected to be
> > > > > > compatible
> > > > > > > > with
> > > > > > > > > >> this
> > > > > > > > > >> > license)
> > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > > > dependencies
> > > > > of
> > > > > > > > Apex
> > > > > > > > > >> are
> > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > > > verification
> > > > > > > check.
> > > > > > > > > >> > 2. Software projects are expected to be compatible
> with
> > > all
> > > > of
> > > > > > the
> > > > > > > > > >> > supported platforms that Bigtop distribution is
> > targeting
> > > > > > > > > >> >     - This needs verifying whether our software runs
> > fine
> > > in
> > > > > > > > centos-6
> > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> opensuse-13.2.
> > > > > > > > > >> > 3. What smoke tests that should be added for
> deployment.
> > > > > > > > > >> > 4. Identifying the test artifacts which goes beyond
> > smoke
> > > > test
> > > > > > > > > >> >     - These are basically the integration tests for
> > > > > verification
> > > > > > > > after
> > > > > > > > > >> the
> > > > > > > > > >> > deployment. This is a soft requirement, but aim is to
> > > > achieve
> > > > > > this
> > > > > > > > as
> > > > > > > > > >> well
> > > > > > > > > >> > or at least have explanation why not to include.
> > > > > > > > > >> >
> > > > > > > > > >> > If there are any from the link which explicitly needs
> to
> > > be
> > > > > > > checked
> > > > > > > > > >> other
> > > > > > > > > >> > than above 4, please let us know.
> > > > > > > > > >> >
> > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > > > >> > From one of the mail archive of Bigtop, it was learnt
> > that
> > > > the
> > > > > > > > bigtop
> > > > > > > > > >> > community want to see the addition of new components
> in
> > > > > phases.
> > > > > > > Here
> > > > > > > > > are
> > > > > > > > > >> > the phases:
> > > > > > > > > >> > 1. Packaging
> > > > > > > > > >> >     - This needs creating of package i.e. rpm & deb
> > files.
> > > > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER,
> NOTES
> > > etc
> > > > if
> > > > > > any
> > > > > > > > > >> needed.
> > > > > > > > > >> >     - Any documentation that need to be added to
> > > > distribution
> > > > > of
> > > > > > > our
> > > > > > > > > >> > software.
> > > > > > > > > >> >     - Any license information of dependencies required
> > to
> > > be
> > > > > > added
> > > > > > > > to
> > > > > > > > > >> > package
> > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > > configuration
> > > > > > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > > > > > >> > 4. Integration tests
> > > > > > > > > >> >     - For verification of deployments.
> > > > > > > > > >> > 5. license clearance:
> > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff is
> > > compliant
> > > > > > with
> > > > > > > > ASF
> > > > > > > > > >> > license requirements. If you add code licenses under
> > > > different
> > > > > > > > > licenses,
> > > > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > > > >> >
> > > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > > >> > We'll start corresponding communication on bigtop
> > mailing
> > > > list
> > > > > > as
> > > > > > > > > well.
> > > > > > > > > >> >
> > > > > > > > > >> > We have some specific questions/suggestions related to
> > > what
> > > > > > should
> > > > > > > > be
> > > > > > > > > >> the
> > > > > > > > > >> > content of the package and what should be the smoke
> > tests,
> > > > but
> > > > > > in
> > > > > > > > the
> > > > > > > > > >> > interest of not having too much content here, we'll
> put
> > > the
> > > > > > > > questions
> > > > > > > > > >> as a
> > > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks,
> > > > > > > > > >> > Chinmay.
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@datatorrent.com>]

Hi Aniruddha,

I like the idea of skeleton directory. But I don't see any use of it,
unless we've anything in default.
If you have something in your mind which should be default, can you please
share it?

Thanks,
Chinmay.


On Wed, Feb 10, 2016 at 9:49 PM, Aniruddha Thombare <
aniruddha@datatorrent.com> wrote:

> On point 2:
> Skeleton directory is for default home dir structure for future users.
> By avoiding that, we can't inconvenience users and admins.
> Defaults in that configuration can be specified or evaluated at install.
>
>
>
> On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <ch...@datatorrent.com>
> wrote:
>
> > Really good points Pradeep and Aniruddha.
> >
> > 1. I believe we won't need to change the dtcli considering it works with
> DT
> > Community edition. We can keep the directory structure similar to that.
> > dt-env.sh has variables which contains information required for dtcli to
> > launch.
> >
> > 2. Let me check with bigtop community that whether they facilitate the
> > installation of rpms/debs before user is created. In either case, current
> > dtcli creates a .dt folder in home directory. Also before putting
> anything
> > in /etc/skel we need to define what are the default contents that should
> go
> > to ~/.dt/ folder. If there is no defaults, probably we should not
> > explicitly add it in /etc/skel.
> >
> > 3.  /etc/profile.d approach looks nice. This way contents of dt-env.sh
> are
> > present as env variables. But I see a catch there. Adding dt-env.sh to
> > /etc/profile.d would make all the variables available at runtime all the
> > time. I feel a little skeptical about that. Maybe a possible collision
> can
> > occur with other application vars.
> > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > "~/.dt/dt-env.sh". So those variables are anyway available.
> >
> > @Aniruddha, the Jira and PR effort is happening at Bigtop
> Jira/repository.
> > Packaging related code usually goes there. (That's what all the
> components
> > in bigtop does).
> > Having said that, once a PR is created, I'll be sharing the link of PR
> > here, so that apex community as well can review it.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > aniruddha@datatorrent.com> wrote:
> >
> > > +1 on suggestions and approach.
> > > We may need to iron out details about exact paths etc.
> > > Which can be done on jira / PR comments.
> > > Is that right @dev?
> > >
> > >
> > >
> > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <ap...@pradeepdalvi.com>
> > > wrote:
> > >
> > > > Inline comments...
> > > >
> > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com>
> > > > wrote:
> > > >
> > > > > @Thomas, Not all the jar present in DT community edition will be
> > > included
> > > > > there.
> > > > >
> > > > > Here is the list of jars I found from common dependencies between
> of
> > > apex
> > > > > and DT Community edition:
> > > > > netlet-1.2.0.jar
> > > > > kryo-2.24.0.jar
> > > > > jackson-core-asl-1.9.2.jar
> > > > > jackson-mapper-asl-1.9.2.jar
> > > > > async-http-client-1.7.20.jar
> > > > > netty-3.6.6.Final.jar
> > > > > validation-api-1.1.0.Final.jar
> > > > > bval-jsr303-0.5.jar
> > > > > bval-core-0.5.jar
> > > > > commons-lang3-3.1.jar
> > > > > commons-beanutils-1.8.3.jar
> > > > > httpclient-4.3.5.jar
> > > > > commons-codec-1.10.jar
> > > > > zip4j-1.3.2.jar
> > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > xbean-asm5-shaded-4.3.jar
> > > > > jersey-apache-client4-1.9.jar
> > > > > jline-2.11.jar
> > > > > ant-1.9.2.jar
> > > > > ant-launcher-1.9.2.jar
> > > > > mbassador-1.1.9.jar
> > > > > jackson-jaxrs-1.9.2.jar
> > > > > jackson-xc-1.9.2.jar
> > > > > hadoop-common-2.2.0-tests.jar
> > > > >
> > > > > Ofcourse, I'll be running some tests do check that dtcli works
> > properly
> > > > for
> > > > > launch+shutdown+kill of apps with only these libraries present in
> > > > isolation
> > > > > without dependency on local m2.
> > > > > I'm believe that there are unwanted jars which are used for compile
> > > time
> > > > > dependency and not runtime in above list which I can drop to keep
> > > package
> > > > > size minimal.
> > > > >
> > > > >
> > > > > @Anniruddha,
> > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required.
> > Only
> > > > > required values I see in it are classpath. and dtcli is capable of
> > > > building
> > > > > it on the fly.
> > > > >
> > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar
> > > path,
> > > > > dt-env.sh and all such conf files. If we need a change in dtcli,
> then
> > > > > community's opinion is required for whether dtcli should change in
> > our
> > > > repo
> > > > > or a copy of that with required changes exist in bigtop repo until
> we
> > > > make
> > > > > the dtcli generic enough.
> > > > >
> > > >
> > > > Can we have these file paths set using environment variables with
> some
> > > > default values in dtcli?
> > > > Then we can set such params in dt-env.sh.
> > > >
> > > >
> > > > > 3. dt-sited.xml -> How is this file different from dt-site.xml? Not
> > > sure
> > > > if
> > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel are
> > copied
> > > to
> > > > > home of new user when useradd program is called. But for existing
> > users
> > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly
> > if
> > > > not
> > > > > encountered for the first time. Again correct me if I'm wrong.
> > > > >
> > > >
> > > > Yes, you are right. However this would probably be necessary step for
> > > > rpm/deb, as they may also get installed during OS install and before
> > user
> > > > accounts were created.
> > > >
> > > >
> > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best
> idea.
> > > > We'll
> > > > > anyway put dtcli in location which is by default present in path
> i.e.
> > > > > /usr/bin.
> > > >
> > > > For other env variables specific to apex, I'll prefer to use
> dt-env.sh
> > > and
> > > > > source it in dtcli rather than changing bashrc etc...
> > > >
> > > >
> > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way
> > to
> > > > achieve the same.
> > > >
> > > >
> > > > >
> > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > aniruddha@datatorrent.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > @Chinmay,
> > > > > >
> > > > > > We need to consider following:
> > > > > > System wide default config files can be located at following
> > > locations:
> > > > > >
> > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on
> how
> > it
> > > > > finds
> > > > > > those locations)
> > > > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on
> how
> > > it
> > > > > > finds those locations)
> > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users that
> will
> > > be
> > > > > > created in system in future.)
> > > > > >
> > > > > > We may also have to modify bashrc / bashprofile for population
> any
> > > > > > variables that are required.
> > > > > >
> > > > > > @dev, please put in your comments / suggestion.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > >
> > > > > > Aniruddha
> > > > > >
> > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > thomas@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > -->
> > > > > > >
> > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > chinmay@datatorrent.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Everyone!!
> > > > > > > >
> > > > > > > > Continuing with packaging effort (rpm+deb) of apex, here are
> > some
> > > > > > > proposals
> > > > > > > > about package structure etc..
> > > > > > > > Before posting it on bbigtop mailing list, I have some
> question
> > > for
> > > > > > apex
> > > > > > > > community.
> > > > > > > >
> > > > > > > > Proposed Directory structure of apex package for both deb &
> > rpm:
> > > > > > > >
> > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > >
> > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > >
> > > > > > > >
> > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > <other dependent jars> = All the 3rd party jars which are
> > > required
> > > > > for
> > > > > > > apex
> > > > > > > > to run. Usually the dependencies are packaged as part of
> > rpm/deb
> > > by
> > > > > any
> > > > > > > > software in bigtop.
> > > > > > > >
> > > > > > >
> > > > > > > Can you specify what those jars are in Bigtop context? Same as
> > > > shipped
> > > > > > with
> > > > > > > DT community addition under lib/ ?
> > > > > > >
> > > > > > >
> > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd party
> jars
> > > > which
> > > > > > > needs
> > > > > > > > to included while packaging.
> > > > > > > >
> > > > > > > > Questions related to this:
> > > > > > > > 1. Should we call the cli of apex as "apex" instead of
> "dtcli"
> > in
> > > > > > bigtop
> > > > > > > > package?
> > > > > > > >
> > > > > > >
> > > > > > > I think we should keep the name until we are able to change it
> in
> > > > Apex.
> > > > > > > Otherwise this may get confusing.
> > > > > > >
> > > > > > >
> > > > > > > > 2. I see that all softwares in bigtop have man page for their
> > > > > > > executables.
> > > > > > > > I think we should have it too for dtcli. Is there any
> > > documentation
> > > > > > > which I
> > > > > > > > can convert to man page? or can I use output of "dtcli
> --help"?
> > > > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > > > > > 3.3.0-incubating?
> > > > > > > >
> > > > > > >
> > > > > > > Has to be -incubating
> > > > > > >
> > > > > > >
> > > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1
> > > version
> > > > of
> > > > > > > > bigtop hadoop? Any problems that we see with this dependency?
> > > > > > > >
> > > > > > >
> > > > > > > Should work without changed. I thought we certified against
> > 2.7.0?
> > > > > > >
> > > > > > >
> > > > > > > > 5. Following is the apache mirror from which bigtop will pick
> > the
> > > > > apex
> > > > > > > > source for compilation and packaging. Please correct if
> > > incorrect:
> > > > > > > >
> http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > >     NOTE: apache.osuosl.org is the mirror used by all the
> > > > softwares
> > > > > in
> > > > > > > > bigtop.
> > > > > > > >
> > > > > > > >
> > > > > > > > Please share your opinion.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chinmay.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > > chinmay@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > >
> > > > > > > > > I've already started discussion on bigtop dev mailing list
> > for
> > > > > > > > > integration. Also created a Jira for the same. For this
> > > > communities
> > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > > thomas@datatorrent.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Chinmay,
> > > > > > > > >>
> > > > > > > > >> I don't see anything under prerequisites that would be a
> > > > problem.
> > > > > We
> > > > > > > > >> looked
> > > > > > > > >> at the ASF licencing compatibility as part of becoming an
> > > > > incubator
> > > > > > > > >> project.
> > > > > > > > >>
> > > > > > > > >> Please focus on the packaging during the next weeks. Since
> > the
> > > > > work
> > > > > > > will
> > > > > > > > >> be
> > > > > > > > >> part of Bigtop, related discussions and JIRAs should also
> be
> > > > > there.
> > > > > > > > >>
> > > > > > > > >> Would be good to have the packaging in place by end Feb.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Thomas
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi All,
> > > > > > > > >> >
> > > > > > > > >> > We're planning a work on adding Apache Apex as a
> component
> > > to
> > > > > > Apache
> > > > > > > > >> > Bigtop.
> > > > > > > > >> > Bigtop is the packaging system for the Apache big data
> > > > > ecosystem.
> > > > > > > > >> Several
> > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > > >> >
> > > > > > > > >> > Here is the tracking Jira task in APEXCORE for the same:
> > > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > >> >
> > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > >> > Apache bigtop has some hard and soft expectation for new
> > > > > > components
> > > > > > > to
> > > > > > > > >> get
> > > > > > > > >> > integrated into Bigtop.
> > > > > > > > >> > Here is the list of it:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > >> >
> > > > > > > > >> > Most of them seems to be standard ASF based
> requirements,
> > > but
> > > > > few
> > > > > > > need
> > > > > > > > >> to
> > > > > > > > >> > be checked for:
> > > > > > > > >> > 1. Software projects are expected to be Licensed under
> > > Apache
> > > > > > > License,
> > > > > > > > >> > Version 2.0 (and their dependencies are expected to be
> > > > > compatible
> > > > > > > with
> > > > > > > > >> this
> > > > > > > > >> > license)
> > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > > dependencies
> > > > of
> > > > > > > Apex
> > > > > > > > >> are
> > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > > verification
> > > > > > check.
> > > > > > > > >> > 2. Software projects are expected to be compatible with
> > all
> > > of
> > > > > the
> > > > > > > > >> > supported platforms that Bigtop distribution is
> targeting
> > > > > > > > >> >     - This needs verifying whether our software runs
> fine
> > in
> > > > > > > centos-6
> > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > > > > > > > >> > 3. What smoke tests that should be added for deployment.
> > > > > > > > >> > 4. Identifying the test artifacts which goes beyond
> smoke
> > > test
> > > > > > > > >> >     - These are basically the integration tests for
> > > > verification
> > > > > > > after
> > > > > > > > >> the
> > > > > > > > >> > deployment. This is a soft requirement, but aim is to
> > > achieve
> > > > > this
> > > > > > > as
> > > > > > > > >> well
> > > > > > > > >> > or at least have explanation why not to include.
> > > > > > > > >> >
> > > > > > > > >> > If there are any from the link which explicitly needs to
> > be
> > > > > > checked
> > > > > > > > >> other
> > > > > > > > >> > than above 4, please let us know.
> > > > > > > > >> >
> > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > > >> > From one of the mail archive of Bigtop, it was learnt
> that
> > > the
> > > > > > > bigtop
> > > > > > > > >> > community want to see the addition of new components in
> > > > phases.
> > > > > > Here
> > > > > > > > are
> > > > > > > > >> > the phases:
> > > > > > > > >> > 1. Packaging
> > > > > > > > >> >     - This needs creating of package i.e. rpm & deb
> files.
> > > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES
> > etc
> > > if
> > > > > any
> > > > > > > > >> needed.
> > > > > > > > >> >     - Any documentation that need to be added to
> > > distribution
> > > > of
> > > > > > our
> > > > > > > > >> > software.
> > > > > > > > >> >     - Any license information of dependencies required
> to
> > be
> > > > > added
> > > > > > > to
> > > > > > > > >> > package
> > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > configuration
> > > > > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > > > > >> > 4. Integration tests
> > > > > > > > >> >     - For verification of deployments.
> > > > > > > > >> > 5. license clearance:
> > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff is
> > compliant
> > > > > with
> > > > > > > ASF
> > > > > > > > >> > license requirements. If you add code licenses under
> > > different
> > > > > > > > licenses,
> > > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > > >> >
> > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > >> > We'll start corresponding communication on bigtop
> mailing
> > > list
> > > > > as
> > > > > > > > well.
> > > > > > > > >> >
> > > > > > > > >> > We have some specific questions/suggestions related to
> > what
> > > > > should
> > > > > > > be
> > > > > > > > >> the
> > > > > > > > >> > content of the package and what should be the smoke
> tests,
> > > but
> > > > > in
> > > > > > > the
> > > > > > > > >> > interest of not having too much content here, we'll put
> > the
> > > > > > > questions
> > > > > > > > >> as a
> > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Chinmay.
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Aniruddha Thombare <an...@datatorrent.com>]

On point 2:
Skeleton directory is for default home dir structure for future users.
By avoiding that, we can't inconvenience users and admins.
Defaults in that configuration can be specified or evaluated at install.



On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> Really good points Pradeep and Aniruddha.
>
> 1. I believe we won't need to change the dtcli considering it works with DT
> Community edition. We can keep the directory structure similar to that.
> dt-env.sh has variables which contains information required for dtcli to
> launch.
>
> 2. Let me check with bigtop community that whether they facilitate the
> installation of rpms/debs before user is created. In either case, current
> dtcli creates a .dt folder in home directory. Also before putting anything
> in /etc/skel we need to define what are the default contents that should go
> to ~/.dt/ folder. If there is no defaults, probably we should not
> explicitly add it in /etc/skel.
>
> 3.  /etc/profile.d approach looks nice. This way contents of dt-env.sh are
> present as env variables. But I see a catch there. Adding dt-env.sh to
> /etc/profile.d would make all the variables available at runtime all the
> time. I feel a little skeptical about that. Maybe a possible collision can
> occur with other application vars.
> Moreover, current dtcli does source "../conf/dt-env.sh" and
> "~/.dt/dt-env.sh". So those variables are anyway available.
>
> @Aniruddha, the Jira and PR effort is happening at Bigtop Jira/repository.
> Packaging related code usually goes there. (That's what all the components
> in bigtop does).
> Having said that, once a PR is created, I'll be sharing the link of PR
> here, so that apex community as well can review it.
>
> Thanks,
> Chinmay.
>
>
> On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> aniruddha@datatorrent.com> wrote:
>
> > +1 on suggestions and approach.
> > We may need to iron out details about exact paths etc.
> > Which can be done on jira / PR comments.
> > Is that right @dev?
> >
> >
> >
> > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <ap...@pradeepdalvi.com>
> > wrote:
> >
> > > Inline comments...
> > >
> > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com>
> > > wrote:
> > >
> > > > @Thomas, Not all the jar present in DT community edition will be
> > included
> > > > there.
> > > >
> > > > Here is the list of jars I found from common dependencies between of
> > apex
> > > > and DT Community edition:
> > > > netlet-1.2.0.jar
> > > > kryo-2.24.0.jar
> > > > jackson-core-asl-1.9.2.jar
> > > > jackson-mapper-asl-1.9.2.jar
> > > > async-http-client-1.7.20.jar
> > > > netty-3.6.6.Final.jar
> > > > validation-api-1.1.0.Final.jar
> > > > bval-jsr303-0.5.jar
> > > > bval-core-0.5.jar
> > > > commons-lang3-3.1.jar
> > > > commons-beanutils-1.8.3.jar
> > > > httpclient-4.3.5.jar
> > > > commons-codec-1.10.jar
> > > > zip4j-1.3.2.jar
> > > > jetty-websocket-8.1.10.v20130312.jar
> > > > xbean-asm5-shaded-4.3.jar
> > > > jersey-apache-client4-1.9.jar
> > > > jline-2.11.jar
> > > > ant-1.9.2.jar
> > > > ant-launcher-1.9.2.jar
> > > > mbassador-1.1.9.jar
> > > > jackson-jaxrs-1.9.2.jar
> > > > jackson-xc-1.9.2.jar
> > > > hadoop-common-2.2.0-tests.jar
> > > >
> > > > Ofcourse, I'll be running some tests do check that dtcli works
> properly
> > > for
> > > > launch+shutdown+kill of apps with only these libraries present in
> > > isolation
> > > > without dependency on local m2.
> > > > I'm believe that there are unwanted jars which are used for compile
> > time
> > > > dependency and not runtime in above list which I can drop to keep
> > package
> > > > size minimal.
> > > >
> > > >
> > > > @Anniruddha,
> > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required.
> Only
> > > > required values I see in it are classpath. and dtcli is capable of
> > > building
> > > > it on the fly.
> > > >
> > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar
> > path,
> > > > dt-env.sh and all such conf files. If we need a change in dtcli, then
> > > > community's opinion is required for whether dtcli should change in
> our
> > > repo
> > > > or a copy of that with required changes exist in bigtop repo until we
> > > make
> > > > the dtcli generic enough.
> > > >
> > >
> > > Can we have these file paths set using environment variables with some
> > > default values in dtcli?
> > > Then we can set such params in dt-env.sh.
> > >
> > >
> > > > 3. dt-sited.xml -> How is this file different from dt-site.xml? Not
> > sure
> > > if
> > > > adding in /etc/skel would help. Files & Dirs from /etc/skel are
> copied
> > to
> > > > home of new user when useradd program is called. But for existing
> users
> > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly
> if
> > > not
> > > > encountered for the first time. Again correct me if I'm wrong.
> > > >
> > >
> > > Yes, you are right. However this would probably be necessary step for
> > > rpm/deb, as they may also get installed during OS install and before
> user
> > > accounts were created.
> > >
> > >
> > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best idea.
> > > We'll
> > > > anyway put dtcli in location which is by default present in path i.e.
> > > > /usr/bin.
> > >
> > > For other env variables specific to apex, I'll prefer to use dt-env.sh
> > and
> > > > source it in dtcli rather than changing bashrc etc...
> > >
> > >
> > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way
> to
> > > achieve the same.
> > >
> > >
> > > >
> > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > >
> > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > aniruddha@datatorrent.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > @Chinmay,
> > > > >
> > > > > We need to consider following:
> > > > > System wide default config files can be located at following
> > locations:
> > > > >
> > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on how
> it
> > > > finds
> > > > > those locations)
> > > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on how
> > it
> > > > > finds those locations)
> > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users that will
> > be
> > > > > created in system in future.)
> > > > >
> > > > > We may also have to modify bashrc / bashprofile for population any
> > > > > variables that are required.
> > > > >
> > > > > @dev, please put in your comments / suggestion.
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > >
> > > > > Aniruddha
> > > > >
> > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > thomas@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > -->
> > > > > >
> > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > chinmay@datatorrent.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Everyone!!
> > > > > > >
> > > > > > > Continuing with packaging effort (rpm+deb) of apex, here are
> some
> > > > > > proposals
> > > > > > > about package structure etc..
> > > > > > > Before posting it on bbigtop mailing list, I have some question
> > for
> > > > > apex
> > > > > > > community.
> > > > > > >
> > > > > > > Proposed Directory structure of apex package for both deb &
> rpm:
> > > > > > >
> > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > >
> > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > >
> > > > > > >
> > > > > > > <version> = 3.3.0-incubating.
> > > > > > > <other dependent jars> = All the 3rd party jars which are
> > required
> > > > for
> > > > > > apex
> > > > > > > to run. Usually the dependencies are packaged as part of
> rpm/deb
> > by
> > > > any
> > > > > > > software in bigtop.
> > > > > > >
> > > > > >
> > > > > > Can you specify what those jars are in Bigtop context? Same as
> > > shipped
> > > > > with
> > > > > > DT community addition under lib/ ?
> > > > > >
> > > > > >
> > > > > > > <package LICENSE> = Licenses of corresponding of 3rd party jars
> > > which
> > > > > > needs
> > > > > > > to included while packaging.
> > > > > > >
> > > > > > > Questions related to this:
> > > > > > > 1. Should we call the cli of apex as "apex" instead of "dtcli"
> in
> > > > > bigtop
> > > > > > > package?
> > > > > > >
> > > > > >
> > > > > > I think we should keep the name until we are able to change it in
> > > Apex.
> > > > > > Otherwise this may get confusing.
> > > > > >
> > > > > >
> > > > > > > 2. I see that all softwares in bigtop have man page for their
> > > > > > executables.
> > > > > > > I think we should have it too for dtcli. Is there any
> > documentation
> > > > > > which I
> > > > > > > can convert to man page? or can I use output of "dtcli --help"?
> > > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > > > > 3.3.0-incubating?
> > > > > > >
> > > > > >
> > > > > > Has to be -incubating
> > > > > >
> > > > > >
> > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1
> > version
> > > of
> > > > > > > bigtop hadoop? Any problems that we see with this dependency?
> > > > > > >
> > > > > >
> > > > > > Should work without changed. I thought we certified against
> 2.7.0?
> > > > > >
> > > > > >
> > > > > > > 5. Following is the apache mirror from which bigtop will pick
> the
> > > > apex
> > > > > > > source for compilation and packaging. Please correct if
> > incorrect:
> > > > > > >     http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > >     NOTE: apache.osuosl.org is the mirror used by all the
> > > softwares
> > > > in
> > > > > > > bigtop.
> > > > > > >
> > > > > > >
> > > > > > > Please share your opinion.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Chinmay.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > chinmay@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > >
> > > > > > > > I've already started discussion on bigtop dev mailing list
> for
> > > > > > > > integration. Also created a Jira for the same. For this
> > > communities
> > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > thomas@datatorrent.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Chinmay,
> > > > > > > >>
> > > > > > > >> I don't see anything under prerequisites that would be a
> > > problem.
> > > > We
> > > > > > > >> looked
> > > > > > > >> at the ASF licencing compatibility as part of becoming an
> > > > incubator
> > > > > > > >> project.
> > > > > > > >>
> > > > > > > >> Please focus on the packaging during the next weeks. Since
> the
> > > > work
> > > > > > will
> > > > > > > >> be
> > > > > > > >> part of Bigtop, related discussions and JIRAs should also be
> > > > there.
> > > > > > > >>
> > > > > > > >> Would be good to have the packaging in place by end Feb.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Thomas
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > > >> chinmay@datatorrent.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Hi All,
> > > > > > > >> >
> > > > > > > >> > We're planning a work on adding Apache Apex as a component
> > to
> > > > > Apache
> > > > > > > >> > Bigtop.
> > > > > > > >> > Bigtop is the packaging system for the Apache big data
> > > > ecosystem.
> > > > > > > >> Several
> > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > >> >
> > > > > > > >> > Here is the tracking Jira task in APEXCORE for the same:
> > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > >> >
> > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > >> > Apache bigtop has some hard and soft expectation for new
> > > > > components
> > > > > > to
> > > > > > > >> get
> > > > > > > >> > integrated into Bigtop.
> > > > > > > >> > Here is the list of it:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > >> >
> > > > > > > >> > Most of them seems to be standard ASF based requirements,
> > but
> > > > few
> > > > > > need
> > > > > > > >> to
> > > > > > > >> > be checked for:
> > > > > > > >> > 1. Software projects are expected to be Licensed under
> > Apache
> > > > > > License,
> > > > > > > >> > Version 2.0 (and their dependencies are expected to be
> > > > compatible
> > > > > > with
> > > > > > > >> this
> > > > > > > >> > license)
> > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > dependencies
> > > of
> > > > > > Apex
> > > > > > > >> are
> > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > verification
> > > > > check.
> > > > > > > >> > 2. Software projects are expected to be compatible with
> all
> > of
> > > > the
> > > > > > > >> > supported platforms that Bigtop distribution is targeting
> > > > > > > >> >     - This needs verifying whether our software runs fine
> in
> > > > > > centos-6
> > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > > > > > > >> > 3. What smoke tests that should be added for deployment.
> > > > > > > >> > 4. Identifying the test artifacts which goes beyond smoke
> > test
> > > > > > > >> >     - These are basically the integration tests for
> > > verification
> > > > > > after
> > > > > > > >> the
> > > > > > > >> > deployment. This is a soft requirement, but aim is to
> > achieve
> > > > this
> > > > > > as
> > > > > > > >> well
> > > > > > > >> > or at least have explanation why not to include.
> > > > > > > >> >
> > > > > > > >> > If there are any from the link which explicitly needs to
> be
> > > > > checked
> > > > > > > >> other
> > > > > > > >> > than above 4, please let us know.
> > > > > > > >> >
> > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > >> > From one of the mail archive of Bigtop, it was learnt that
> > the
> > > > > > bigtop
> > > > > > > >> > community want to see the addition of new components in
> > > phases.
> > > > > Here
> > > > > > > are
> > > > > > > >> > the phases:
> > > > > > > >> > 1. Packaging
> > > > > > > >> >     - This needs creating of package i.e. rpm & deb files.
> > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES
> etc
> > if
> > > > any
> > > > > > > >> needed.
> > > > > > > >> >     - Any documentation that need to be added to
> > distribution
> > > of
> > > > > our
> > > > > > > >> > software.
> > > > > > > >> >     - Any license information of dependencies required to
> be
> > > > added
> > > > > > to
> > > > > > > >> > package
> > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > >> > 3. Puppet recipes for automatic deployment and
> configuration
> > > > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > > > >> > 4. Integration tests
> > > > > > > >> >     - For verification of deployments.
> > > > > > > >> > 5. license clearance:
> > > > > > > >> >     Run 'gradle rat' to make sure all new stuff is
> compliant
> > > > with
> > > > > > ASF
> > > > > > > >> > license requirements. If you add code licenses under
> > different
> > > > > > > licenses,
> > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > >> >
> > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > >> > We'll start corresponding communication on bigtop mailing
> > list
> > > > as
> > > > > > > well.
> > > > > > > >> >
> > > > > > > >> > We have some specific questions/suggestions related to
> what
> > > > should
> > > > > > be
> > > > > > > >> the
> > > > > > > >> > content of the package and what should be the smoke tests,
> > but
> > > > in
> > > > > > the
> > > > > > > >> > interest of not having too much content here, we'll put
> the
> > > > > > questions
> > > > > > > >> as a
> > > > > > > >> > separate mail in this mailthread.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Chinmay.
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Munagala Ramanath <ra...@datatorrent.com>]

The output from the "help" command from dtcli seems like  a good starting
point.

Ram

On Mon, Feb 15, 2016 at 7:23 AM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Hello Community,
>
> Most of the other integrations in bigtop have man page for their
> executable.
> Should we have a man page for apex too?
>
> If yes, then what should be the content of it?
> Please suggest.
>
> Thanks,
> Chinmay.
>
>
> On Mon, Feb 15, 2016 at 8:50 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Alright. Will not add it.
> >
> > On Mon, Feb 15, 2016 at 3:24 PM, Pradeep A. Dalvi <pr...@apache.org>
> wrote:
> >
> >> We may not need /usr/lib/apex/bin/dtcli (symlink to
> >> /usr/lib/apex/bin/apex), when there are /usr/bin/apex & /usr/bin/dtcli
> >> (which are in PATH).
> >>
> >> On Mon, Feb 15, 2016 at 10:54 AM, Chinmay Kolhatkar <chinmay@apache.org
> >
> >> wrote:
> >>
> >> > /usr/lib/apex/bin contains executables related to apex. dtcli, even
> >> though
> >> > its a symlink, for current users its still an executable. Hence for
> the
> >> > sake of completion, added dtcli link under /usr/lib/apex/bin.
> >> >
> >> > But if you think otherwise, please let me know.
> >> >
> >> > Thanks,
> >> > Chinmay.
> >> >
> >> >
> >> > On Mon, Feb 15, 2016 at 10:42 AM, Thomas Weise <
> thomas@datatorrent.com>
> >> > wrote:
> >> >
> >> > > Why the dtcli link under lib?
> >> > >
> >> > >
> >> > > On Sun, Feb 14, 2016 at 9:09 PM, Chinmay Kolhatkar <
> >> chinmay@apache.org>
> >> > > wrote:
> >> > >
> >> > > > Hmmm... I get your point. In development mode, it tries to find
> the
> >> > > hadoop
> >> > > > executable like following:
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> HADOOP_SEARCH_PATH="${HADOOP_PREFIX}/bin:${HADOOP_HOME}/bin:${PATH}:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:."
> >> > > >   export DT_HADOOP=`PATH=${HADOOP_SEARCH_PATH} && command -v
> hadoop
> >> > > > 2>/dev/null`
> >> > > >
> >> > > > I will check with bigtop community that what is the best way to
> >> ensure
> >> > > > during installation that hadoop binary will be present in above
> >> search
> >> > > path
> >> > > > search.
> >> > > >
> >> > > > Also, I checked as you suggested, i.e. removed dt-env.sh from
> >> package
> >> > and
> >> > > > tried to run the dtcli... It works!!!
> >> > > >
> >> > > > Thank you for pointing that out Thomas.
> >> > > >
> >> > > > Now I agree with you that, we can get rid of conf/dt-env.sh.. As
> >> this
> >> > was
> >> > > > the only file in conf/, unless someone has anything to include in
> >> conf/
> >> > > > folder, I'll remove that too then.
> >> > > >
> >> > > > Also, this would mean, only dt* that remains in package are dtcli
> >> links
> >> > > to
> >> > > > apex script added for backward compatibility.
> >> > > > i.e following:
> >> > > >
> >> > > > /
> >> > > > `-- usr
> >> > > >     |-- bin
> >> > > >     |   |-- apex -> ../lib/apex/bin/apex
> >> > > >     |   `-- dtcli -> ../lib/apex/bin/apex
> >> > > >     |-- lib
> >> > > >     |   `-- apex
> >> > > >     |       |-- bin
> >> > > >     |       |   |-- apex   // This is the real script
> >> > > >     |       |   `-- dtcli -> apex
> >> > > >
> >> > > >
> >> > > >
> >> > > > Thanks,
> >> > > > Chinmay.
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Feb 15, 2016 at 10:22 AM, Thomas Weise <
> >> thomas@datatorrent.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > I'm not convinced dt-env.sh is even needed. In development mode,
> >> the
> >> > > file
> >> > > > > doesn't exist but it will still find Hadoop.
> >> > > > >
> >> > > > > The absolute path to hadoop script was introduced to deal with
> the
> >> > > layout
> >> > > > > of specific distros. You will know whether that applies to
> Bigtop
> >> or
> >> > > not.
> >> > > > >
> >> > > > > Thomas
> >> > > > >
> >> > > > > On Sun, Feb 14, 2016 at 8:16 PM, Chinmay Kolhatkar <
> >> > chinmay@apache.org
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi Thomas,
> >> > > > > >
> >> > > > > > There are only 2 things that are required from dt-env.sh:
> >> > > > > > 1. DT_HADOOP variable which is exported by dt-env.sh This
> >> variable
> >> > > has
> >> > > > > > absolute path for hadoop binary
> >> > > > > > 2. A find_hadoop() method which find the path of hadoop
> binary.
> >> > > > > >
> >> > > > > > JAVA_HOME is not required for this.
> >> > > > > >
> >> > > > > > The idea is not to change the dtcli. Current dtcli (in apex
> >> > > > > > 3.3.0-incubating src code), sources dt-env.sh for above 2
> >> > > information.
> >> > > > > > Hence included dt-env.sh
> >> > > > > >
> >> > > > > > If the community agrees, we can make dtcli more generic so
> that
> >> in
> >> > > > future
> >> > > > > > packing of bigtop, dt-env.sh can be completely excluded OR at
> >> least
> >> > > > > change
> >> > > > > > the name to apex-env.sh.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Chinmay.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise <
> >> > > thomas@datatorrent.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Chinmay,
> >> > > > > > >
> >> > > > > > > Can you please specify what settings in dt-env.sh are
> required
> >> > in a
> >> > > > > > bigtop
> >> > > > > > > environment?
> >> > > > > > >
> >> > > > > > > Location of hadoop and JAVA_HOME should be provided by the
> >> stack
> >> > > > > already?
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > Thomas
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> >> > > > chinmay@apache.org
> >> > > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Here is a suggestion I have related to name change:
> >> > > > > > > >
> >> > > > > > > > 1. While packaging we change the name of "dtcli" script to
> >> > > "apex".
> >> > > > > This
> >> > > > > > > is
> >> > > > > > > > done so that in future when we change the name to "apex",
> >> users
> >> > > of
> >> > > > > > bigtop
> >> > > > > > > > apex don't have to transition much.
> >> > > > > > > > 2. We also keep a symlink named dtcli which points to apex
> >> > > script.
> >> > > > > This
> >> > > > > > > is
> >> > > > > > > > for backward compatibility.
> >> > > > > > > > 3. We still keep conf/dt-env.sh for first integration in
> >> > bigtop.
> >> > > > This
> >> > > > > > is
> >> > > > > > > in
> >> > > > > > > > the interest of not changing the content of dtcli or apex
> >> > script.
> >> > > > > > > > 4. In apex 3.4.0, we take care of all these naming related
> >> > > changes
> >> > > > > and
> >> > > > > > > > update bigtop repository later to remove references to
> dtcli
> >> > all
> >> > > > > > > together.
> >> > > > > > > >
> >> > > > > > > > Please share your thoughts on above approach.
> >> > > > > > > >
> >> > > > > > > > Also, please share what could be the man page content for
> >> > "apex".
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > Chinmay.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> >> > > > > chinmay@apache.org
> >> > > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Yes.. It'll be build from source tar downloaded from one
> >> of
> >> > the
> >> > > > > > apache
> >> > > > > > > > > mirror: apache.osuosl.org
> >> > > > > > > > >
> >> > > > > > > > > The reason for dt-env.sh is it's sourced from dtcli.
> This
> >> > > leaves
> >> > > > > us 3
> >> > > > > > > > > options:
> >> > > > > > > > > If we should not have any dt-env.sh, dtcli would need a
> >> > change
> >> > > > > while
> >> > > > > > > > > packaging.
> >> > > > > > > > >
> >> > > > > > > > > But if we want to use dtcli as it is, then we would need
> >> > > > dt-env.sh
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> >> > > > > > thomas@datatorrent.com>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Looks good overall, though there shouldn't be any
> >> dt-env.sh
> >> > > > > > > > >>
> >> > > > > > > > >> What will the bigtop package be built from, the source
> >> tar?
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> >> > > > > > > chinmay@apache.org>
> >> > > > > > > > >> wrote:
> >> > > > > > > > >>
> >> > > > > > > > >> > Hi Thomas,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks for the feedback.
> >> > > > > > > > >> >
> >> > > > > > > > >> > First, we're not changing the name anywhere. We'll
> >> follow
> >> > > what
> >> > > > > > > > >> currently is
> >> > > > > > > > >> > in source tarball for 3.3.0-incubating version.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Secondly, I've mentioned a directory structure below
> >> which
> >> > > is
> >> > > > > > inline
> >> > > > > > > > >> with
> >> > > > > > > > >> > other existing integrations with bigtop.
> >> > > > > > > > >> >
> >> > > > > > > > >> > The need for each file is as follows:
> >> > > > > > > > >> > 1. bin/dtcli -> CLI for apex picked from
> >> > > > > engine/src/main/scripts/
> >> > > > > > of
> >> > > > > > > > >> source
> >> > > > > > > > >> > code apex 3.3.0-incubating.
> >> > > > > > > > >> >
> >> > > > > > > > >> > 2. conf/dt-env.sh -> This is source from dtcli. This
> >> > > searches
> >> > > > > for
> >> > > > > > > > hadoop
> >> > > > > > > > >> > binary path and exports an env variable for dtcli to
> >> use
> >> > it.
> >> > > > > > > > >> >
> >> > > > > > > > >> > 3. lib/ -> Libraries for CLI. This list is extracted
> >> in a
> >> > > > > similar
> >> > > > > > > way
> >> > > > > > > > to
> >> > > > > > > > >> > how DT Community Edition finds dependency jars.
> >> > > > > > > > >> > Please note that this is the first iteration list of
> >> jars.
> >> > > I'm
> >> > > > > > > trying
> >> > > > > > > > to
> >> > > > > > > > >> > narrow this down to only those which are really
> >> required.
> >> > > > > > > > >> > The test that I'm running to check if dtcli runs fine
> >> with
> >> > > > given
> >> > > > > > > > >> dependency
> >> > > > > > > > >> > is to launch+shutdown+kill+status for pi demo.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Above is the directory structure which is required
> for
> >> CLI
> >> > > to
> >> > > > > > work.
> >> > > > > > > > >> > What aniruddha and pradeep has mentioned are some
> >> > additional
> >> > > > > files
> >> > > > > > > > which
> >> > > > > > > > >> > makes the like of administrator easier. For eg.
> >> /etc/skel
> >> > > > > > > > >> >
> >> > > > > > > > >> > Please let me know if this is inline with what you're
> >> > > > thinking.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks,
> >> > > > > > > > >> > Chinmay.
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >> > .
> >> > > > > > > > >> > |-- bin
> >> > > > > > > > >> > |   `-- dtcli
> >> > > > > > > > >> > |-- conf
> >> > > > > > > > >> > |   `-- dt-env.sh
> >> > > > > > > > >> > `-- lib
> >> > > > > > > > >> >     |-- ant-1.9.2.jar
> >> > > > > > > > >> >     |-- ant-launcher-1.9.2.jar
> >> > > > > > > > >> >     |-- apex-api-3.3.0-incubating.jar
> >> > > > > > > > >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> >> > > > > > > > >> >     |-- apex-common-3.3.0-incubating.jar
> >> > > > > > > > >> >     |-- apex-engine.jar
> >> > > > > > > > >> >     |-- apex-engine-tests.jar
> >> > > > > > > > >> >     |-- async-http-client-1.7.20.jar
> >> > > > > > > > >> >     |-- bval-core-0.5.jar
> >> > > > > > > > >> >     |-- bval-jsr303-0.5.jar
> >> > > > > > > > >> >     |-- commons-beanutils-1.8.3.jar
> >> > > > > > > > >> >     |-- commons-codec-1.10.jar
> >> > > > > > > > >> >     |-- commons-lang3-3.1.jar
> >> > > > > > > > >> >     |-- commons-logging-1.1.3.jar
> >> > > > > > > > >> >     |-- grizzly-http-servlet-2.1.2.jar
> >> > > > > > > > >> >     |-- hadoop-common-2.2.0-tests.jar
> >> > > > > > > > >> >     |-- httpclient-4.3.5.jar
> >> > > > > > > > >> >     |-- httpcore-4.3.2.jar
> >> > > > > > > > >> >     |-- jackson-core-asl-1.9.2.jar
> >> > > > > > > > >> >     |-- jackson-mapper-asl-1.9.2.jar
> >> > > > > > > > >> >     |-- javax.servlet-3.1.jar
> >> > > > > > > > >> >     |-- javax.servlet-api-3.0.1.jar
> >> > > > > > > > >> >     |-- jersey-apache-client4-1.9.jar
> >> > > > > > > > >> >     |-- jersey-client-1.9.jar
> >> > > > > > > > >> >     |-- jetty-http-8.1.10.v20130312.jar
> >> > > > > > > > >> >     |-- jetty-io-8.1.10.v20130312.jar
> >> > > > > > > > >> >     |-- jetty-util-8.1.10.v20130312.jar
> >> > > > > > > > >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> >> > > > > > > > >> >     |-- jline-2.11.jar
> >> > > > > > > > >> >     |-- kryo-2.24.0.jar
> >> > > > > > > > >> >     |-- mbassador-1.1.9.jar
> >> > > > > > > > >> >     |-- minlog-1.2.jar
> >> > > > > > > > >> >     |-- netlet-1.2.0.jar
> >> > > > > > > > >> >     |-- netty-3.6.6.Final.jar
> >> > > > > > > > >> >     |-- objenesis-2.1.jar
> >> > > > > > > > >> >     |-- validation-api-1.1.0.Final.jar
> >> > > > > > > > >> >     |-- xbean-asm5-shaded-4.3.jar
> >> > > > > > > > >> >     `-- zip4j-1.3.2.jar
> >> > > > > > > > >> >
> >> > > > > > > > >> > 3 directories, 40 files
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> >> > > > > > > thomas@datatorrent.com
> >> > > > > > > > >
> >> > > > > > > > >> > wrote:
> >> > > > > > > > >> >
> >> > > > > > > > >> > > Chinmay,
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > Before discussing where to put the files, let's
> make
> >> > sure
> >> > > > they
> >> > > > > > are
> >> > > > > > > > >> really
> >> > > > > > > > >> > > needed for the operation of the CLI. As for names,
> >> > > anything
> >> > > > > new
> >> > > > > > > > needs
> >> > > > > > > > >> to
> >> > > > > > > > >> > > reflect Apex in the name and should follow common
> >> > > > conventions,
> >> > > > > > > > >> especially
> >> > > > > > > > >> > > in Bigtop where there are many existing
> integrations
> >> to
> >> > > look
> >> > > > > at.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > Thanks,
> >> > > > > > > > >> > > Thomas
> >> > > > > > > > >> > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar
> <
> >> > > > > > > > >> > > chinmay@datatorrent.com>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > > Really good points Pradeep and Aniruddha.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > 1. I believe we won't need to change the dtcli
> >> > > considering
> >> > > > > it
> >> > > > > > > > works
> >> > > > > > > > >> > with
> >> > > > > > > > >> > > DT
> >> > > > > > > > >> > > > Community edition. We can keep the directory
> >> structure
> >> > > > > similar
> >> > > > > > > to
> >> > > > > > > > >> that.
> >> > > > > > > > >> > > > dt-env.sh has variables which contains
> information
> >> > > > required
> >> > > > > > for
> >> > > > > > > > >> dtcli
> >> > > > > > > > >> > to
> >> > > > > > > > >> > > > launch.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > 2. Let me check with bigtop community that
> whether
> >> > they
> >> > > > > > > facilitate
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > installation of rpms/debs before user is created.
> >> In
> >> > > > either
> >> > > > > > > case,
> >> > > > > > > > >> > current
> >> > > > > > > > >> > > > dtcli creates a .dt folder in home directory.
> Also
> >> > > before
> >> > > > > > > putting
> >> > > > > > > > >> > > anything
> >> > > > > > > > >> > > > in /etc/skel we need to define what are the
> default
> >> > > > contents
> >> > > > > > > that
> >> > > > > > > > >> > should
> >> > > > > > > > >> > > go
> >> > > > > > > > >> > > > to ~/.dt/ folder. If there is no defaults,
> >> probably we
> >> > > > > should
> >> > > > > > > not
> >> > > > > > > > >> > > > explicitly add it in /etc/skel.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > 3.  /etc/profile.d approach looks nice. This way
> >> > > contents
> >> > > > of
> >> > > > > > > > >> dt-env.sh
> >> > > > > > > > >> > > are
> >> > > > > > > > >> > > > present as env variables. But I see a catch
> there.
> >> > > Adding
> >> > > > > > > > dt-env.sh
> >> > > > > > > > >> to
> >> > > > > > > > >> > > > /etc/profile.d would make all the variables
> >> available
> >> > at
> >> > > > > > runtime
> >> > > > > > > > all
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > time. I feel a little skeptical about that.
> Maybe a
> >> > > > possible
> >> > > > > > > > >> collision
> >> > > > > > > > >> > > can
> >> > > > > > > > >> > > > occur with other application vars.
> >> > > > > > > > >> > > > Moreover, current dtcli does source
> >> > "../conf/dt-env.sh"
> >> > > > and
> >> > > > > > > > >> > > > "~/.dt/dt-env.sh". So those variables are anyway
> >> > > > available.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > @Aniruddha, the Jira and PR effort is happening
> at
> >> > > Bigtop
> >> > > > > > > > >> > > Jira/repository.
> >> > > > > > > > >> > > > Packaging related code usually goes there.
> (That's
> >> > what
> >> > > > all
> >> > > > > > the
> >> > > > > > > > >> > > components
> >> > > > > > > > >> > > > in bigtop does).
> >> > > > > > > > >> > > > Having said that, once a PR is created, I'll be
> >> > sharing
> >> > > > the
> >> > > > > > link
> >> > > > > > > > of
> >> > > > > > > > >> PR
> >> > > > > > > > >> > > > here, so that apex community as well can review
> it.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > Thanks,
> >> > > > > > > > >> > > > Chinmay.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha
> >> Thombare <
> >> > > > > > > > >> > > > aniruddha@datatorrent.com> wrote:
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > > +1 on suggestions and approach.
> >> > > > > > > > >> > > > > We may need to iron out details about exact
> paths
> >> > etc.
> >> > > > > > > > >> > > > > Which can be done on jira / PR comments.
> >> > > > > > > > >> > > > > Is that right @dev?
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> >> > > > > > > > >> > apache@pradeepdalvi.com>
> >> > > > > > > > >> > > > > wrote:
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > > Inline comments...
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
> >> > Kolhatkar <
> >> > > > > > > > >> > > > > > chinmay@datatorrent.com>
> >> > > > > > > > >> > > > > > wrote:
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > > @Thomas, Not all the jar present in DT
> >> community
> >> > > > > edition
> >> > > > > > > > will
> >> > > > > > > > >> be
> >> > > > > > > > >> > > > > included
> >> > > > > > > > >> > > > > > > there.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Here is the list of jars I found from
> common
> >> > > > > > dependencies
> >> > > > > > > > >> between
> >> > > > > > > > >> > > of
> >> > > > > > > > >> > > > > apex
> >> > > > > > > > >> > > > > > > and DT Community edition:
> >> > > > > > > > >> > > > > > > netlet-1.2.0.jar
> >> > > > > > > > >> > > > > > > kryo-2.24.0.jar
> >> > > > > > > > >> > > > > > > jackson-core-asl-1.9.2.jar
> >> > > > > > > > >> > > > > > > jackson-mapper-asl-1.9.2.jar
> >> > > > > > > > >> > > > > > > async-http-client-1.7.20.jar
> >> > > > > > > > >> > > > > > > netty-3.6.6.Final.jar
> >> > > > > > > > >> > > > > > > validation-api-1.1.0.Final.jar
> >> > > > > > > > >> > > > > > > bval-jsr303-0.5.jar
> >> > > > > > > > >> > > > > > > bval-core-0.5.jar
> >> > > > > > > > >> > > > > > > commons-lang3-3.1.jar
> >> > > > > > > > >> > > > > > > commons-beanutils-1.8.3.jar
> >> > > > > > > > >> > > > > > > httpclient-4.3.5.jar
> >> > > > > > > > >> > > > > > > commons-codec-1.10.jar
> >> > > > > > > > >> > > > > > > zip4j-1.3.2.jar
> >> > > > > > > > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> >> > > > > > > > >> > > > > > > xbean-asm5-shaded-4.3.jar
> >> > > > > > > > >> > > > > > > jersey-apache-client4-1.9.jar
> >> > > > > > > > >> > > > > > > jline-2.11.jar
> >> > > > > > > > >> > > > > > > ant-1.9.2.jar
> >> > > > > > > > >> > > > > > > ant-launcher-1.9.2.jar
> >> > > > > > > > >> > > > > > > mbassador-1.1.9.jar
> >> > > > > > > > >> > > > > > > jackson-jaxrs-1.9.2.jar
> >> > > > > > > > >> > > > > > > jackson-xc-1.9.2.jar
> >> > > > > > > > >> > > > > > > hadoop-common-2.2.0-tests.jar
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Ofcourse, I'll be running some tests do
> check
> >> > that
> >> > > > > dtcli
> >> > > > > > > > works
> >> > > > > > > > >> > > > properly
> >> > > > > > > > >> > > > > > for
> >> > > > > > > > >> > > > > > > launch+shutdown+kill of apps with only
> these
> >> > > > libraries
> >> > > > > > > > >> present in
> >> > > > > > > > >> > > > > > isolation
> >> > > > > > > > >> > > > > > > without dependency on local m2.
> >> > > > > > > > >> > > > > > > I'm believe that there are unwanted jars
> >> which
> >> > are
> >> > > > > used
> >> > > > > > > for
> >> > > > > > > > >> > compile
> >> > > > > > > > >> > > > > time
> >> > > > > > > > >> > > > > > > dependency and not runtime in above list
> >> which I
> >> > > can
> >> > > > > > drop
> >> > > > > > > to
> >> > > > > > > > >> keep
> >> > > > > > > > >> > > > > package
> >> > > > > > > > >> > > > > > > size minimal.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > @Anniruddha,
> >> > > > > > > > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
> >> > > dt-env.sh
> >> > > > > is
> >> > > > > > > > >> required.
> >> > > > > > > > >> > > > Only
> >> > > > > > > > >> > > > > > > required values I see in it are classpath.
> >> and
> >> > > dtcli
> >> > > > > is
> >> > > > > > > > >> capable
> >> > > > > > > > >> > of
> >> > > > > > > > >> > > > > > building
> >> > > > > > > > >> > > > > > > it on the fly.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see
> how
> >> > dtcli
> >> > > > can
> >> > > > > > > find
> >> > > > > > > > >> the
> >> > > > > > > > >> > jar
> >> > > > > > > > >> > > > > path,
> >> > > > > > > > >> > > > > > > dt-env.sh and all such conf files. If we
> >> need a
> >> > > > change
> >> > > > > > in
> >> > > > > > > > >> dtcli,
> >> > > > > > > > >> > > then
> >> > > > > > > > >> > > > > > > community's opinion is required for whether
> >> > dtcli
> >> > > > > should
> >> > > > > > > > >> change
> >> > > > > > > > >> > in
> >> > > > > > > > >> > > > our
> >> > > > > > > > >> > > > > > repo
> >> > > > > > > > >> > > > > > > or a copy of that with required changes
> >> exist in
> >> > > > > bigtop
> >> > > > > > > repo
> >> > > > > > > > >> > until
> >> > > > > > > > >> > > we
> >> > > > > > > > >> > > > > > make
> >> > > > > > > > >> > > > > > > the dtcli generic enough.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Can we have these file paths set using
> >> environment
> >> > > > > > variables
> >> > > > > > > > >> with
> >> > > > > > > > >> > > some
> >> > > > > > > > >> > > > > > default values in dtcli?
> >> > > > > > > > >> > > > > > Then we can set such params in dt-env.sh.
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > > 3. dt-sited.xml -> How is this file
> different
> >> > from
> >> > > > > > > > >> dt-site.xml?
> >> > > > > > > > >> > Not
> >> > > > > > > > >> > > > > sure
> >> > > > > > > > >> > > > > > if
> >> > > > > > > > >> > > > > > > adding in /etc/skel would help. Files &
> Dirs
> >> > from
> >> > > > > > > /etc/skel
> >> > > > > > > > >> are
> >> > > > > > > > >> > > > copied
> >> > > > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > > home of new user when useradd program is
> >> called.
> >> > > But
> >> > > > > for
> >> > > > > > > > >> existing
> >> > > > > > > > >> > > > users
> >> > > > > > > > >> > > > > > > this won't be of any use. Moreover, dtcli
> >> > creates
> >> > > > > ~/.dt/
> >> > > > > > > on
> >> > > > > > > > >> the
> >> > > > > > > > >> > fly
> >> > > > > > > > >> > > > if
> >> > > > > > > > >> > > > > > not
> >> > > > > > > > >> > > > > > > encountered for the first time. Again
> >> correct me
> >> > > if
> >> > > > > I'm
> >> > > > > > > > wrong.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Yes, you are right. However this would
> >> probably be
> >> > > > > > necessary
> >> > > > > > > > >> step
> >> > > > > > > > >> > for
> >> > > > > > > > >> > > > > > rpm/deb, as they may also get installed
> during
> >> OS
> >> > > > > install
> >> > > > > > > and
> >> > > > > > > > >> > before
> >> > > > > > > > >> > > > user
> >> > > > > > > > >> > > > > > accounts were created.
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm
> not
> >> > sure
> >> > > > its
> >> > > > > > the
> >> > > > > > > > best
> >> > > > > > > > >> > > idea.
> >> > > > > > > > >> > > > > > We'll
> >> > > > > > > > >> > > > > > > anyway put dtcli in location which is by
> >> default
> >> > > > > present
> >> > > > > > > in
> >> > > > > > > > >> path
> >> > > > > > > > >> > > i.e.
> >> > > > > > > > >> > > > > > > /usr/bin.
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > For other env variables specific to apex,
> I'll
> >> > > prefer
> >> > > > to
> >> > > > > > use
> >> > > > > > > > >> > > dt-env.sh
> >> > > > > > > > >> > > > > and
> >> > > > > > > > >> > > > > > > source it in dtcli rather than changing
> >> bashrc
> >> > > > etc...
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/
> >> shall
> >> > > be
> >> > > > > the
> >> > > > > > > > >> cleaner
> >> > > > > > > > >> > way
> >> > > > > > > > >> > > > to
> >> > > > > > > > >> > > > > > achieve the same.
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > Thanks,
> >> > > > > > > > >> > > > > > > Chinmay.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
> >> > > Thombare
> >> > > > <
> >> > > > > > > > >> > > > > > > aniruddha@datatorrent.com> wrote:
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > > Hi,
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > @Chinmay,
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > We need to consider following:
> >> > > > > > > > >> > > > > > > > System wide default config files can be
> >> > located
> >> > > at
> >> > > > > > > > following
> >> > > > > > > > >> > > > > locations:
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > /etc/apex/dt-env.sh  (we may have to
> change
> >> > > dtcli
> >> > > > > > > > behaviour
> >> > > > > > > > >> on
> >> > > > > > > > >> > > how
> >> > > > > > > > >> > > > it
> >> > > > > > > > >> > > > > > > finds
> >> > > > > > > > >> > > > > > > > those locations)
> >> > > > > > > > >> > > > > > > > /etc/apex/dt-site.xml (we may have to
> >> change
> >> > > dtcli
> >> > > > > > > > >> behaviour on
> >> > > > > > > > >> > > how
> >> > > > > > > > >> > > > > it
> >> > > > > > > > >> > > > > > > > finds those locations)
> >> > > > > > > > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other
> files
> >> > (for
> >> > > > new
> >> > > > > > > users
> >> > > > > > > > >> that
> >> > > > > > > > >> > > will
> >> > > > > > > > >> > > > > be
> >> > > > > > > > >> > > > > > > > created in system in future.)
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > We may also have to modify bashrc /
> >> > bashprofile
> >> > > > for
> >> > > > > > > > >> population
> >> > > > > > > > >> > > any
> >> > > > > > > > >> > > > > > > > variables that are required.
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > @dev, please put in your comments /
> >> > suggestion.
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > Thanks,
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > Aniruddha
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas
> >> Weise
> >> > <
> >> > > > > > > > >> > > > > thomas@datatorrent.com>
> >> > > > > > > > >> > > > > > > > wrote:
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > > -->
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> >> > > > Kolhatkar
> >> > > > > <
> >> > > > > > > > >> > > > > > > > chinmay@datatorrent.com
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > wrote:
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > Hello Everyone!!
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > Continuing with packaging effort
> >> (rpm+deb)
> >> > > of
> >> > > > > > apex,
> >> > > > > > > > here
> >> > > > > > > > >> > are
> >> > > > > > > > >> > > > some
> >> > > > > > > > >> > > > > > > > > proposals
> >> > > > > > > > >> > > > > > > > > > about package structure etc..
> >> > > > > > > > >> > > > > > > > > > Before posting it on bbigtop mailing
> >> > list, I
> >> > > > > have
> >> > > > > > > some
> >> > > > > > > > >> > > question
> >> > > > > > > > >> > > > > for
> >> > > > > > > > >> > > > > > > > apex
> >> > > > > > > > >> > > > > > > > > > community.
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > Proposed Directory structure of apex
> >> > package
> >> > > > for
> >> > > > > > > both
> >> > > > > > > > >> deb &
> >> > > > > > > > >> > > > rpm:
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> >> > > > > > > > >> > > > > > > > > >
> >> /usr/lib/apex/lib/apex-api-<version>.jar
> >> > > > > > > > >> > > > > > > > > >
> >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >> > > > > > > > >> > > > > > > > > >
> >> > /usr/lib/apex/lib/apex-common-<version>.jar
> >> > > > > > > > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent
> >> jars>
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > /usr/bin/dtcli ->
> /usr/lib/apex/bin/dtcli
> >> > > > > > > > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> >> > > > > > > > >> > > > > > > > > >
> /usr/share/doc/apex/license/LICENSE.txt
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > > > > > > > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> >> > > > > > > > >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > <version> = 3.3.0-incubating.
> >> > > > > > > > >> > > > > > > > > > <other dependent jars> = All the 3rd
> >> party
> >> > > > jars
> >> > > > > > > which
> >> > > > > > > > >> are
> >> > > > > > > > >> > > > > required
> >> > > > > > > > >> > > > > > > for
> >> > > > > > > > >> > > > > > > > > apex
> >> > > > > > > > >> > > > > > > > > > to run. Usually the dependencies are
> >> > > packaged
> >> > > > as
> >> > > > > > > part
> >> > > > > > > > of
> >> > > > > > > > >> > > > rpm/deb
> >> > > > > > > > >> > > > > by
> >> > > > > > > > >> > > > > > > any
> >> > > > > > > > >> > > > > > > > > > software in bigtop.
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Can you specify what those jars are in
> >> > Bigtop
> >> > > > > > context?
> >> > > > > > > > >> Same
> >> > > > > > > > >> > as
> >> > > > > > > > >> > > > > > shipped
> >> > > > > > > > >> > > > > > > > with
> >> > > > > > > > >> > > > > > > > > DT community addition under lib/ ?
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > <package LICENSE> = Licenses of
> >> > > corresponding
> >> > > > of
> >> > > > > > 3rd
> >> > > > > > > > >> party
> >> > > > > > > > >> > > jars
> >> > > > > > > > >> > > > > > which
> >> > > > > > > > >> > > > > > > > > needs
> >> > > > > > > > >> > > > > > > > > > to included while packaging.
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > Questions related to this:
> >> > > > > > > > >> > > > > > > > > > 1. Should we call the cli of apex as
> >> > "apex"
> >> > > > > > instead
> >> > > > > > > of
> >> > > > > > > > >> > > "dtcli"
> >> > > > > > > > >> > > > in
> >> > > > > > > > >> > > > > > > > bigtop
> >> > > > > > > > >> > > > > > > > > > package?
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > I think we should keep the name until
> we
> >> are
> >> > > > able
> >> > > > > to
> >> > > > > > > > >> change
> >> > > > > > > > >> > it
> >> > > > > > > > >> > > in
> >> > > > > > > > >> > > > > > Apex.
> >> > > > > > > > >> > > > > > > > > Otherwise this may get confusing.
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > 2. I see that all softwares in bigtop
> >> have
> >> > > man
> >> > > > > > page
> >> > > > > > > > for
> >> > > > > > > > >> > their
> >> > > > > > > > >> > > > > > > > > executables.
> >> > > > > > > > >> > > > > > > > > > I think we should have it too for
> >> dtcli.
> >> > Is
> >> > > > > there
> >> > > > > > > any
> >> > > > > > > > >> > > > > documentation
> >> > > > > > > > >> > > > > > > > > which I
> >> > > > > > > > >> > > > > > > > > > can convert to man page? or can I use
> >> > output
> >> > > > of
> >> > > > > > > "dtcli
> >> > > > > > > > >> > > --help"?
> >> > > > > > > > >> > > > > > > > > > 3. Do we want to call version of apex
> >> in
> >> > > > Bigtop
> >> > > > > as
> >> > > > > > > > >> 3.3.0 OR
> >> > > > > > > > >> > > > > > > > > > 3.3.0-incubating?
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Has to be -incubating
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > 4. Is it ok for apex package of
> bigtop
> >> to
> >> > > > depend
> >> > > > > > on
> >> > > > > > > > >> 2.7.1
> >> > > > > > > > >> > > > > version
> >> > > > > > > > >> > > > > > of
> >> > > > > > > > >> > > > > > > > > > bigtop hadoop? Any problems that we
> see
> >> > with
> >> > > > > this
> >> > > > > > > > >> > dependency?
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Should work without changed. I thought
> we
> >> > > > > certified
> >> > > > > > > > >> against
> >> > > > > > > > >> > > > 2.7.0?
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > 5. Following is the apache mirror
> from
> >> > which
> >> > > > > > bigtop
> >> > > > > > > > will
> >> > > > > > > > >> > pick
> >> > > > > > > > >> > > > the
> >> > > > > > > > >> > > > > > > apex
> >> > > > > > > > >> > > > > > > > > > source for compilation and packaging.
> >> > Please
> >> > > > > > correct
> >> > > > > > > > if
> >> > > > > > > > >> > > > > incorrect:
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > >
> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >> > > > > > > > >> > > > > > > > > >     NOTE: apache.osuosl.org is the
> >> mirror
> >> > > > used
> >> > > > > by
> >> > > > > > > all
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > > > softwares
> >> > > > > > > > >> > > > > > > in
> >> > > > > > > > >> > > > > > > > > > bigtop.
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > Please share your opinion.
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > Thanks,
> >> > > > > > > > >> > > > > > > > > > Chinmay.
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM,
> >> Chinmay
> >> > > > > > Kolhatkar <
> >> > > > > > > > >> > > > > > > > > > chinmay@datatorrent.com>
> >> > > > > > > > >> > > > > > > > > > wrote:
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > > Yes.. Starting to work on the
> >> packaging.
> >> > > > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > > I've already started discussion on
> >> > bigtop
> >> > > > dev
> >> > > > > > > > mailing
> >> > > > > > > > >> > list
> >> > > > > > > > >> > > > for
> >> > > > > > > > >> > > > > > > > > > > integration. Also created a Jira
> for
> >> the
> >> > > > same.
> >> > > > > > For
> >> > > > > > > > >> this
> >> > > > > > > > >> > > > > > communities
> >> > > > > > > > >> > > > > > > > > > > reference, here is the bigtop Jira:
> >> > > > > BIGTOP-2313.
> >> > > > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM,
> >> Thomas
> >> > > > Weise <
> >> > > > > > > > >> > > > > > > thomas@datatorrent.com
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > > wrote:
> >> > > > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > >> Chinmay,
> >> > > > > > > > >> > > > > > > > > > >>
> >> > > > > > > > >> > > > > > > > > > >> I don't see anything under
> >> > prerequisites
> >> > > > that
> >> > > > > > > would
> >> > > > > > > > >> be a
> >> > > > > > > > >> > > > > > problem.
> >> > > > > > > > >> > > > > > > We
> >> > > > > > > > >> > > > > > > > > > >> looked
> >> > > > > > > > >> > > > > > > > > > >> at the ASF licencing compatibility
> >> as
> >> > > part
> >> > > > of
> >> > > > > > > > >> becoming
> >> > > > > > > > >> > an
> >> > > > > > > > >> > > > > > > incubator
> >> > > > > > > > >> > > > > > > > > > >> project.
> >> > > > > > > > >> > > > > > > > > > >>
> >> > > > > > > > >> > > > > > > > > > >> Please focus on the packaging
> during
> >> > the
> >> > > > next
> >> > > > > > > > weeks.
> >> > > > > > > > >> > Since
> >> > > > > > > > >> > > > the
> >> > > > > > > > >> > > > > > > work
> >> > > > > > > > >> > > > > > > > > will
> >> > > > > > > > >> > > > > > > > > > >> be
> >> > > > > > > > >> > > > > > > > > > >> part of Bigtop, related
> discussions
> >> and
> >> > > > JIRAs
> >> > > > > > > > should
> >> > > > > > > > >> > also
> >> > > > > > > > >> > > be
> >> > > > > > > > >> > > > > > > there.
> >> > > > > > > > >> > > > > > > > > > >>
> >> > > > > > > > >> > > > > > > > > > >> Would be good to have the
> packaging
> >> in
> >> > > > place
> >> > > > > by
> >> > > > > > > end
> >> > > > > > > > >> Feb.
> >> > > > > > > > >> > > > > > > > > > >>
> >> > > > > > > > >> > > > > > > > > > >> Thanks,
> >> > > > > > > > >> > > > > > > > > > >> Thomas
> >> > > > > > > > >> > > > > > > > > > >>
> >> > > > > > > > >> > > > > > > > > > >>
> >> > > > > > > > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM,
> >> > Chinmay
> >> > > > > > > Kolhatkar
> >> > > > > > > > <
> >> > > > > > > > >> > > > > > > > > > >> chinmay@datatorrent.com>
> >> > > > > > > > >> > > > > > > > > > >> wrote:
> >> > > > > > > > >> > > > > > > > > > >>
> >> > > > > > > > >> > > > > > > > > > >> > Hi All,
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >> > We're planning a work on adding
> >> > Apache
> >> > > > Apex
> >> > > > > > as
> >> > > > > > > a
> >> > > > > > > > >> > > component
> >> > > > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > > > Apache
> >> > > > > > > > >> > > > > > > > > > >> > Bigtop.
> >> > > > > > > > >> > > > > > > > > > >> > Bigtop is the packaging system
> for
> >> > the
> >> > > > > Apache
> >> > > > > > > big
> >> > > > > > > > >> data
> >> > > > > > > > >> > > > > > > ecosystem.
> >> > > > > > > > >> > > > > > > > > > >> Several
> >> > > > > > > > >> > > > > > > > > > >> > Hadoop distros use it, most
> >> recently
> >> > > EMR.
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >> > Here is the tracking Jira task
> in
> >> > > > APEXCORE
> >> > > > > > for
> >> > > > > > > > the
> >> > > > > > > > >> > same:
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > https://issues.apache.org/jira/browse/APEXCORE-331
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >> > Proposed plan of execution is as
> >> > > follows:
> >> > > > > > > > >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> >> > > > > > > > >> > > > > > > > > > >> > Apache bigtop has some hard and
> >> soft
> >> > > > > > > expectation
> >> > > > > > > > >> for
> >> > > > > > > > >> > new
> >> > > > > > > > >> > > > > > > > components
> >> > > > > > > > >> > > > > > > > > to
> >> > > > > > > > >> > > > > > > > > > >> get
> >> > > > > > > > >> > > > > > > > > > >> > integrated into Bigtop.
> >> > > > > > > > >> > > > > > > > > > >> > Here is the list of it:
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >>
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >> > Most of them seems to be
> standard
> >> ASF
> >> > > > based
> >> > > > > > > > >> > > requirements,
> >> > > > > > > > >> > > > > but
> >> > > > > > > > >> > > > > > > few
> >> > > > > > > > >> > > > > > > > > need
> >> > > > > > > > >> > > > > > > > > > >> to
> >> > > > > > > > >> > > > > > > > > > >> > be checked for:
> >> > > > > > > > >> > > > > > > > > > >> > 1. Software projects are
> expected
> >> to
> >> > be
> >> > > > > > > Licensed
> >> > > > > > > > >> under
> >> > > > > > > > >> > > > > Apache
> >> > > > > > > > >> > > > > > > > > License,
> >> > > > > > > > >> > > > > > > > > > >> > Version 2.0 (and their
> >> dependencies
> >> > are
> >> > > > > > > expected
> >> > > > > > > > >> to be
> >> > > > > > > > >> > > > > > > compatible
> >> > > > > > > > >> > > > > > > > > with
> >> > > > > > > > >> > > > > > > > > > >> this
> >> > > > > > > > >> > > > > > > > > > >> > license)
> >> > > > > > > > >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but
> >> need
> >> > to
> >> > > > > check
> >> > > > > > > if
> >> > > > > > > > >> > > > > dependencies
> >> > > > > > > > >> > > > > > of
> >> > > > > > > > >> > > > > > > > > Apex
> >> > > > > > > > >> > > > > > > > > > >> are
> >> > > > > > > > >> > > > > > > > > > >> > compatible with ASL 2.0. This I
> >> guess
> >> > > > would
> >> > > > > > be
> >> > > > > > > a
> >> > > > > > > > >> > > > > verification
> >> > > > > > > > >> > > > > > > > check.
> >> > > > > > > > >> > > > > > > > > > >> > 2. Software projects are
> expected
> >> to
> >> > be
> >> > > > > > > > compatible
> >> > > > > > > > >> > with
> >> > > > > > > > >> > > > all
> >> > > > > > > > >> > > > > of
> >> > > > > > > > >> > > > > > > the
> >> > > > > > > > >> > > > > > > > > > >> > supported platforms that Bigtop
> >> > > > > distribution
> >> > > > > > is
> >> > > > > > > > >> > > targeting
> >> > > > > > > > >> > > > > > > > > > >> >     - This needs verifying
> whether
> >> > our
> >> > > > > > software
> >> > > > > > > > >> runs
> >> > > > > > > > >> > > fine
> >> > > > > > > > >> > > > in
> >> > > > > > > > >> > > > > > > > > centos-6
> >> > > > > > > > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
> >> > > debian-8
> >> > > > > > > > >> > opensuse-13.2.
> >> > > > > > > > >> > > > > > > > > > >> > 3. What smoke tests that should
> be
> >> > > added
> >> > > > > for
> >> > > > > > > > >> > deployment.
> >> > > > > > > > >> > > > > > > > > > >> > 4. Identifying the test
> artifacts
> >> > which
> >> > > > > goes
> >> > > > > > > > beyond
> >> > > > > > > > >> > > smoke
> >> > > > > > > > >> > > > > test
> >> > > > > > > > >> > > > > > > > > > >> >     - These are basically the
> >> > > integration
> >> > > > > > tests
> >> > > > > > > > for
> >> > > > > > > > >> > > > > > verification
> >> > > > > > > > >> > > > > > > > > after
> >> > > > > > > > >> > > > > > > > > > >> the
> >> > > > > > > > >> > > > > > > > > > >> > deployment. This is a soft
> >> > requirement,
> >> > > > but
> >> > > > > > aim
> >> > > > > > > > is
> >> > > > > > > > >> to
> >> > > > > > > > >> > > > > achieve
> >> > > > > > > > >> > > > > > > this
> >> > > > > > > > >> > > > > > > > > as
> >> > > > > > > > >> > > > > > > > > > >> well
> >> > > > > > > > >> > > > > > > > > > >> > or at least have explanation why
> >> not
> >> > to
> >> > > > > > > include.
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >> > If there are any from the link
> >> which
> >> > > > > > explicitly
> >> > > > > > > > >> needs
> >> > > > > > > > >> > to
> >> > > > > > > > >> > > > be
> >> > > > > > > > >> > > > > > > > checked
> >> > > > > > > > >> > > > > > > > > > >> other
> >> > > > > > > > >> > > > > > > > > > >> > than above 4, please let us
> know.
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >> > *Step 2) Adding Apex as
> component
> >> to
> >> > > > > Bigtop*
> >> > > > > > > > >> > > > > > > > > > >> > From one of the mail archive of
> >> > Bigtop,
> >> > > > it
> >> > > > > > was
> >> > > > > > > > >> learnt
> >> > > > > > > > >> > > that
> >> > > > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > > > > bigtop
> >> > > > > > > > >> > > > > > > > > > >> > community want to see the
> >> addition of
> >> > > new
> >> > > > > > > > >> components
> >> > > > > > > > >> > in
> >> > > > > > > > >> > > > > > phases.
> >> > > > > > > > >> > > > > > > > Here
> >> > > > > > > > >> > > > > > > > > > are
> >> > > > > > > > >> > > > > > > > > > >> > the phases:
> >> > > > > > > > >> > > > > > > > > > >> > 1. Packaging
> >> > > > > > > > >> > > > > > > > > > >> >     - This needs creating of
> >> package
> >> > > i.e.
> >> > > > > > rpm &
> >> > > > > > > > deb
> >> > > > > > > > >> > > files.
> >> > > > > > > > >> > > > > > > > > > >> >     - documentations/READMEs,
> >> > LICENSE,
> >> > > > > > > DISCLAMER,
> >> > > > > > > > >> > NOTES
> >> > > > > > > > >> > > > etc
> >> > > > > > > > >> > > > > if
> >> > > > > > > > >> > > > > > > any
> >> > > > > > > > >> > > > > > > > > > >> needed.
> >> > > > > > > > >> > > > > > > > > > >> >     - Any documentation that
> need
> >> to
> >> > be
> >> > > > > added
> >> > > > > > > to
> >> > > > > > > > >> > > > > distribution
> >> > > > > > > > >> > > > > > of
> >> > > > > > > > >> > > > > > > > our
> >> > > > > > > > >> > > > > > > > > > >> > software.
> >> > > > > > > > >> > > > > > > > > > >> >     - Any license information of
> >> > > > > dependencies
> >> > > > > > > > >> required
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > be
> >> > > > > > > > >> > > > > > > added
> >> > > > > > > > >> > > > > > > > > to
> >> > > > > > > > >> > > > > > > > > > >> > package
> >> > > > > > > > >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> >> > > > > > > > >> > > > > > > > > > >> >     - Adding smoke test for
> >> > packaging.
> >> > > > > > > > >> > > > > > > > > > >> > 3. Puppet recipes for automatic
> >> > > > deployment
> >> > > > > > and
> >> > > > > > > > >> > > > configuration
> >> > > > > > > > >> > > > > > > > > > >> >     - Add puppet recipes for
> >> > automatic
> >> > > > > > > deployment
> >> > > > > > > > >> > > > > > > > > > >> > 4. Integration tests
> >> > > > > > > > >> > > > > > > > > > >> >     - For verification of
> >> > deployments.
> >> > > > > > > > >> > > > > > > > > > >> > 5. license clearance:
> >> > > > > > > > >> > > > > > > > > > >> >     Run 'gradle rat' to make
> sure
> >> all
> >> > > new
> >> > > > > > stuff
> >> > > > > > > > is
> >> > > > > > > > >> > > > compliant
> >> > > > > > > > >> > > > > > > with
> >> > > > > > > > >> > > > > > > > > ASF
> >> > > > > > > > >> > > > > > > > > > >> > license requirements. If you add
> >> code
> >> > > > > > licenses
> >> > > > > > > > >> under
> >> > > > > > > > >> > > > > different
> >> > > > > > > > >> > > > > > > > > > licenses,
> >> > > > > > > > >> > > > > > > > > > >> > those would need to be listed in
> >> the
> >> > > > > NOTICE.
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >> > Please share your thoughts on
> the
> >> > > > approach.
> >> > > > > > > > >> > > > > > > > > > >> > We'll start corresponding
> >> > communication
> >> > > > on
> >> > > > > > > bigtop
> >> > > > > > > > >> > > mailing
> >> > > > > > > > >> > > > > list
> >> > > > > > > > >> > > > > > > as
> >> > > > > > > > >> > > > > > > > > > well.
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >> > We have some specific
> >> > > > questions/suggestions
> >> > > > > > > > >> related to
> >> > > > > > > > >> > > > what
> >> > > > > > > > >> > > > > > > should
> >> > > > > > > > >> > > > > > > > > be
> >> > > > > > > > >> > > > > > > > > > >> the
> >> > > > > > > > >> > > > > > > > > > >> > content of the package and what
> >> > should
> >> > > be
> >> > > > > the
> >> > > > > > > > smoke
> >> > > > > > > > >> > > tests,
> >> > > > > > > > >> > > > > but
> >> > > > > > > > >> > > > > > > in
> >> > > > > > > > >> > > > > > > > > the
> >> > > > > > > > >> > > > > > > > > > >> > interest of not having too much
> >> > content
> >> > > > > here,
> >> > > > > > > > we'll
> >> > > > > > > > >> > put
> >> > > > > > > > >> > > > the
> >> > > > > > > > >> > > > > > > > > questions
> >> > > > > > > > >> > > > > > > > > > >> as a
> >> > > > > > > > >> > > > > > > > > > >> > separate mail in this
> mailthread.
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >> > Thanks,
> >> > > > > > > > >> > > > > > > > > > >> > Chinmay.
> >> > > > > > > > >> > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > > > > > >>
> >> > > > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Hello Community,

Most of the other integrations in bigtop have man page for their executable.
Should we have a man page for apex too?

If yes, then what should be the content of it?
Please suggest.

Thanks,
Chinmay.


On Mon, Feb 15, 2016 at 8:50 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Alright. Will not add it.
>
> On Mon, Feb 15, 2016 at 3:24 PM, Pradeep A. Dalvi <pr...@apache.org> wrote:
>
>> We may not need /usr/lib/apex/bin/dtcli (symlink to
>> /usr/lib/apex/bin/apex), when there are /usr/bin/apex & /usr/bin/dtcli
>> (which are in PATH).
>>
>> On Mon, Feb 15, 2016 at 10:54 AM, Chinmay Kolhatkar <ch...@apache.org>
>> wrote:
>>
>> > /usr/lib/apex/bin contains executables related to apex. dtcli, even
>> though
>> > its a symlink, for current users its still an executable. Hence for the
>> > sake of completion, added dtcli link under /usr/lib/apex/bin.
>> >
>> > But if you think otherwise, please let me know.
>> >
>> > Thanks,
>> > Chinmay.
>> >
>> >
>> > On Mon, Feb 15, 2016 at 10:42 AM, Thomas Weise <th...@datatorrent.com>
>> > wrote:
>> >
>> > > Why the dtcli link under lib?
>> > >
>> > >
>> > > On Sun, Feb 14, 2016 at 9:09 PM, Chinmay Kolhatkar <
>> chinmay@apache.org>
>> > > wrote:
>> > >
>> > > > Hmmm... I get your point. In development mode, it tries to find the
>> > > hadoop
>> > > > executable like following:
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> >
>> HADOOP_SEARCH_PATH="${HADOOP_PREFIX}/bin:${HADOOP_HOME}/bin:${PATH}:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:."
>> > > >   export DT_HADOOP=`PATH=${HADOOP_SEARCH_PATH} && command -v hadoop
>> > > > 2>/dev/null`
>> > > >
>> > > > I will check with bigtop community that what is the best way to
>> ensure
>> > > > during installation that hadoop binary will be present in above
>> search
>> > > path
>> > > > search.
>> > > >
>> > > > Also, I checked as you suggested, i.e. removed dt-env.sh from
>> package
>> > and
>> > > > tried to run the dtcli... It works!!!
>> > > >
>> > > > Thank you for pointing that out Thomas.
>> > > >
>> > > > Now I agree with you that, we can get rid of conf/dt-env.sh.. As
>> this
>> > was
>> > > > the only file in conf/, unless someone has anything to include in
>> conf/
>> > > > folder, I'll remove that too then.
>> > > >
>> > > > Also, this would mean, only dt* that remains in package are dtcli
>> links
>> > > to
>> > > > apex script added for backward compatibility.
>> > > > i.e following:
>> > > >
>> > > > /
>> > > > `-- usr
>> > > >     |-- bin
>> > > >     |   |-- apex -> ../lib/apex/bin/apex
>> > > >     |   `-- dtcli -> ../lib/apex/bin/apex
>> > > >     |-- lib
>> > > >     |   `-- apex
>> > > >     |       |-- bin
>> > > >     |       |   |-- apex   // This is the real script
>> > > >     |       |   `-- dtcli -> apex
>> > > >
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Chinmay.
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Feb 15, 2016 at 10:22 AM, Thomas Weise <
>> thomas@datatorrent.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > I'm not convinced dt-env.sh is even needed. In development mode,
>> the
>> > > file
>> > > > > doesn't exist but it will still find Hadoop.
>> > > > >
>> > > > > The absolute path to hadoop script was introduced to deal with the
>> > > layout
>> > > > > of specific distros. You will know whether that applies to Bigtop
>> or
>> > > not.
>> > > > >
>> > > > > Thomas
>> > > > >
>> > > > > On Sun, Feb 14, 2016 at 8:16 PM, Chinmay Kolhatkar <
>> > chinmay@apache.org
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Thomas,
>> > > > > >
>> > > > > > There are only 2 things that are required from dt-env.sh:
>> > > > > > 1. DT_HADOOP variable which is exported by dt-env.sh This
>> variable
>> > > has
>> > > > > > absolute path for hadoop binary
>> > > > > > 2. A find_hadoop() method which find the path of hadoop binary.
>> > > > > >
>> > > > > > JAVA_HOME is not required for this.
>> > > > > >
>> > > > > > The idea is not to change the dtcli. Current dtcli (in apex
>> > > > > > 3.3.0-incubating src code), sources dt-env.sh for above 2
>> > > information.
>> > > > > > Hence included dt-env.sh
>> > > > > >
>> > > > > > If the community agrees, we can make dtcli more generic so that
>> in
>> > > > future
>> > > > > > packing of bigtop, dt-env.sh can be completely excluded OR at
>> least
>> > > > > change
>> > > > > > the name to apex-env.sh.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Chinmay.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise <
>> > > thomas@datatorrent.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Chinmay,
>> > > > > > >
>> > > > > > > Can you please specify what settings in dt-env.sh are required
>> > in a
>> > > > > > bigtop
>> > > > > > > environment?
>> > > > > > >
>> > > > > > > Location of hadoop and JAVA_HOME should be provided by the
>> stack
>> > > > > already?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Thomas
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
>> > > > chinmay@apache.org
>> > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Here is a suggestion I have related to name change:
>> > > > > > > >
>> > > > > > > > 1. While packaging we change the name of "dtcli" script to
>> > > "apex".
>> > > > > This
>> > > > > > > is
>> > > > > > > > done so that in future when we change the name to "apex",
>> users
>> > > of
>> > > > > > bigtop
>> > > > > > > > apex don't have to transition much.
>> > > > > > > > 2. We also keep a symlink named dtcli which points to apex
>> > > script.
>> > > > > This
>> > > > > > > is
>> > > > > > > > for backward compatibility.
>> > > > > > > > 3. We still keep conf/dt-env.sh for first integration in
>> > bigtop.
>> > > > This
>> > > > > > is
>> > > > > > > in
>> > > > > > > > the interest of not changing the content of dtcli or apex
>> > script.
>> > > > > > > > 4. In apex 3.4.0, we take care of all these naming related
>> > > changes
>> > > > > and
>> > > > > > > > update bigtop repository later to remove references to dtcli
>> > all
>> > > > > > > together.
>> > > > > > > >
>> > > > > > > > Please share your thoughts on above approach.
>> > > > > > > >
>> > > > > > > > Also, please share what could be the man page content for
>> > "apex".
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Chinmay.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
>> > > > > chinmay@apache.org
>> > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Yes.. It'll be build from source tar downloaded from one
>> of
>> > the
>> > > > > > apache
>> > > > > > > > > mirror: apache.osuosl.org
>> > > > > > > > >
>> > > > > > > > > The reason for dt-env.sh is it's sourced from dtcli. This
>> > > leaves
>> > > > > us 3
>> > > > > > > > > options:
>> > > > > > > > > If we should not have any dt-env.sh, dtcli would need a
>> > change
>> > > > > while
>> > > > > > > > > packaging.
>> > > > > > > > >
>> > > > > > > > > But if we want to use dtcli as it is, then we would need
>> > > > dt-env.sh
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
>> > > > > > thomas@datatorrent.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Looks good overall, though there shouldn't be any
>> dt-env.sh
>> > > > > > > > >>
>> > > > > > > > >> What will the bigtop package be built from, the source
>> tar?
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
>> > > > > > > chinmay@apache.org>
>> > > > > > > > >> wrote:
>> > > > > > > > >>
>> > > > > > > > >> > Hi Thomas,
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks for the feedback.
>> > > > > > > > >> >
>> > > > > > > > >> > First, we're not changing the name anywhere. We'll
>> follow
>> > > what
>> > > > > > > > >> currently is
>> > > > > > > > >> > in source tarball for 3.3.0-incubating version.
>> > > > > > > > >> >
>> > > > > > > > >> > Secondly, I've mentioned a directory structure below
>> which
>> > > is
>> > > > > > inline
>> > > > > > > > >> with
>> > > > > > > > >> > other existing integrations with bigtop.
>> > > > > > > > >> >
>> > > > > > > > >> > The need for each file is as follows:
>> > > > > > > > >> > 1. bin/dtcli -> CLI for apex picked from
>> > > > > engine/src/main/scripts/
>> > > > > > of
>> > > > > > > > >> source
>> > > > > > > > >> > code apex 3.3.0-incubating.
>> > > > > > > > >> >
>> > > > > > > > >> > 2. conf/dt-env.sh -> This is source from dtcli. This
>> > > searches
>> > > > > for
>> > > > > > > > hadoop
>> > > > > > > > >> > binary path and exports an env variable for dtcli to
>> use
>> > it.
>> > > > > > > > >> >
>> > > > > > > > >> > 3. lib/ -> Libraries for CLI. This list is extracted
>> in a
>> > > > > similar
>> > > > > > > way
>> > > > > > > > to
>> > > > > > > > >> > how DT Community Edition finds dependency jars.
>> > > > > > > > >> > Please note that this is the first iteration list of
>> jars.
>> > > I'm
>> > > > > > > trying
>> > > > > > > > to
>> > > > > > > > >> > narrow this down to only those which are really
>> required.
>> > > > > > > > >> > The test that I'm running to check if dtcli runs fine
>> with
>> > > > given
>> > > > > > > > >> dependency
>> > > > > > > > >> > is to launch+shutdown+kill+status for pi demo.
>> > > > > > > > >> >
>> > > > > > > > >> > Above is the directory structure which is required for
>> CLI
>> > > to
>> > > > > > work.
>> > > > > > > > >> > What aniruddha and pradeep has mentioned are some
>> > additional
>> > > > > files
>> > > > > > > > which
>> > > > > > > > >> > makes the like of administrator easier. For eg.
>> /etc/skel
>> > > > > > > > >> >
>> > > > > > > > >> > Please let me know if this is inline with what you're
>> > > > thinking.
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks,
>> > > > > > > > >> > Chinmay.
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> > .
>> > > > > > > > >> > |-- bin
>> > > > > > > > >> > |   `-- dtcli
>> > > > > > > > >> > |-- conf
>> > > > > > > > >> > |   `-- dt-env.sh
>> > > > > > > > >> > `-- lib
>> > > > > > > > >> >     |-- ant-1.9.2.jar
>> > > > > > > > >> >     |-- ant-launcher-1.9.2.jar
>> > > > > > > > >> >     |-- apex-api-3.3.0-incubating.jar
>> > > > > > > > >> >     |-- apex-bufferserver-3.3.0-incubating.jar
>> > > > > > > > >> >     |-- apex-common-3.3.0-incubating.jar
>> > > > > > > > >> >     |-- apex-engine.jar
>> > > > > > > > >> >     |-- apex-engine-tests.jar
>> > > > > > > > >> >     |-- async-http-client-1.7.20.jar
>> > > > > > > > >> >     |-- bval-core-0.5.jar
>> > > > > > > > >> >     |-- bval-jsr303-0.5.jar
>> > > > > > > > >> >     |-- commons-beanutils-1.8.3.jar
>> > > > > > > > >> >     |-- commons-codec-1.10.jar
>> > > > > > > > >> >     |-- commons-lang3-3.1.jar
>> > > > > > > > >> >     |-- commons-logging-1.1.3.jar
>> > > > > > > > >> >     |-- grizzly-http-servlet-2.1.2.jar
>> > > > > > > > >> >     |-- hadoop-common-2.2.0-tests.jar
>> > > > > > > > >> >     |-- httpclient-4.3.5.jar
>> > > > > > > > >> >     |-- httpcore-4.3.2.jar
>> > > > > > > > >> >     |-- jackson-core-asl-1.9.2.jar
>> > > > > > > > >> >     |-- jackson-mapper-asl-1.9.2.jar
>> > > > > > > > >> >     |-- javax.servlet-3.1.jar
>> > > > > > > > >> >     |-- javax.servlet-api-3.0.1.jar
>> > > > > > > > >> >     |-- jersey-apache-client4-1.9.jar
>> > > > > > > > >> >     |-- jersey-client-1.9.jar
>> > > > > > > > >> >     |-- jetty-http-8.1.10.v20130312.jar
>> > > > > > > > >> >     |-- jetty-io-8.1.10.v20130312.jar
>> > > > > > > > >> >     |-- jetty-util-8.1.10.v20130312.jar
>> > > > > > > > >> >     |-- jetty-websocket-8.1.10.v20130312.jar
>> > > > > > > > >> >     |-- jline-2.11.jar
>> > > > > > > > >> >     |-- kryo-2.24.0.jar
>> > > > > > > > >> >     |-- mbassador-1.1.9.jar
>> > > > > > > > >> >     |-- minlog-1.2.jar
>> > > > > > > > >> >     |-- netlet-1.2.0.jar
>> > > > > > > > >> >     |-- netty-3.6.6.Final.jar
>> > > > > > > > >> >     |-- objenesis-2.1.jar
>> > > > > > > > >> >     |-- validation-api-1.1.0.Final.jar
>> > > > > > > > >> >     |-- xbean-asm5-shaded-4.3.jar
>> > > > > > > > >> >     `-- zip4j-1.3.2.jar
>> > > > > > > > >> >
>> > > > > > > > >> > 3 directories, 40 files
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
>> > > > > > > thomas@datatorrent.com
>> > > > > > > > >
>> > > > > > > > >> > wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > > Chinmay,
>> > > > > > > > >> > >
>> > > > > > > > >> > > Before discussing where to put the files, let's make
>> > sure
>> > > > they
>> > > > > > are
>> > > > > > > > >> really
>> > > > > > > > >> > > needed for the operation of the CLI. As for names,
>> > > anything
>> > > > > new
>> > > > > > > > needs
>> > > > > > > > >> to
>> > > > > > > > >> > > reflect Apex in the name and should follow common
>> > > > conventions,
>> > > > > > > > >> especially
>> > > > > > > > >> > > in Bigtop where there are many existing integrations
>> to
>> > > look
>> > > > > at.
>> > > > > > > > >> > >
>> > > > > > > > >> > > Thanks,
>> > > > > > > > >> > > Thomas
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
>> > > > > > > > >> > > chinmay@datatorrent.com>
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > >
>> > > > > > > > >> > > > Really good points Pradeep and Aniruddha.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > 1. I believe we won't need to change the dtcli
>> > > considering
>> > > > > it
>> > > > > > > > works
>> > > > > > > > >> > with
>> > > > > > > > >> > > DT
>> > > > > > > > >> > > > Community edition. We can keep the directory
>> structure
>> > > > > similar
>> > > > > > > to
>> > > > > > > > >> that.
>> > > > > > > > >> > > > dt-env.sh has variables which contains information
>> > > > required
>> > > > > > for
>> > > > > > > > >> dtcli
>> > > > > > > > >> > to
>> > > > > > > > >> > > > launch.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > 2. Let me check with bigtop community that whether
>> > they
>> > > > > > > facilitate
>> > > > > > > > >> the
>> > > > > > > > >> > > > installation of rpms/debs before user is created.
>> In
>> > > > either
>> > > > > > > case,
>> > > > > > > > >> > current
>> > > > > > > > >> > > > dtcli creates a .dt folder in home directory. Also
>> > > before
>> > > > > > > putting
>> > > > > > > > >> > > anything
>> > > > > > > > >> > > > in /etc/skel we need to define what are the default
>> > > > contents
>> > > > > > > that
>> > > > > > > > >> > should
>> > > > > > > > >> > > go
>> > > > > > > > >> > > > to ~/.dt/ folder. If there is no defaults,
>> probably we
>> > > > > should
>> > > > > > > not
>> > > > > > > > >> > > > explicitly add it in /etc/skel.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > 3.  /etc/profile.d approach looks nice. This way
>> > > contents
>> > > > of
>> > > > > > > > >> dt-env.sh
>> > > > > > > > >> > > are
>> > > > > > > > >> > > > present as env variables. But I see a catch there.
>> > > Adding
>> > > > > > > > dt-env.sh
>> > > > > > > > >> to
>> > > > > > > > >> > > > /etc/profile.d would make all the variables
>> available
>> > at
>> > > > > > runtime
>> > > > > > > > all
>> > > > > > > > >> > the
>> > > > > > > > >> > > > time. I feel a little skeptical about that. Maybe a
>> > > > possible
>> > > > > > > > >> collision
>> > > > > > > > >> > > can
>> > > > > > > > >> > > > occur with other application vars.
>> > > > > > > > >> > > > Moreover, current dtcli does source
>> > "../conf/dt-env.sh"
>> > > > and
>> > > > > > > > >> > > > "~/.dt/dt-env.sh". So those variables are anyway
>> > > > available.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > @Aniruddha, the Jira and PR effort is happening at
>> > > Bigtop
>> > > > > > > > >> > > Jira/repository.
>> > > > > > > > >> > > > Packaging related code usually goes there. (That's
>> > what
>> > > > all
>> > > > > > the
>> > > > > > > > >> > > components
>> > > > > > > > >> > > > in bigtop does).
>> > > > > > > > >> > > > Having said that, once a PR is created, I'll be
>> > sharing
>> > > > the
>> > > > > > link
>> > > > > > > > of
>> > > > > > > > >> PR
>> > > > > > > > >> > > > here, so that apex community as well can review it.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > Thanks,
>> > > > > > > > >> > > > Chinmay.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha
>> Thombare <
>> > > > > > > > >> > > > aniruddha@datatorrent.com> wrote:
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > > +1 on suggestions and approach.
>> > > > > > > > >> > > > > We may need to iron out details about exact paths
>> > etc.
>> > > > > > > > >> > > > > Which can be done on jira / PR comments.
>> > > > > > > > >> > > > > Is that right @dev?
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
>> > > > > > > > >> > apache@pradeepdalvi.com>
>> > > > > > > > >> > > > > wrote:
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > > Inline comments...
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
>> > Kolhatkar <
>> > > > > > > > >> > > > > > chinmay@datatorrent.com>
>> > > > > > > > >> > > > > > wrote:
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > > @Thomas, Not all the jar present in DT
>> community
>> > > > > edition
>> > > > > > > > will
>> > > > > > > > >> be
>> > > > > > > > >> > > > > included
>> > > > > > > > >> > > > > > > there.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Here is the list of jars I found from common
>> > > > > > dependencies
>> > > > > > > > >> between
>> > > > > > > > >> > > of
>> > > > > > > > >> > > > > apex
>> > > > > > > > >> > > > > > > and DT Community edition:
>> > > > > > > > >> > > > > > > netlet-1.2.0.jar
>> > > > > > > > >> > > > > > > kryo-2.24.0.jar
>> > > > > > > > >> > > > > > > jackson-core-asl-1.9.2.jar
>> > > > > > > > >> > > > > > > jackson-mapper-asl-1.9.2.jar
>> > > > > > > > >> > > > > > > async-http-client-1.7.20.jar
>> > > > > > > > >> > > > > > > netty-3.6.6.Final.jar
>> > > > > > > > >> > > > > > > validation-api-1.1.0.Final.jar
>> > > > > > > > >> > > > > > > bval-jsr303-0.5.jar
>> > > > > > > > >> > > > > > > bval-core-0.5.jar
>> > > > > > > > >> > > > > > > commons-lang3-3.1.jar
>> > > > > > > > >> > > > > > > commons-beanutils-1.8.3.jar
>> > > > > > > > >> > > > > > > httpclient-4.3.5.jar
>> > > > > > > > >> > > > > > > commons-codec-1.10.jar
>> > > > > > > > >> > > > > > > zip4j-1.3.2.jar
>> > > > > > > > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
>> > > > > > > > >> > > > > > > xbean-asm5-shaded-4.3.jar
>> > > > > > > > >> > > > > > > jersey-apache-client4-1.9.jar
>> > > > > > > > >> > > > > > > jline-2.11.jar
>> > > > > > > > >> > > > > > > ant-1.9.2.jar
>> > > > > > > > >> > > > > > > ant-launcher-1.9.2.jar
>> > > > > > > > >> > > > > > > mbassador-1.1.9.jar
>> > > > > > > > >> > > > > > > jackson-jaxrs-1.9.2.jar
>> > > > > > > > >> > > > > > > jackson-xc-1.9.2.jar
>> > > > > > > > >> > > > > > > hadoop-common-2.2.0-tests.jar
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Ofcourse, I'll be running some tests do check
>> > that
>> > > > > dtcli
>> > > > > > > > works
>> > > > > > > > >> > > > properly
>> > > > > > > > >> > > > > > for
>> > > > > > > > >> > > > > > > launch+shutdown+kill of apps with only these
>> > > > libraries
>> > > > > > > > >> present in
>> > > > > > > > >> > > > > > isolation
>> > > > > > > > >> > > > > > > without dependency on local m2.
>> > > > > > > > >> > > > > > > I'm believe that there are unwanted jars
>> which
>> > are
>> > > > > used
>> > > > > > > for
>> > > > > > > > >> > compile
>> > > > > > > > >> > > > > time
>> > > > > > > > >> > > > > > > dependency and not runtime in above list
>> which I
>> > > can
>> > > > > > drop
>> > > > > > > to
>> > > > > > > > >> keep
>> > > > > > > > >> > > > > package
>> > > > > > > > >> > > > > > > size minimal.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > @Anniruddha,
>> > > > > > > > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
>> > > dt-env.sh
>> > > > > is
>> > > > > > > > >> required.
>> > > > > > > > >> > > > Only
>> > > > > > > > >> > > > > > > required values I see in it are classpath.
>> and
>> > > dtcli
>> > > > > is
>> > > > > > > > >> capable
>> > > > > > > > >> > of
>> > > > > > > > >> > > > > > building
>> > > > > > > > >> > > > > > > it on the fly.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how
>> > dtcli
>> > > > can
>> > > > > > > find
>> > > > > > > > >> the
>> > > > > > > > >> > jar
>> > > > > > > > >> > > > > path,
>> > > > > > > > >> > > > > > > dt-env.sh and all such conf files. If we
>> need a
>> > > > change
>> > > > > > in
>> > > > > > > > >> dtcli,
>> > > > > > > > >> > > then
>> > > > > > > > >> > > > > > > community's opinion is required for whether
>> > dtcli
>> > > > > should
>> > > > > > > > >> change
>> > > > > > > > >> > in
>> > > > > > > > >> > > > our
>> > > > > > > > >> > > > > > repo
>> > > > > > > > >> > > > > > > or a copy of that with required changes
>> exist in
>> > > > > bigtop
>> > > > > > > repo
>> > > > > > > > >> > until
>> > > > > > > > >> > > we
>> > > > > > > > >> > > > > > make
>> > > > > > > > >> > > > > > > the dtcli generic enough.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > Can we have these file paths set using
>> environment
>> > > > > > variables
>> > > > > > > > >> with
>> > > > > > > > >> > > some
>> > > > > > > > >> > > > > > default values in dtcli?
>> > > > > > > > >> > > > > > Then we can set such params in dt-env.sh.
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > > 3. dt-sited.xml -> How is this file different
>> > from
>> > > > > > > > >> dt-site.xml?
>> > > > > > > > >> > Not
>> > > > > > > > >> > > > > sure
>> > > > > > > > >> > > > > > if
>> > > > > > > > >> > > > > > > adding in /etc/skel would help. Files & Dirs
>> > from
>> > > > > > > /etc/skel
>> > > > > > > > >> are
>> > > > > > > > >> > > > copied
>> > > > > > > > >> > > > > to
>> > > > > > > > >> > > > > > > home of new user when useradd program is
>> called.
>> > > But
>> > > > > for
>> > > > > > > > >> existing
>> > > > > > > > >> > > > users
>> > > > > > > > >> > > > > > > this won't be of any use. Moreover, dtcli
>> > creates
>> > > > > ~/.dt/
>> > > > > > > on
>> > > > > > > > >> the
>> > > > > > > > >> > fly
>> > > > > > > > >> > > > if
>> > > > > > > > >> > > > > > not
>> > > > > > > > >> > > > > > > encountered for the first time. Again
>> correct me
>> > > if
>> > > > > I'm
>> > > > > > > > wrong.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > Yes, you are right. However this would
>> probably be
>> > > > > > necessary
>> > > > > > > > >> step
>> > > > > > > > >> > for
>> > > > > > > > >> > > > > > rpm/deb, as they may also get installed during
>> OS
>> > > > > install
>> > > > > > > and
>> > > > > > > > >> > before
>> > > > > > > > >> > > > user
>> > > > > > > > >> > > > > > accounts were created.
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not
>> > sure
>> > > > its
>> > > > > > the
>> > > > > > > > best
>> > > > > > > > >> > > idea.
>> > > > > > > > >> > > > > > We'll
>> > > > > > > > >> > > > > > > anyway put dtcli in location which is by
>> default
>> > > > > present
>> > > > > > > in
>> > > > > > > > >> path
>> > > > > > > > >> > > i.e.
>> > > > > > > > >> > > > > > > /usr/bin.
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > For other env variables specific to apex, I'll
>> > > prefer
>> > > > to
>> > > > > > use
>> > > > > > > > >> > > dt-env.sh
>> > > > > > > > >> > > > > and
>> > > > > > > > >> > > > > > > source it in dtcli rather than changing
>> bashrc
>> > > > etc...
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/
>> shall
>> > > be
>> > > > > the
>> > > > > > > > >> cleaner
>> > > > > > > > >> > way
>> > > > > > > > >> > > > to
>> > > > > > > > >> > > > > > achieve the same.
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > Thanks,
>> > > > > > > > >> > > > > > > Chinmay.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
>> > > Thombare
>> > > > <
>> > > > > > > > >> > > > > > > aniruddha@datatorrent.com> wrote:
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > > Hi,
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > @Chinmay,
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > We need to consider following:
>> > > > > > > > >> > > > > > > > System wide default config files can be
>> > located
>> > > at
>> > > > > > > > following
>> > > > > > > > >> > > > > locations:
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change
>> > > dtcli
>> > > > > > > > behaviour
>> > > > > > > > >> on
>> > > > > > > > >> > > how
>> > > > > > > > >> > > > it
>> > > > > > > > >> > > > > > > finds
>> > > > > > > > >> > > > > > > > those locations)
>> > > > > > > > >> > > > > > > > /etc/apex/dt-site.xml (we may have to
>> change
>> > > dtcli
>> > > > > > > > >> behaviour on
>> > > > > > > > >> > > how
>> > > > > > > > >> > > > > it
>> > > > > > > > >> > > > > > > > finds those locations)
>> > > > > > > > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files
>> > (for
>> > > > new
>> > > > > > > users
>> > > > > > > > >> that
>> > > > > > > > >> > > will
>> > > > > > > > >> > > > > be
>> > > > > > > > >> > > > > > > > created in system in future.)
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > We may also have to modify bashrc /
>> > bashprofile
>> > > > for
>> > > > > > > > >> population
>> > > > > > > > >> > > any
>> > > > > > > > >> > > > > > > > variables that are required.
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > @dev, please put in your comments /
>> > suggestion.
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > Thanks,
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > Aniruddha
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas
>> Weise
>> > <
>> > > > > > > > >> > > > > thomas@datatorrent.com>
>> > > > > > > > >> > > > > > > > wrote:
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > > -->
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
>> > > > Kolhatkar
>> > > > > <
>> > > > > > > > >> > > > > > > > chinmay@datatorrent.com
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > wrote:
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > > Hello Everyone!!
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > Continuing with packaging effort
>> (rpm+deb)
>> > > of
>> > > > > > apex,
>> > > > > > > > here
>> > > > > > > > >> > are
>> > > > > > > > >> > > > some
>> > > > > > > > >> > > > > > > > > proposals
>> > > > > > > > >> > > > > > > > > > about package structure etc..
>> > > > > > > > >> > > > > > > > > > Before posting it on bbigtop mailing
>> > list, I
>> > > > > have
>> > > > > > > some
>> > > > > > > > >> > > question
>> > > > > > > > >> > > > > for
>> > > > > > > > >> > > > > > > > apex
>> > > > > > > > >> > > > > > > > > > community.
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > Proposed Directory structure of apex
>> > package
>> > > > for
>> > > > > > > both
>> > > > > > > > >> deb &
>> > > > > > > > >> > > > rpm:
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > /usr/lib/apex/bin/dtcli
>> > > > > > > > >> > > > > > > > > >
>> /usr/lib/apex/lib/apex-api-<version>.jar
>> > > > > > > > >> > > > > > > > > >
>> > /usr/lib/apex/lib/apex-engine-<version>.jar
>> > > > > > > > >> > > > > > > > > >
>> > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
>> > > > > > > > >> > > > > > > > > >
>> > /usr/lib/apex/lib/apex-common-<version>.jar
>> > > > > > > > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent
>> jars>
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
>> > > > > > > > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
>> > > > > > > > >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
>> > > > > > > > >> > > > > > > > > >
>> > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
>> > > > > > > > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
>> > > > > > > > >> > > > > > > > > > /usr/share/doc/apex/NOTICE
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > <version> = 3.3.0-incubating.
>> > > > > > > > >> > > > > > > > > > <other dependent jars> = All the 3rd
>> party
>> > > > jars
>> > > > > > > which
>> > > > > > > > >> are
>> > > > > > > > >> > > > > required
>> > > > > > > > >> > > > > > > for
>> > > > > > > > >> > > > > > > > > apex
>> > > > > > > > >> > > > > > > > > > to run. Usually the dependencies are
>> > > packaged
>> > > > as
>> > > > > > > part
>> > > > > > > > of
>> > > > > > > > >> > > > rpm/deb
>> > > > > > > > >> > > > > by
>> > > > > > > > >> > > > > > > any
>> > > > > > > > >> > > > > > > > > > software in bigtop.
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Can you specify what those jars are in
>> > Bigtop
>> > > > > > context?
>> > > > > > > > >> Same
>> > > > > > > > >> > as
>> > > > > > > > >> > > > > > shipped
>> > > > > > > > >> > > > > > > > with
>> > > > > > > > >> > > > > > > > > DT community addition under lib/ ?
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > > <package LICENSE> = Licenses of
>> > > corresponding
>> > > > of
>> > > > > > 3rd
>> > > > > > > > >> party
>> > > > > > > > >> > > jars
>> > > > > > > > >> > > > > > which
>> > > > > > > > >> > > > > > > > > needs
>> > > > > > > > >> > > > > > > > > > to included while packaging.
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > Questions related to this:
>> > > > > > > > >> > > > > > > > > > 1. Should we call the cli of apex as
>> > "apex"
>> > > > > > instead
>> > > > > > > of
>> > > > > > > > >> > > "dtcli"
>> > > > > > > > >> > > > in
>> > > > > > > > >> > > > > > > > bigtop
>> > > > > > > > >> > > > > > > > > > package?
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > I think we should keep the name until we
>> are
>> > > > able
>> > > > > to
>> > > > > > > > >> change
>> > > > > > > > >> > it
>> > > > > > > > >> > > in
>> > > > > > > > >> > > > > > Apex.
>> > > > > > > > >> > > > > > > > > Otherwise this may get confusing.
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > > 2. I see that all softwares in bigtop
>> have
>> > > man
>> > > > > > page
>> > > > > > > > for
>> > > > > > > > >> > their
>> > > > > > > > >> > > > > > > > > executables.
>> > > > > > > > >> > > > > > > > > > I think we should have it too for
>> dtcli.
>> > Is
>> > > > > there
>> > > > > > > any
>> > > > > > > > >> > > > > documentation
>> > > > > > > > >> > > > > > > > > which I
>> > > > > > > > >> > > > > > > > > > can convert to man page? or can I use
>> > output
>> > > > of
>> > > > > > > "dtcli
>> > > > > > > > >> > > --help"?
>> > > > > > > > >> > > > > > > > > > 3. Do we want to call version of apex
>> in
>> > > > Bigtop
>> > > > > as
>> > > > > > > > >> 3.3.0 OR
>> > > > > > > > >> > > > > > > > > > 3.3.0-incubating?
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Has to be -incubating
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > > 4. Is it ok for apex package of bigtop
>> to
>> > > > depend
>> > > > > > on
>> > > > > > > > >> 2.7.1
>> > > > > > > > >> > > > > version
>> > > > > > > > >> > > > > > of
>> > > > > > > > >> > > > > > > > > > bigtop hadoop? Any problems that we see
>> > with
>> > > > > this
>> > > > > > > > >> > dependency?
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Should work without changed. I thought we
>> > > > > certified
>> > > > > > > > >> against
>> > > > > > > > >> > > > 2.7.0?
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > > 5. Following is the apache mirror from
>> > which
>> > > > > > bigtop
>> > > > > > > > will
>> > > > > > > > >> > pick
>> > > > > > > > >> > > > the
>> > > > > > > > >> > > > > > > apex
>> > > > > > > > >> > > > > > > > > > source for compilation and packaging.
>> > Please
>> > > > > > correct
>> > > > > > > > if
>> > > > > > > > >> > > > > incorrect:
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > >
>> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>> > > > > > > > >> > > > > > > > > >     NOTE: apache.osuosl.org is the
>> mirror
>> > > > used
>> > > > > by
>> > > > > > > all
>> > > > > > > > >> the
>> > > > > > > > >> > > > > > softwares
>> > > > > > > > >> > > > > > > in
>> > > > > > > > >> > > > > > > > > > bigtop.
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > Please share your opinion.
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > Thanks,
>> > > > > > > > >> > > > > > > > > > Chinmay.
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM,
>> Chinmay
>> > > > > > Kolhatkar <
>> > > > > > > > >> > > > > > > > > > chinmay@datatorrent.com>
>> > > > > > > > >> > > > > > > > > > wrote:
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > > Yes.. Starting to work on the
>> packaging.
>> > > > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > > I've already started discussion on
>> > bigtop
>> > > > dev
>> > > > > > > > mailing
>> > > > > > > > >> > list
>> > > > > > > > >> > > > for
>> > > > > > > > >> > > > > > > > > > > integration. Also created a Jira for
>> the
>> > > > same.
>> > > > > > For
>> > > > > > > > >> this
>> > > > > > > > >> > > > > > communities
>> > > > > > > > >> > > > > > > > > > > reference, here is the bigtop Jira:
>> > > > > BIGTOP-2313.
>> > > > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM,
>> Thomas
>> > > > Weise <
>> > > > > > > > >> > > > > > > thomas@datatorrent.com
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > > > wrote:
>> > > > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > >> Chinmay,
>> > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > >> > > > > > > > > > >> I don't see anything under
>> > prerequisites
>> > > > that
>> > > > > > > would
>> > > > > > > > >> be a
>> > > > > > > > >> > > > > > problem.
>> > > > > > > > >> > > > > > > We
>> > > > > > > > >> > > > > > > > > > >> looked
>> > > > > > > > >> > > > > > > > > > >> at the ASF licencing compatibility
>> as
>> > > part
>> > > > of
>> > > > > > > > >> becoming
>> > > > > > > > >> > an
>> > > > > > > > >> > > > > > > incubator
>> > > > > > > > >> > > > > > > > > > >> project.
>> > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > >> > > > > > > > > > >> Please focus on the packaging during
>> > the
>> > > > next
>> > > > > > > > weeks.
>> > > > > > > > >> > Since
>> > > > > > > > >> > > > the
>> > > > > > > > >> > > > > > > work
>> > > > > > > > >> > > > > > > > > will
>> > > > > > > > >> > > > > > > > > > >> be
>> > > > > > > > >> > > > > > > > > > >> part of Bigtop, related discussions
>> and
>> > > > JIRAs
>> > > > > > > > should
>> > > > > > > > >> > also
>> > > > > > > > >> > > be
>> > > > > > > > >> > > > > > > there.
>> > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > >> > > > > > > > > > >> Would be good to have the packaging
>> in
>> > > > place
>> > > > > by
>> > > > > > > end
>> > > > > > > > >> Feb.
>> > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > >> > > > > > > > > > >> Thanks,
>> > > > > > > > >> > > > > > > > > > >> Thomas
>> > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM,
>> > Chinmay
>> > > > > > > Kolhatkar
>> > > > > > > > <
>> > > > > > > > >> > > > > > > > > > >> chinmay@datatorrent.com>
>> > > > > > > > >> > > > > > > > > > >> wrote:
>> > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > >> > > > > > > > > > >> > Hi All,
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >> > We're planning a work on adding
>> > Apache
>> > > > Apex
>> > > > > > as
>> > > > > > > a
>> > > > > > > > >> > > component
>> > > > > > > > >> > > > > to
>> > > > > > > > >> > > > > > > > Apache
>> > > > > > > > >> > > > > > > > > > >> > Bigtop.
>> > > > > > > > >> > > > > > > > > > >> > Bigtop is the packaging system for
>> > the
>> > > > > Apache
>> > > > > > > big
>> > > > > > > > >> data
>> > > > > > > > >> > > > > > > ecosystem.
>> > > > > > > > >> > > > > > > > > > >> Several
>> > > > > > > > >> > > > > > > > > > >> > Hadoop distros use it, most
>> recently
>> > > EMR.
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >> > Here is the tracking Jira task in
>> > > > APEXCORE
>> > > > > > for
>> > > > > > > > the
>> > > > > > > > >> > same:
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > https://issues.apache.org/jira/browse/APEXCORE-331
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >> > Proposed plan of execution is as
>> > > follows:
>> > > > > > > > >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
>> > > > > > > > >> > > > > > > > > > >> > Apache bigtop has some hard and
>> soft
>> > > > > > > expectation
>> > > > > > > > >> for
>> > > > > > > > >> > new
>> > > > > > > > >> > > > > > > > components
>> > > > > > > > >> > > > > > > > > to
>> > > > > > > > >> > > > > > > > > > >> get
>> > > > > > > > >> > > > > > > > > > >> > integrated into Bigtop.
>> > > > > > > > >> > > > > > > > > > >> > Here is the list of it:
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >> > Most of them seems to be standard
>> ASF
>> > > > based
>> > > > > > > > >> > > requirements,
>> > > > > > > > >> > > > > but
>> > > > > > > > >> > > > > > > few
>> > > > > > > > >> > > > > > > > > need
>> > > > > > > > >> > > > > > > > > > >> to
>> > > > > > > > >> > > > > > > > > > >> > be checked for:
>> > > > > > > > >> > > > > > > > > > >> > 1. Software projects are expected
>> to
>> > be
>> > > > > > > Licensed
>> > > > > > > > >> under
>> > > > > > > > >> > > > > Apache
>> > > > > > > > >> > > > > > > > > License,
>> > > > > > > > >> > > > > > > > > > >> > Version 2.0 (and their
>> dependencies
>> > are
>> > > > > > > expected
>> > > > > > > > >> to be
>> > > > > > > > >> > > > > > > compatible
>> > > > > > > > >> > > > > > > > > with
>> > > > > > > > >> > > > > > > > > > >> this
>> > > > > > > > >> > > > > > > > > > >> > license)
>> > > > > > > > >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but
>> need
>> > to
>> > > > > check
>> > > > > > > if
>> > > > > > > > >> > > > > dependencies
>> > > > > > > > >> > > > > > of
>> > > > > > > > >> > > > > > > > > Apex
>> > > > > > > > >> > > > > > > > > > >> are
>> > > > > > > > >> > > > > > > > > > >> > compatible with ASL 2.0. This I
>> guess
>> > > > would
>> > > > > > be
>> > > > > > > a
>> > > > > > > > >> > > > > verification
>> > > > > > > > >> > > > > > > > check.
>> > > > > > > > >> > > > > > > > > > >> > 2. Software projects are expected
>> to
>> > be
>> > > > > > > > compatible
>> > > > > > > > >> > with
>> > > > > > > > >> > > > all
>> > > > > > > > >> > > > > of
>> > > > > > > > >> > > > > > > the
>> > > > > > > > >> > > > > > > > > > >> > supported platforms that Bigtop
>> > > > > distribution
>> > > > > > is
>> > > > > > > > >> > > targeting
>> > > > > > > > >> > > > > > > > > > >> >     - This needs verifying whether
>> > our
>> > > > > > software
>> > > > > > > > >> runs
>> > > > > > > > >> > > fine
>> > > > > > > > >> > > > in
>> > > > > > > > >> > > > > > > > > centos-6
>> > > > > > > > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
>> > > debian-8
>> > > > > > > > >> > opensuse-13.2.
>> > > > > > > > >> > > > > > > > > > >> > 3. What smoke tests that should be
>> > > added
>> > > > > for
>> > > > > > > > >> > deployment.
>> > > > > > > > >> > > > > > > > > > >> > 4. Identifying the test artifacts
>> > which
>> > > > > goes
>> > > > > > > > beyond
>> > > > > > > > >> > > smoke
>> > > > > > > > >> > > > > test
>> > > > > > > > >> > > > > > > > > > >> >     - These are basically the
>> > > integration
>> > > > > > tests
>> > > > > > > > for
>> > > > > > > > >> > > > > > verification
>> > > > > > > > >> > > > > > > > > after
>> > > > > > > > >> > > > > > > > > > >> the
>> > > > > > > > >> > > > > > > > > > >> > deployment. This is a soft
>> > requirement,
>> > > > but
>> > > > > > aim
>> > > > > > > > is
>> > > > > > > > >> to
>> > > > > > > > >> > > > > achieve
>> > > > > > > > >> > > > > > > this
>> > > > > > > > >> > > > > > > > > as
>> > > > > > > > >> > > > > > > > > > >> well
>> > > > > > > > >> > > > > > > > > > >> > or at least have explanation why
>> not
>> > to
>> > > > > > > include.
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >> > If there are any from the link
>> which
>> > > > > > explicitly
>> > > > > > > > >> needs
>> > > > > > > > >> > to
>> > > > > > > > >> > > > be
>> > > > > > > > >> > > > > > > > checked
>> > > > > > > > >> > > > > > > > > > >> other
>> > > > > > > > >> > > > > > > > > > >> > than above 4, please let us know.
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >> > *Step 2) Adding Apex as component
>> to
>> > > > > Bigtop*
>> > > > > > > > >> > > > > > > > > > >> > From one of the mail archive of
>> > Bigtop,
>> > > > it
>> > > > > > was
>> > > > > > > > >> learnt
>> > > > > > > > >> > > that
>> > > > > > > > >> > > > > the
>> > > > > > > > >> > > > > > > > > bigtop
>> > > > > > > > >> > > > > > > > > > >> > community want to see the
>> addition of
>> > > new
>> > > > > > > > >> components
>> > > > > > > > >> > in
>> > > > > > > > >> > > > > > phases.
>> > > > > > > > >> > > > > > > > Here
>> > > > > > > > >> > > > > > > > > > are
>> > > > > > > > >> > > > > > > > > > >> > the phases:
>> > > > > > > > >> > > > > > > > > > >> > 1. Packaging
>> > > > > > > > >> > > > > > > > > > >> >     - This needs creating of
>> package
>> > > i.e.
>> > > > > > rpm &
>> > > > > > > > deb
>> > > > > > > > >> > > files.
>> > > > > > > > >> > > > > > > > > > >> >     - documentations/READMEs,
>> > LICENSE,
>> > > > > > > DISCLAMER,
>> > > > > > > > >> > NOTES
>> > > > > > > > >> > > > etc
>> > > > > > > > >> > > > > if
>> > > > > > > > >> > > > > > > any
>> > > > > > > > >> > > > > > > > > > >> needed.
>> > > > > > > > >> > > > > > > > > > >> >     - Any documentation that need
>> to
>> > be
>> > > > > added
>> > > > > > > to
>> > > > > > > > >> > > > > distribution
>> > > > > > > > >> > > > > > of
>> > > > > > > > >> > > > > > > > our
>> > > > > > > > >> > > > > > > > > > >> > software.
>> > > > > > > > >> > > > > > > > > > >> >     - Any license information of
>> > > > > dependencies
>> > > > > > > > >> required
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > be
>> > > > > > > > >> > > > > > > added
>> > > > > > > > >> > > > > > > > > to
>> > > > > > > > >> > > > > > > > > > >> > package
>> > > > > > > > >> > > > > > > > > > >> > 2. Smoke tests (at very least)
>> > > > > > > > >> > > > > > > > > > >> >     - Adding smoke test for
>> > packaging.
>> > > > > > > > >> > > > > > > > > > >> > 3. Puppet recipes for automatic
>> > > > deployment
>> > > > > > and
>> > > > > > > > >> > > > configuration
>> > > > > > > > >> > > > > > > > > > >> >     - Add puppet recipes for
>> > automatic
>> > > > > > > deployment
>> > > > > > > > >> > > > > > > > > > >> > 4. Integration tests
>> > > > > > > > >> > > > > > > > > > >> >     - For verification of
>> > deployments.
>> > > > > > > > >> > > > > > > > > > >> > 5. license clearance:
>> > > > > > > > >> > > > > > > > > > >> >     Run 'gradle rat' to make sure
>> all
>> > > new
>> > > > > > stuff
>> > > > > > > > is
>> > > > > > > > >> > > > compliant
>> > > > > > > > >> > > > > > > with
>> > > > > > > > >> > > > > > > > > ASF
>> > > > > > > > >> > > > > > > > > > >> > license requirements. If you add
>> code
>> > > > > > licenses
>> > > > > > > > >> under
>> > > > > > > > >> > > > > different
>> > > > > > > > >> > > > > > > > > > licenses,
>> > > > > > > > >> > > > > > > > > > >> > those would need to be listed in
>> the
>> > > > > NOTICE.
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >> > Please share your thoughts on the
>> > > > approach.
>> > > > > > > > >> > > > > > > > > > >> > We'll start corresponding
>> > communication
>> > > > on
>> > > > > > > bigtop
>> > > > > > > > >> > > mailing
>> > > > > > > > >> > > > > list
>> > > > > > > > >> > > > > > > as
>> > > > > > > > >> > > > > > > > > > well.
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >> > We have some specific
>> > > > questions/suggestions
>> > > > > > > > >> related to
>> > > > > > > > >> > > > what
>> > > > > > > > >> > > > > > > should
>> > > > > > > > >> > > > > > > > > be
>> > > > > > > > >> > > > > > > > > > >> the
>> > > > > > > > >> > > > > > > > > > >> > content of the package and what
>> > should
>> > > be
>> > > > > the
>> > > > > > > > smoke
>> > > > > > > > >> > > tests,
>> > > > > > > > >> > > > > but
>> > > > > > > > >> > > > > > > in
>> > > > > > > > >> > > > > > > > > the
>> > > > > > > > >> > > > > > > > > > >> > interest of not having too much
>> > content
>> > > > > here,
>> > > > > > > > we'll
>> > > > > > > > >> > put
>> > > > > > > > >> > > > the
>> > > > > > > > >> > > > > > > > > questions
>> > > > > > > > >> > > > > > > > > > >> as a
>> > > > > > > > >> > > > > > > > > > >> > separate mail in this mailthread.
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >> > Thanks,
>> > > > > > > > >> > > > > > > > > > >> > Chinmay.
>> > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Alright. Will not add it.

On Mon, Feb 15, 2016 at 3:24 PM, Pradeep A. Dalvi <pr...@apache.org> wrote:

> We may not need /usr/lib/apex/bin/dtcli (symlink to
> /usr/lib/apex/bin/apex), when there are /usr/bin/apex & /usr/bin/dtcli
> (which are in PATH).
>
> On Mon, Feb 15, 2016 at 10:54 AM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > /usr/lib/apex/bin contains executables related to apex. dtcli, even
> though
> > its a symlink, for current users its still an executable. Hence for the
> > sake of completion, added dtcli link under /usr/lib/apex/bin.
> >
> > But if you think otherwise, please let me know.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Mon, Feb 15, 2016 at 10:42 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> > > Why the dtcli link under lib?
> > >
> > >
> > > On Sun, Feb 14, 2016 at 9:09 PM, Chinmay Kolhatkar <chinmay@apache.org
> >
> > > wrote:
> > >
> > > > Hmmm... I get your point. In development mode, it tries to find the
> > > hadoop
> > > > executable like following:
> > > >
> > > >
> > > >
> > > >
> > >
> >
> HADOOP_SEARCH_PATH="${HADOOP_PREFIX}/bin:${HADOOP_HOME}/bin:${PATH}:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:."
> > > >   export DT_HADOOP=`PATH=${HADOOP_SEARCH_PATH} && command -v hadoop
> > > > 2>/dev/null`
> > > >
> > > > I will check with bigtop community that what is the best way to
> ensure
> > > > during installation that hadoop binary will be present in above
> search
> > > path
> > > > search.
> > > >
> > > > Also, I checked as you suggested, i.e. removed dt-env.sh from package
> > and
> > > > tried to run the dtcli... It works!!!
> > > >
> > > > Thank you for pointing that out Thomas.
> > > >
> > > > Now I agree with you that, we can get rid of conf/dt-env.sh.. As this
> > was
> > > > the only file in conf/, unless someone has anything to include in
> conf/
> > > > folder, I'll remove that too then.
> > > >
> > > > Also, this would mean, only dt* that remains in package are dtcli
> links
> > > to
> > > > apex script added for backward compatibility.
> > > > i.e following:
> > > >
> > > > /
> > > > `-- usr
> > > >     |-- bin
> > > >     |   |-- apex -> ../lib/apex/bin/apex
> > > >     |   `-- dtcli -> ../lib/apex/bin/apex
> > > >     |-- lib
> > > >     |   `-- apex
> > > >     |       |-- bin
> > > >     |       |   |-- apex   // This is the real script
> > > >     |       |   `-- dtcli -> apex
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > >
> > > > On Mon, Feb 15, 2016 at 10:22 AM, Thomas Weise <
> thomas@datatorrent.com
> > >
> > > > wrote:
> > > >
> > > > > I'm not convinced dt-env.sh is even needed. In development mode,
> the
> > > file
> > > > > doesn't exist but it will still find Hadoop.
> > > > >
> > > > > The absolute path to hadoop script was introduced to deal with the
> > > layout
> > > > > of specific distros. You will know whether that applies to Bigtop
> or
> > > not.
> > > > >
> > > > > Thomas
> > > > >
> > > > > On Sun, Feb 14, 2016 at 8:16 PM, Chinmay Kolhatkar <
> > chinmay@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Thomas,
> > > > > >
> > > > > > There are only 2 things that are required from dt-env.sh:
> > > > > > 1. DT_HADOOP variable which is exported by dt-env.sh This
> variable
> > > has
> > > > > > absolute path for hadoop binary
> > > > > > 2. A find_hadoop() method which find the path of hadoop binary.
> > > > > >
> > > > > > JAVA_HOME is not required for this.
> > > > > >
> > > > > > The idea is not to change the dtcli. Current dtcli (in apex
> > > > > > 3.3.0-incubating src code), sources dt-env.sh for above 2
> > > information.
> > > > > > Hence included dt-env.sh
> > > > > >
> > > > > > If the community agrees, we can make dtcli more generic so that
> in
> > > > future
> > > > > > packing of bigtop, dt-env.sh can be completely excluded OR at
> least
> > > > > change
> > > > > > the name to apex-env.sh.
> > > > > >
> > > > > > Thanks,
> > > > > > Chinmay.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise <
> > > thomas@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Chinmay,
> > > > > > >
> > > > > > > Can you please specify what settings in dt-env.sh are required
> > in a
> > > > > > bigtop
> > > > > > > environment?
> > > > > > >
> > > > > > > Location of hadoop and JAVA_HOME should be provided by the
> stack
> > > > > already?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Thomas
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> > > > chinmay@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Here is a suggestion I have related to name change:
> > > > > > > >
> > > > > > > > 1. While packaging we change the name of "dtcli" script to
> > > "apex".
> > > > > This
> > > > > > > is
> > > > > > > > done so that in future when we change the name to "apex",
> users
> > > of
> > > > > > bigtop
> > > > > > > > apex don't have to transition much.
> > > > > > > > 2. We also keep a symlink named dtcli which points to apex
> > > script.
> > > > > This
> > > > > > > is
> > > > > > > > for backward compatibility.
> > > > > > > > 3. We still keep conf/dt-env.sh for first integration in
> > bigtop.
> > > > This
> > > > > > is
> > > > > > > in
> > > > > > > > the interest of not changing the content of dtcli or apex
> > script.
> > > > > > > > 4. In apex 3.4.0, we take care of all these naming related
> > > changes
> > > > > and
> > > > > > > > update bigtop repository later to remove references to dtcli
> > all
> > > > > > > together.
> > > > > > > >
> > > > > > > > Please share your thoughts on above approach.
> > > > > > > >
> > > > > > > > Also, please share what could be the man page content for
> > "apex".
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chinmay.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> > > > > chinmay@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Yes.. It'll be build from source tar downloaded from one of
> > the
> > > > > > apache
> > > > > > > > > mirror: apache.osuosl.org
> > > > > > > > >
> > > > > > > > > The reason for dt-env.sh is it's sourced from dtcli. This
> > > leaves
> > > > > us 3
> > > > > > > > > options:
> > > > > > > > > If we should not have any dt-env.sh, dtcli would need a
> > change
> > > > > while
> > > > > > > > > packaging.
> > > > > > > > >
> > > > > > > > > But if we want to use dtcli as it is, then we would need
> > > > dt-env.sh
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> > > > > > thomas@datatorrent.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Looks good overall, though there shouldn't be any
> dt-env.sh
> > > > > > > > >>
> > > > > > > > >> What will the bigtop package be built from, the source
> tar?
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > > > > > > chinmay@apache.org>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Thomas,
> > > > > > > > >> >
> > > > > > > > >> > Thanks for the feedback.
> > > > > > > > >> >
> > > > > > > > >> > First, we're not changing the name anywhere. We'll
> follow
> > > what
> > > > > > > > >> currently is
> > > > > > > > >> > in source tarball for 3.3.0-incubating version.
> > > > > > > > >> >
> > > > > > > > >> > Secondly, I've mentioned a directory structure below
> which
> > > is
> > > > > > inline
> > > > > > > > >> with
> > > > > > > > >> > other existing integrations with bigtop.
> > > > > > > > >> >
> > > > > > > > >> > The need for each file is as follows:
> > > > > > > > >> > 1. bin/dtcli -> CLI for apex picked from
> > > > > engine/src/main/scripts/
> > > > > > of
> > > > > > > > >> source
> > > > > > > > >> > code apex 3.3.0-incubating.
> > > > > > > > >> >
> > > > > > > > >> > 2. conf/dt-env.sh -> This is source from dtcli. This
> > > searches
> > > > > for
> > > > > > > > hadoop
> > > > > > > > >> > binary path and exports an env variable for dtcli to use
> > it.
> > > > > > > > >> >
> > > > > > > > >> > 3. lib/ -> Libraries for CLI. This list is extracted in
> a
> > > > > similar
> > > > > > > way
> > > > > > > > to
> > > > > > > > >> > how DT Community Edition finds dependency jars.
> > > > > > > > >> > Please note that this is the first iteration list of
> jars.
> > > I'm
> > > > > > > trying
> > > > > > > > to
> > > > > > > > >> > narrow this down to only those which are really
> required.
> > > > > > > > >> > The test that I'm running to check if dtcli runs fine
> with
> > > > given
> > > > > > > > >> dependency
> > > > > > > > >> > is to launch+shutdown+kill+status for pi demo.
> > > > > > > > >> >
> > > > > > > > >> > Above is the directory structure which is required for
> CLI
> > > to
> > > > > > work.
> > > > > > > > >> > What aniruddha and pradeep has mentioned are some
> > additional
> > > > > files
> > > > > > > > which
> > > > > > > > >> > makes the like of administrator easier. For eg.
> /etc/skel
> > > > > > > > >> >
> > > > > > > > >> > Please let me know if this is inline with what you're
> > > > thinking.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Chinmay.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > .
> > > > > > > > >> > |-- bin
> > > > > > > > >> > |   `-- dtcli
> > > > > > > > >> > |-- conf
> > > > > > > > >> > |   `-- dt-env.sh
> > > > > > > > >> > `-- lib
> > > > > > > > >> >     |-- ant-1.9.2.jar
> > > > > > > > >> >     |-- ant-launcher-1.9.2.jar
> > > > > > > > >> >     |-- apex-api-3.3.0-incubating.jar
> > > > > > > > >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > > > > > > > >> >     |-- apex-common-3.3.0-incubating.jar
> > > > > > > > >> >     |-- apex-engine.jar
> > > > > > > > >> >     |-- apex-engine-tests.jar
> > > > > > > > >> >     |-- async-http-client-1.7.20.jar
> > > > > > > > >> >     |-- bval-core-0.5.jar
> > > > > > > > >> >     |-- bval-jsr303-0.5.jar
> > > > > > > > >> >     |-- commons-beanutils-1.8.3.jar
> > > > > > > > >> >     |-- commons-codec-1.10.jar
> > > > > > > > >> >     |-- commons-lang3-3.1.jar
> > > > > > > > >> >     |-- commons-logging-1.1.3.jar
> > > > > > > > >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > > > > > > >> >     |-- hadoop-common-2.2.0-tests.jar
> > > > > > > > >> >     |-- httpclient-4.3.5.jar
> > > > > > > > >> >     |-- httpcore-4.3.2.jar
> > > > > > > > >> >     |-- jackson-core-asl-1.9.2.jar
> > > > > > > > >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > > > > > > >> >     |-- javax.servlet-3.1.jar
> > > > > > > > >> >     |-- javax.servlet-api-3.0.1.jar
> > > > > > > > >> >     |-- jersey-apache-client4-1.9.jar
> > > > > > > > >> >     |-- jersey-client-1.9.jar
> > > > > > > > >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > > > > > > >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > > > > > > >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > > > > > > >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > > > > > > > >> >     |-- jline-2.11.jar
> > > > > > > > >> >     |-- kryo-2.24.0.jar
> > > > > > > > >> >     |-- mbassador-1.1.9.jar
> > > > > > > > >> >     |-- minlog-1.2.jar
> > > > > > > > >> >     |-- netlet-1.2.0.jar
> > > > > > > > >> >     |-- netty-3.6.6.Final.jar
> > > > > > > > >> >     |-- objenesis-2.1.jar
> > > > > > > > >> >     |-- validation-api-1.1.0.Final.jar
> > > > > > > > >> >     |-- xbean-asm5-shaded-4.3.jar
> > > > > > > > >> >     `-- zip4j-1.3.2.jar
> > > > > > > > >> >
> > > > > > > > >> > 3 directories, 40 files
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > > > > > > thomas@datatorrent.com
> > > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Chinmay,
> > > > > > > > >> > >
> > > > > > > > >> > > Before discussing where to put the files, let's make
> > sure
> > > > they
> > > > > > are
> > > > > > > > >> really
> > > > > > > > >> > > needed for the operation of the CLI. As for names,
> > > anything
> > > > > new
> > > > > > > > needs
> > > > > > > > >> to
> > > > > > > > >> > > reflect Apex in the name and should follow common
> > > > conventions,
> > > > > > > > >> especially
> > > > > > > > >> > > in Bigtop where there are many existing integrations
> to
> > > look
> > > > > at.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > > Thomas
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > > > > > > > >> > > chinmay@datatorrent.com>
> > > > > > > > >> > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > Really good points Pradeep and Aniruddha.
> > > > > > > > >> > > >
> > > > > > > > >> > > > 1. I believe we won't need to change the dtcli
> > > considering
> > > > > it
> > > > > > > > works
> > > > > > > > >> > with
> > > > > > > > >> > > DT
> > > > > > > > >> > > > Community edition. We can keep the directory
> structure
> > > > > similar
> > > > > > > to
> > > > > > > > >> that.
> > > > > > > > >> > > > dt-env.sh has variables which contains information
> > > > required
> > > > > > for
> > > > > > > > >> dtcli
> > > > > > > > >> > to
> > > > > > > > >> > > > launch.
> > > > > > > > >> > > >
> > > > > > > > >> > > > 2. Let me check with bigtop community that whether
> > they
> > > > > > > facilitate
> > > > > > > > >> the
> > > > > > > > >> > > > installation of rpms/debs before user is created. In
> > > > either
> > > > > > > case,
> > > > > > > > >> > current
> > > > > > > > >> > > > dtcli creates a .dt folder in home directory. Also
> > > before
> > > > > > > putting
> > > > > > > > >> > > anything
> > > > > > > > >> > > > in /etc/skel we need to define what are the default
> > > > contents
> > > > > > > that
> > > > > > > > >> > should
> > > > > > > > >> > > go
> > > > > > > > >> > > > to ~/.dt/ folder. If there is no defaults, probably
> we
> > > > > should
> > > > > > > not
> > > > > > > > >> > > > explicitly add it in /etc/skel.
> > > > > > > > >> > > >
> > > > > > > > >> > > > 3.  /etc/profile.d approach looks nice. This way
> > > contents
> > > > of
> > > > > > > > >> dt-env.sh
> > > > > > > > >> > > are
> > > > > > > > >> > > > present as env variables. But I see a catch there.
> > > Adding
> > > > > > > > dt-env.sh
> > > > > > > > >> to
> > > > > > > > >> > > > /etc/profile.d would make all the variables
> available
> > at
> > > > > > runtime
> > > > > > > > all
> > > > > > > > >> > the
> > > > > > > > >> > > > time. I feel a little skeptical about that. Maybe a
> > > > possible
> > > > > > > > >> collision
> > > > > > > > >> > > can
> > > > > > > > >> > > > occur with other application vars.
> > > > > > > > >> > > > Moreover, current dtcli does source
> > "../conf/dt-env.sh"
> > > > and
> > > > > > > > >> > > > "~/.dt/dt-env.sh". So those variables are anyway
> > > > available.
> > > > > > > > >> > > >
> > > > > > > > >> > > > @Aniruddha, the Jira and PR effort is happening at
> > > Bigtop
> > > > > > > > >> > > Jira/repository.
> > > > > > > > >> > > > Packaging related code usually goes there. (That's
> > what
> > > > all
> > > > > > the
> > > > > > > > >> > > components
> > > > > > > > >> > > > in bigtop does).
> > > > > > > > >> > > > Having said that, once a PR is created, I'll be
> > sharing
> > > > the
> > > > > > link
> > > > > > > > of
> > > > > > > > >> PR
> > > > > > > > >> > > > here, so that apex community as well can review it.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Thanks,
> > > > > > > > >> > > > Chinmay.
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare
> <
> > > > > > > > >> > > > aniruddha@datatorrent.com> wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > > +1 on suggestions and approach.
> > > > > > > > >> > > > > We may need to iron out details about exact paths
> > etc.
> > > > > > > > >> > > > > Which can be done on jira / PR comments.
> > > > > > > > >> > > > > Is that right @dev?
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > > > > > > >> > apache@pradeepdalvi.com>
> > > > > > > > >> > > > > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > Inline comments...
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
> > Kolhatkar <
> > > > > > > > >> > > > > > chinmay@datatorrent.com>
> > > > > > > > >> > > > > > wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > @Thomas, Not all the jar present in DT
> community
> > > > > edition
> > > > > > > > will
> > > > > > > > >> be
> > > > > > > > >> > > > > included
> > > > > > > > >> > > > > > > there.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Here is the list of jars I found from common
> > > > > > dependencies
> > > > > > > > >> between
> > > > > > > > >> > > of
> > > > > > > > >> > > > > apex
> > > > > > > > >> > > > > > > and DT Community edition:
> > > > > > > > >> > > > > > > netlet-1.2.0.jar
> > > > > > > > >> > > > > > > kryo-2.24.0.jar
> > > > > > > > >> > > > > > > jackson-core-asl-1.9.2.jar
> > > > > > > > >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > > > > >> > > > > > > async-http-client-1.7.20.jar
> > > > > > > > >> > > > > > > netty-3.6.6.Final.jar
> > > > > > > > >> > > > > > > validation-api-1.1.0.Final.jar
> > > > > > > > >> > > > > > > bval-jsr303-0.5.jar
> > > > > > > > >> > > > > > > bval-core-0.5.jar
> > > > > > > > >> > > > > > > commons-lang3-3.1.jar
> > > > > > > > >> > > > > > > commons-beanutils-1.8.3.jar
> > > > > > > > >> > > > > > > httpclient-4.3.5.jar
> > > > > > > > >> > > > > > > commons-codec-1.10.jar
> > > > > > > > >> > > > > > > zip4j-1.3.2.jar
> > > > > > > > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > > > >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > > > >> > > > > > > jersey-apache-client4-1.9.jar
> > > > > > > > >> > > > > > > jline-2.11.jar
> > > > > > > > >> > > > > > > ant-1.9.2.jar
> > > > > > > > >> > > > > > > ant-launcher-1.9.2.jar
> > > > > > > > >> > > > > > > mbassador-1.1.9.jar
> > > > > > > > >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > > > >> > > > > > > jackson-xc-1.9.2.jar
> > > > > > > > >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Ofcourse, I'll be running some tests do check
> > that
> > > > > dtcli
> > > > > > > > works
> > > > > > > > >> > > > properly
> > > > > > > > >> > > > > > for
> > > > > > > > >> > > > > > > launch+shutdown+kill of apps with only these
> > > > libraries
> > > > > > > > >> present in
> > > > > > > > >> > > > > > isolation
> > > > > > > > >> > > > > > > without dependency on local m2.
> > > > > > > > >> > > > > > > I'm believe that there are unwanted jars which
> > are
> > > > > used
> > > > > > > for
> > > > > > > > >> > compile
> > > > > > > > >> > > > > time
> > > > > > > > >> > > > > > > dependency and not runtime in above list
> which I
> > > can
> > > > > > drop
> > > > > > > to
> > > > > > > > >> keep
> > > > > > > > >> > > > > package
> > > > > > > > >> > > > > > > size minimal.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > @Anniruddha,
> > > > > > > > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
> > > dt-env.sh
> > > > > is
> > > > > > > > >> required.
> > > > > > > > >> > > > Only
> > > > > > > > >> > > > > > > required values I see in it are classpath. and
> > > dtcli
> > > > > is
> > > > > > > > >> capable
> > > > > > > > >> > of
> > > > > > > > >> > > > > > building
> > > > > > > > >> > > > > > > it on the fly.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how
> > dtcli
> > > > can
> > > > > > > find
> > > > > > > > >> the
> > > > > > > > >> > jar
> > > > > > > > >> > > > > path,
> > > > > > > > >> > > > > > > dt-env.sh and all such conf files. If we need
> a
> > > > change
> > > > > > in
> > > > > > > > >> dtcli,
> > > > > > > > >> > > then
> > > > > > > > >> > > > > > > community's opinion is required for whether
> > dtcli
> > > > > should
> > > > > > > > >> change
> > > > > > > > >> > in
> > > > > > > > >> > > > our
> > > > > > > > >> > > > > > repo
> > > > > > > > >> > > > > > > or a copy of that with required changes exist
> in
> > > > > bigtop
> > > > > > > repo
> > > > > > > > >> > until
> > > > > > > > >> > > we
> > > > > > > > >> > > > > > make
> > > > > > > > >> > > > > > > the dtcli generic enough.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Can we have these file paths set using
> environment
> > > > > > variables
> > > > > > > > >> with
> > > > > > > > >> > > some
> > > > > > > > >> > > > > > default values in dtcli?
> > > > > > > > >> > > > > > Then we can set such params in dt-env.sh.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > 3. dt-sited.xml -> How is this file different
> > from
> > > > > > > > >> dt-site.xml?
> > > > > > > > >> > Not
> > > > > > > > >> > > > > sure
> > > > > > > > >> > > > > > if
> > > > > > > > >> > > > > > > adding in /etc/skel would help. Files & Dirs
> > from
> > > > > > > /etc/skel
> > > > > > > > >> are
> > > > > > > > >> > > > copied
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > home of new user when useradd program is
> called.
> > > But
> > > > > for
> > > > > > > > >> existing
> > > > > > > > >> > > > users
> > > > > > > > >> > > > > > > this won't be of any use. Moreover, dtcli
> > creates
> > > > > ~/.dt/
> > > > > > > on
> > > > > > > > >> the
> > > > > > > > >> > fly
> > > > > > > > >> > > > if
> > > > > > > > >> > > > > > not
> > > > > > > > >> > > > > > > encountered for the first time. Again correct
> me
> > > if
> > > > > I'm
> > > > > > > > wrong.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Yes, you are right. However this would probably
> be
> > > > > > necessary
> > > > > > > > >> step
> > > > > > > > >> > for
> > > > > > > > >> > > > > > rpm/deb, as they may also get installed during
> OS
> > > > > install
> > > > > > > and
> > > > > > > > >> > before
> > > > > > > > >> > > > user
> > > > > > > > >> > > > > > accounts were created.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not
> > sure
> > > > its
> > > > > > the
> > > > > > > > best
> > > > > > > > >> > > idea.
> > > > > > > > >> > > > > > We'll
> > > > > > > > >> > > > > > > anyway put dtcli in location which is by
> default
> > > > > present
> > > > > > > in
> > > > > > > > >> path
> > > > > > > > >> > > i.e.
> > > > > > > > >> > > > > > > /usr/bin.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > For other env variables specific to apex, I'll
> > > prefer
> > > > to
> > > > > > use
> > > > > > > > >> > > dt-env.sh
> > > > > > > > >> > > > > and
> > > > > > > > >> > > > > > > source it in dtcli rather than changing bashrc
> > > > etc...
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/
> shall
> > > be
> > > > > the
> > > > > > > > >> cleaner
> > > > > > > > >> > way
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > achieve the same.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > Thanks,
> > > > > > > > >> > > > > > > Chinmay.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
> > > Thombare
> > > > <
> > > > > > > > >> > > > > > > aniruddha@datatorrent.com> wrote:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > Hi,
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > @Chinmay,
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > We need to consider following:
> > > > > > > > >> > > > > > > > System wide default config files can be
> > located
> > > at
> > > > > > > > following
> > > > > > > > >> > > > > locations:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change
> > > dtcli
> > > > > > > > behaviour
> > > > > > > > >> on
> > > > > > > > >> > > how
> > > > > > > > >> > > > it
> > > > > > > > >> > > > > > > finds
> > > > > > > > >> > > > > > > > those locations)
> > > > > > > > >> > > > > > > > /etc/apex/dt-site.xml (we may have to change
> > > dtcli
> > > > > > > > >> behaviour on
> > > > > > > > >> > > how
> > > > > > > > >> > > > > it
> > > > > > > > >> > > > > > > > finds those locations)
> > > > > > > > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files
> > (for
> > > > new
> > > > > > > users
> > > > > > > > >> that
> > > > > > > > >> > > will
> > > > > > > > >> > > > > be
> > > > > > > > >> > > > > > > > created in system in future.)
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > We may also have to modify bashrc /
> > bashprofile
> > > > for
> > > > > > > > >> population
> > > > > > > > >> > > any
> > > > > > > > >> > > > > > > > variables that are required.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > @dev, please put in your comments /
> > suggestion.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Aniruddha
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas
> Weise
> > <
> > > > > > > > >> > > > > thomas@datatorrent.com>
> > > > > > > > >> > > > > > > > wrote:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > -->
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> > > > Kolhatkar
> > > > > <
> > > > > > > > >> > > > > > > > chinmay@datatorrent.com
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > Hello Everyone!!
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Continuing with packaging effort
> (rpm+deb)
> > > of
> > > > > > apex,
> > > > > > > > here
> > > > > > > > >> > are
> > > > > > > > >> > > > some
> > > > > > > > >> > > > > > > > > proposals
> > > > > > > > >> > > > > > > > > > about package structure etc..
> > > > > > > > >> > > > > > > > > > Before posting it on bbigtop mailing
> > list, I
> > > > > have
> > > > > > > some
> > > > > > > > >> > > question
> > > > > > > > >> > > > > for
> > > > > > > > >> > > > > > > > apex
> > > > > > > > >> > > > > > > > > > community.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Proposed Directory structure of apex
> > package
> > > > for
> > > > > > > both
> > > > > > > > >> deb &
> > > > > > > > >> > > > rpm:
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > >> > > > > > > > > >
> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > >> > > > > > > > > >
> > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > >> > > > > > > > > >
> > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > >> > > > > > > > > >
> > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > >> > > > > > > > > > <other dependent jars> = All the 3rd
> party
> > > > jars
> > > > > > > which
> > > > > > > > >> are
> > > > > > > > >> > > > > required
> > > > > > > > >> > > > > > > for
> > > > > > > > >> > > > > > > > > apex
> > > > > > > > >> > > > > > > > > > to run. Usually the dependencies are
> > > packaged
> > > > as
> > > > > > > part
> > > > > > > > of
> > > > > > > > >> > > > rpm/deb
> > > > > > > > >> > > > > by
> > > > > > > > >> > > > > > > any
> > > > > > > > >> > > > > > > > > > software in bigtop.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Can you specify what those jars are in
> > Bigtop
> > > > > > context?
> > > > > > > > >> Same
> > > > > > > > >> > as
> > > > > > > > >> > > > > > shipped
> > > > > > > > >> > > > > > > > with
> > > > > > > > >> > > > > > > > > DT community addition under lib/ ?
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > <package LICENSE> = Licenses of
> > > corresponding
> > > > of
> > > > > > 3rd
> > > > > > > > >> party
> > > > > > > > >> > > jars
> > > > > > > > >> > > > > > which
> > > > > > > > >> > > > > > > > > needs
> > > > > > > > >> > > > > > > > > > to included while packaging.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Questions related to this:
> > > > > > > > >> > > > > > > > > > 1. Should we call the cli of apex as
> > "apex"
> > > > > > instead
> > > > > > > of
> > > > > > > > >> > > "dtcli"
> > > > > > > > >> > > > in
> > > > > > > > >> > > > > > > > bigtop
> > > > > > > > >> > > > > > > > > > package?
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > I think we should keep the name until we
> are
> > > > able
> > > > > to
> > > > > > > > >> change
> > > > > > > > >> > it
> > > > > > > > >> > > in
> > > > > > > > >> > > > > > Apex.
> > > > > > > > >> > > > > > > > > Otherwise this may get confusing.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > 2. I see that all softwares in bigtop
> have
> > > man
> > > > > > page
> > > > > > > > for
> > > > > > > > >> > their
> > > > > > > > >> > > > > > > > > executables.
> > > > > > > > >> > > > > > > > > > I think we should have it too for dtcli.
> > Is
> > > > > there
> > > > > > > any
> > > > > > > > >> > > > > documentation
> > > > > > > > >> > > > > > > > > which I
> > > > > > > > >> > > > > > > > > > can convert to man page? or can I use
> > output
> > > > of
> > > > > > > "dtcli
> > > > > > > > >> > > --help"?
> > > > > > > > >> > > > > > > > > > 3. Do we want to call version of apex in
> > > > Bigtop
> > > > > as
> > > > > > > > >> 3.3.0 OR
> > > > > > > > >> > > > > > > > > > 3.3.0-incubating?
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Has to be -incubating
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > 4. Is it ok for apex package of bigtop
> to
> > > > depend
> > > > > > on
> > > > > > > > >> 2.7.1
> > > > > > > > >> > > > > version
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > > > > bigtop hadoop? Any problems that we see
> > with
> > > > > this
> > > > > > > > >> > dependency?
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Should work without changed. I thought we
> > > > > certified
> > > > > > > > >> against
> > > > > > > > >> > > > 2.7.0?
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > 5. Following is the apache mirror from
> > which
> > > > > > bigtop
> > > > > > > > will
> > > > > > > > >> > pick
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > apex
> > > > > > > > >> > > > > > > > > > source for compilation and packaging.
> > Please
> > > > > > correct
> > > > > > > > if
> > > > > > > > >> > > > > incorrect:
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > >
> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > > >> > > > > > > > > >     NOTE: apache.osuosl.org is the
> mirror
> > > > used
> > > > > by
> > > > > > > all
> > > > > > > > >> the
> > > > > > > > >> > > > > > softwares
> > > > > > > > >> > > > > > > in
> > > > > > > > >> > > > > > > > > > bigtop.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Please share your opinion.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Thanks,
> > > > > > > > >> > > > > > > > > > Chinmay.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> > > > > > Kolhatkar <
> > > > > > > > >> > > > > > > > > > chinmay@datatorrent.com>
> > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Yes.. Starting to work on the
> packaging.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > I've already started discussion on
> > bigtop
> > > > dev
> > > > > > > > mailing
> > > > > > > > >> > list
> > > > > > > > >> > > > for
> > > > > > > > >> > > > > > > > > > > integration. Also created a Jira for
> the
> > > > same.
> > > > > > For
> > > > > > > > >> this
> > > > > > > > >> > > > > > communities
> > > > > > > > >> > > > > > > > > > > reference, here is the bigtop Jira:
> > > > > BIGTOP-2313.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas
> > > > Weise <
> > > > > > > > >> > > > > > > thomas@datatorrent.com
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >> Chinmay,
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >> I don't see anything under
> > prerequisites
> > > > that
> > > > > > > would
> > > > > > > > >> be a
> > > > > > > > >> > > > > > problem.
> > > > > > > > >> > > > > > > We
> > > > > > > > >> > > > > > > > > > >> looked
> > > > > > > > >> > > > > > > > > > >> at the ASF licencing compatibility as
> > > part
> > > > of
> > > > > > > > >> becoming
> > > > > > > > >> > an
> > > > > > > > >> > > > > > > incubator
> > > > > > > > >> > > > > > > > > > >> project.
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >> Please focus on the packaging during
> > the
> > > > next
> > > > > > > > weeks.
> > > > > > > > >> > Since
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > work
> > > > > > > > >> > > > > > > > > will
> > > > > > > > >> > > > > > > > > > >> be
> > > > > > > > >> > > > > > > > > > >> part of Bigtop, related discussions
> and
> > > > JIRAs
> > > > > > > > should
> > > > > > > > >> > also
> > > > > > > > >> > > be
> > > > > > > > >> > > > > > > there.
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >> Would be good to have the packaging
> in
> > > > place
> > > > > by
> > > > > > > end
> > > > > > > > >> Feb.
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >> Thanks,
> > > > > > > > >> > > > > > > > > > >> Thomas
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM,
> > Chinmay
> > > > > > > Kolhatkar
> > > > > > > > <
> > > > > > > > >> > > > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > >> > > > > > > > > > >> wrote:
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >> > Hi All,
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > We're planning a work on adding
> > Apache
> > > > Apex
> > > > > > as
> > > > > > > a
> > > > > > > > >> > > component
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > > Apache
> > > > > > > > >> > > > > > > > > > >> > Bigtop.
> > > > > > > > >> > > > > > > > > > >> > Bigtop is the packaging system for
> > the
> > > > > Apache
> > > > > > > big
> > > > > > > > >> data
> > > > > > > > >> > > > > > > ecosystem.
> > > > > > > > >> > > > > > > > > > >> Several
> > > > > > > > >> > > > > > > > > > >> > Hadoop distros use it, most
> recently
> > > EMR.
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > Here is the tracking Jira task in
> > > > APEXCORE
> > > > > > for
> > > > > > > > the
> > > > > > > > >> > same:
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > Proposed plan of execution is as
> > > follows:
> > > > > > > > >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > >> > > > > > > > > > >> > Apache bigtop has some hard and
> soft
> > > > > > > expectation
> > > > > > > > >> for
> > > > > > > > >> > new
> > > > > > > > >> > > > > > > > components
> > > > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > > > > > >> get
> > > > > > > > >> > > > > > > > > > >> > integrated into Bigtop.
> > > > > > > > >> > > > > > > > > > >> > Here is the list of it:
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > Most of them seems to be standard
> ASF
> > > > based
> > > > > > > > >> > > requirements,
> > > > > > > > >> > > > > but
> > > > > > > > >> > > > > > > few
> > > > > > > > >> > > > > > > > > need
> > > > > > > > >> > > > > > > > > > >> to
> > > > > > > > >> > > > > > > > > > >> > be checked for:
> > > > > > > > >> > > > > > > > > > >> > 1. Software projects are expected
> to
> > be
> > > > > > > Licensed
> > > > > > > > >> under
> > > > > > > > >> > > > > Apache
> > > > > > > > >> > > > > > > > > License,
> > > > > > > > >> > > > > > > > > > >> > Version 2.0 (and their dependencies
> > are
> > > > > > > expected
> > > > > > > > >> to be
> > > > > > > > >> > > > > > > compatible
> > > > > > > > >> > > > > > > > > with
> > > > > > > > >> > > > > > > > > > >> this
> > > > > > > > >> > > > > > > > > > >> > license)
> > > > > > > > >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but
> need
> > to
> > > > > check
> > > > > > > if
> > > > > > > > >> > > > > dependencies
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > > > Apex
> > > > > > > > >> > > > > > > > > > >> are
> > > > > > > > >> > > > > > > > > > >> > compatible with ASL 2.0. This I
> guess
> > > > would
> > > > > > be
> > > > > > > a
> > > > > > > > >> > > > > verification
> > > > > > > > >> > > > > > > > check.
> > > > > > > > >> > > > > > > > > > >> > 2. Software projects are expected
> to
> > be
> > > > > > > > compatible
> > > > > > > > >> > with
> > > > > > > > >> > > > all
> > > > > > > > >> > > > > of
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > > >> > supported platforms that Bigtop
> > > > > distribution
> > > > > > is
> > > > > > > > >> > > targeting
> > > > > > > > >> > > > > > > > > > >> >     - This needs verifying whether
> > our
> > > > > > software
> > > > > > > > >> runs
> > > > > > > > >> > > fine
> > > > > > > > >> > > > in
> > > > > > > > >> > > > > > > > > centos-6
> > > > > > > > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
> > > debian-8
> > > > > > > > >> > opensuse-13.2.
> > > > > > > > >> > > > > > > > > > >> > 3. What smoke tests that should be
> > > added
> > > > > for
> > > > > > > > >> > deployment.
> > > > > > > > >> > > > > > > > > > >> > 4. Identifying the test artifacts
> > which
> > > > > goes
> > > > > > > > beyond
> > > > > > > > >> > > smoke
> > > > > > > > >> > > > > test
> > > > > > > > >> > > > > > > > > > >> >     - These are basically the
> > > integration
> > > > > > tests
> > > > > > > > for
> > > > > > > > >> > > > > > verification
> > > > > > > > >> > > > > > > > > after
> > > > > > > > >> > > > > > > > > > >> the
> > > > > > > > >> > > > > > > > > > >> > deployment. This is a soft
> > requirement,
> > > > but
> > > > > > aim
> > > > > > > > is
> > > > > > > > >> to
> > > > > > > > >> > > > > achieve
> > > > > > > > >> > > > > > > this
> > > > > > > > >> > > > > > > > > as
> > > > > > > > >> > > > > > > > > > >> well
> > > > > > > > >> > > > > > > > > > >> > or at least have explanation why
> not
> > to
> > > > > > > include.
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > If there are any from the link
> which
> > > > > > explicitly
> > > > > > > > >> needs
> > > > > > > > >> > to
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > > > > checked
> > > > > > > > >> > > > > > > > > > >> other
> > > > > > > > >> > > > > > > > > > >> > than above 4, please let us know.
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > *Step 2) Adding Apex as component
> to
> > > > > Bigtop*
> > > > > > > > >> > > > > > > > > > >> > From one of the mail archive of
> > Bigtop,
> > > > it
> > > > > > was
> > > > > > > > >> learnt
> > > > > > > > >> > > that
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > > > bigtop
> > > > > > > > >> > > > > > > > > > >> > community want to see the addition
> of
> > > new
> > > > > > > > >> components
> > > > > > > > >> > in
> > > > > > > > >> > > > > > phases.
> > > > > > > > >> > > > > > > > Here
> > > > > > > > >> > > > > > > > > > are
> > > > > > > > >> > > > > > > > > > >> > the phases:
> > > > > > > > >> > > > > > > > > > >> > 1. Packaging
> > > > > > > > >> > > > > > > > > > >> >     - This needs creating of
> package
> > > i.e.
> > > > > > rpm &
> > > > > > > > deb
> > > > > > > > >> > > files.
> > > > > > > > >> > > > > > > > > > >> >     - documentations/READMEs,
> > LICENSE,
> > > > > > > DISCLAMER,
> > > > > > > > >> > NOTES
> > > > > > > > >> > > > etc
> > > > > > > > >> > > > > if
> > > > > > > > >> > > > > > > any
> > > > > > > > >> > > > > > > > > > >> needed.
> > > > > > > > >> > > > > > > > > > >> >     - Any documentation that need
> to
> > be
> > > > > added
> > > > > > > to
> > > > > > > > >> > > > > distribution
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > > our
> > > > > > > > >> > > > > > > > > > >> > software.
> > > > > > > > >> > > > > > > > > > >> >     - Any license information of
> > > > > dependencies
> > > > > > > > >> required
> > > > > > > > >> > > to
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > > > added
> > > > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > > > > > >> > package
> > > > > > > > >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > >> > > > > > > > > > >> >     - Adding smoke test for
> > packaging.
> > > > > > > > >> > > > > > > > > > >> > 3. Puppet recipes for automatic
> > > > deployment
> > > > > > and
> > > > > > > > >> > > > configuration
> > > > > > > > >> > > > > > > > > > >> >     - Add puppet recipes for
> > automatic
> > > > > > > deployment
> > > > > > > > >> > > > > > > > > > >> > 4. Integration tests
> > > > > > > > >> > > > > > > > > > >> >     - For verification of
> > deployments.
> > > > > > > > >> > > > > > > > > > >> > 5. license clearance:
> > > > > > > > >> > > > > > > > > > >> >     Run 'gradle rat' to make sure
> all
> > > new
> > > > > > stuff
> > > > > > > > is
> > > > > > > > >> > > > compliant
> > > > > > > > >> > > > > > > with
> > > > > > > > >> > > > > > > > > ASF
> > > > > > > > >> > > > > > > > > > >> > license requirements. If you add
> code
> > > > > > licenses
> > > > > > > > >> under
> > > > > > > > >> > > > > different
> > > > > > > > >> > > > > > > > > > licenses,
> > > > > > > > >> > > > > > > > > > >> > those would need to be listed in
> the
> > > > > NOTICE.
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > Please share your thoughts on the
> > > > approach.
> > > > > > > > >> > > > > > > > > > >> > We'll start corresponding
> > communication
> > > > on
> > > > > > > bigtop
> > > > > > > > >> > > mailing
> > > > > > > > >> > > > > list
> > > > > > > > >> > > > > > > as
> > > > > > > > >> > > > > > > > > > well.
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > We have some specific
> > > > questions/suggestions
> > > > > > > > >> related to
> > > > > > > > >> > > > what
> > > > > > > > >> > > > > > > should
> > > > > > > > >> > > > > > > > > be
> > > > > > > > >> > > > > > > > > > >> the
> > > > > > > > >> > > > > > > > > > >> > content of the package and what
> > should
> > > be
> > > > > the
> > > > > > > > smoke
> > > > > > > > >> > > tests,
> > > > > > > > >> > > > > but
> > > > > > > > >> > > > > > > in
> > > > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > > > > > >> > interest of not having too much
> > content
> > > > > here,
> > > > > > > > we'll
> > > > > > > > >> > put
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > > > questions
> > > > > > > > >> > > > > > > > > > >> as a
> > > > > > > > >> > > > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > Thanks,
> > > > > > > > >> > > > > > > > > > >> > Chinmay.
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

["Pradeep A. Dalvi" <pr...@apache.org>]

We may not need /usr/lib/apex/bin/dtcli (symlink to
/usr/lib/apex/bin/apex), when there are /usr/bin/apex & /usr/bin/dtcli
(which are in PATH).

On Mon, Feb 15, 2016 at 10:54 AM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> /usr/lib/apex/bin contains executables related to apex. dtcli, even though
> its a symlink, for current users its still an executable. Hence for the
> sake of completion, added dtcli link under /usr/lib/apex/bin.
>
> But if you think otherwise, please let me know.
>
> Thanks,
> Chinmay.
>
>
> On Mon, Feb 15, 2016 at 10:42 AM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
> > Why the dtcli link under lib?
> >
> >
> > On Sun, Feb 14, 2016 at 9:09 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Hmmm... I get your point. In development mode, it tries to find the
> > hadoop
> > > executable like following:
> > >
> > >
> > >
> > >
> >
> HADOOP_SEARCH_PATH="${HADOOP_PREFIX}/bin:${HADOOP_HOME}/bin:${PATH}:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:."
> > >   export DT_HADOOP=`PATH=${HADOOP_SEARCH_PATH} && command -v hadoop
> > > 2>/dev/null`
> > >
> > > I will check with bigtop community that what is the best way to ensure
> > > during installation that hadoop binary will be present in above search
> > path
> > > search.
> > >
> > > Also, I checked as you suggested, i.e. removed dt-env.sh from package
> and
> > > tried to run the dtcli... It works!!!
> > >
> > > Thank you for pointing that out Thomas.
> > >
> > > Now I agree with you that, we can get rid of conf/dt-env.sh.. As this
> was
> > > the only file in conf/, unless someone has anything to include in conf/
> > > folder, I'll remove that too then.
> > >
> > > Also, this would mean, only dt* that remains in package are dtcli links
> > to
> > > apex script added for backward compatibility.
> > > i.e following:
> > >
> > > /
> > > `-- usr
> > >     |-- bin
> > >     |   |-- apex -> ../lib/apex/bin/apex
> > >     |   `-- dtcli -> ../lib/apex/bin/apex
> > >     |-- lib
> > >     |   `-- apex
> > >     |       |-- bin
> > >     |       |   |-- apex   // This is the real script
> > >     |       |   `-- dtcli -> apex
> > >
> > >
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > >
> > > On Mon, Feb 15, 2016 at 10:22 AM, Thomas Weise <thomas@datatorrent.com
> >
> > > wrote:
> > >
> > > > I'm not convinced dt-env.sh is even needed. In development mode, the
> > file
> > > > doesn't exist but it will still find Hadoop.
> > > >
> > > > The absolute path to hadoop script was introduced to deal with the
> > layout
> > > > of specific distros. You will know whether that applies to Bigtop or
> > not.
> > > >
> > > > Thomas
> > > >
> > > > On Sun, Feb 14, 2016 at 8:16 PM, Chinmay Kolhatkar <
> chinmay@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hi Thomas,
> > > > >
> > > > > There are only 2 things that are required from dt-env.sh:
> > > > > 1. DT_HADOOP variable which is exported by dt-env.sh This variable
> > has
> > > > > absolute path for hadoop binary
> > > > > 2. A find_hadoop() method which find the path of hadoop binary.
> > > > >
> > > > > JAVA_HOME is not required for this.
> > > > >
> > > > > The idea is not to change the dtcli. Current dtcli (in apex
> > > > > 3.3.0-incubating src code), sources dt-env.sh for above 2
> > information.
> > > > > Hence included dt-env.sh
> > > > >
> > > > > If the community agrees, we can make dtcli more generic so that in
> > > future
> > > > > packing of bigtop, dt-env.sh can be completely excluded OR at least
> > > > change
> > > > > the name to apex-env.sh.
> > > > >
> > > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise <
> > thomas@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Chinmay,
> > > > > >
> > > > > > Can you please specify what settings in dt-env.sh are required
> in a
> > > > > bigtop
> > > > > > environment?
> > > > > >
> > > > > > Location of hadoop and JAVA_HOME should be provided by the stack
> > > > already?
> > > > > >
> > > > > > Thanks,
> > > > > > Thomas
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> > > chinmay@apache.org
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Here is a suggestion I have related to name change:
> > > > > > >
> > > > > > > 1. While packaging we change the name of "dtcli" script to
> > "apex".
> > > > This
> > > > > > is
> > > > > > > done so that in future when we change the name to "apex", users
> > of
> > > > > bigtop
> > > > > > > apex don't have to transition much.
> > > > > > > 2. We also keep a symlink named dtcli which points to apex
> > script.
> > > > This
> > > > > > is
> > > > > > > for backward compatibility.
> > > > > > > 3. We still keep conf/dt-env.sh for first integration in
> bigtop.
> > > This
> > > > > is
> > > > > > in
> > > > > > > the interest of not changing the content of dtcli or apex
> script.
> > > > > > > 4. In apex 3.4.0, we take care of all these naming related
> > changes
> > > > and
> > > > > > > update bigtop repository later to remove references to dtcli
> all
> > > > > > together.
> > > > > > >
> > > > > > > Please share your thoughts on above approach.
> > > > > > >
> > > > > > > Also, please share what could be the man page content for
> "apex".
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Chinmay.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> > > > chinmay@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yes.. It'll be build from source tar downloaded from one of
> the
> > > > > apache
> > > > > > > > mirror: apache.osuosl.org
> > > > > > > >
> > > > > > > > The reason for dt-env.sh is it's sourced from dtcli. This
> > leaves
> > > > us 3
> > > > > > > > options:
> > > > > > > > If we should not have any dt-env.sh, dtcli would need a
> change
> > > > while
> > > > > > > > packaging.
> > > > > > > >
> > > > > > > > But if we want to use dtcli as it is, then we would need
> > > dt-env.sh
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> > > > > thomas@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Looks good overall, though there shouldn't be any dt-env.sh
> > > > > > > >>
> > > > > > > >> What will the bigtop package be built from, the source tar?
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > > > > > chinmay@apache.org>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Hi Thomas,
> > > > > > > >> >
> > > > > > > >> > Thanks for the feedback.
> > > > > > > >> >
> > > > > > > >> > First, we're not changing the name anywhere. We'll follow
> > what
> > > > > > > >> currently is
> > > > > > > >> > in source tarball for 3.3.0-incubating version.
> > > > > > > >> >
> > > > > > > >> > Secondly, I've mentioned a directory structure below which
> > is
> > > > > inline
> > > > > > > >> with
> > > > > > > >> > other existing integrations with bigtop.
> > > > > > > >> >
> > > > > > > >> > The need for each file is as follows:
> > > > > > > >> > 1. bin/dtcli -> CLI for apex picked from
> > > > engine/src/main/scripts/
> > > > > of
> > > > > > > >> source
> > > > > > > >> > code apex 3.3.0-incubating.
> > > > > > > >> >
> > > > > > > >> > 2. conf/dt-env.sh -> This is source from dtcli. This
> > searches
> > > > for
> > > > > > > hadoop
> > > > > > > >> > binary path and exports an env variable for dtcli to use
> it.
> > > > > > > >> >
> > > > > > > >> > 3. lib/ -> Libraries for CLI. This list is extracted in a
> > > > similar
> > > > > > way
> > > > > > > to
> > > > > > > >> > how DT Community Edition finds dependency jars.
> > > > > > > >> > Please note that this is the first iteration list of jars.
> > I'm
> > > > > > trying
> > > > > > > to
> > > > > > > >> > narrow this down to only those which are really required.
> > > > > > > >> > The test that I'm running to check if dtcli runs fine with
> > > given
> > > > > > > >> dependency
> > > > > > > >> > is to launch+shutdown+kill+status for pi demo.
> > > > > > > >> >
> > > > > > > >> > Above is the directory structure which is required for CLI
> > to
> > > > > work.
> > > > > > > >> > What aniruddha and pradeep has mentioned are some
> additional
> > > > files
> > > > > > > which
> > > > > > > >> > makes the like of administrator easier. For eg. /etc/skel
> > > > > > > >> >
> > > > > > > >> > Please let me know if this is inline with what you're
> > > thinking.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Chinmay.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > .
> > > > > > > >> > |-- bin
> > > > > > > >> > |   `-- dtcli
> > > > > > > >> > |-- conf
> > > > > > > >> > |   `-- dt-env.sh
> > > > > > > >> > `-- lib
> > > > > > > >> >     |-- ant-1.9.2.jar
> > > > > > > >> >     |-- ant-launcher-1.9.2.jar
> > > > > > > >> >     |-- apex-api-3.3.0-incubating.jar
> > > > > > > >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > > > > > > >> >     |-- apex-common-3.3.0-incubating.jar
> > > > > > > >> >     |-- apex-engine.jar
> > > > > > > >> >     |-- apex-engine-tests.jar
> > > > > > > >> >     |-- async-http-client-1.7.20.jar
> > > > > > > >> >     |-- bval-core-0.5.jar
> > > > > > > >> >     |-- bval-jsr303-0.5.jar
> > > > > > > >> >     |-- commons-beanutils-1.8.3.jar
> > > > > > > >> >     |-- commons-codec-1.10.jar
> > > > > > > >> >     |-- commons-lang3-3.1.jar
> > > > > > > >> >     |-- commons-logging-1.1.3.jar
> > > > > > > >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > > > > > >> >     |-- hadoop-common-2.2.0-tests.jar
> > > > > > > >> >     |-- httpclient-4.3.5.jar
> > > > > > > >> >     |-- httpcore-4.3.2.jar
> > > > > > > >> >     |-- jackson-core-asl-1.9.2.jar
> > > > > > > >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > > > > > >> >     |-- javax.servlet-3.1.jar
> > > > > > > >> >     |-- javax.servlet-api-3.0.1.jar
> > > > > > > >> >     |-- jersey-apache-client4-1.9.jar
> > > > > > > >> >     |-- jersey-client-1.9.jar
> > > > > > > >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > > > > > >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > > > > > >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > > > > > >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > > > > > > >> >     |-- jline-2.11.jar
> > > > > > > >> >     |-- kryo-2.24.0.jar
> > > > > > > >> >     |-- mbassador-1.1.9.jar
> > > > > > > >> >     |-- minlog-1.2.jar
> > > > > > > >> >     |-- netlet-1.2.0.jar
> > > > > > > >> >     |-- netty-3.6.6.Final.jar
> > > > > > > >> >     |-- objenesis-2.1.jar
> > > > > > > >> >     |-- validation-api-1.1.0.Final.jar
> > > > > > > >> >     |-- xbean-asm5-shaded-4.3.jar
> > > > > > > >> >     `-- zip4j-1.3.2.jar
> > > > > > > >> >
> > > > > > > >> > 3 directories, 40 files
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > > > > > thomas@datatorrent.com
> > > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Chinmay,
> > > > > > > >> > >
> > > > > > > >> > > Before discussing where to put the files, let's make
> sure
> > > they
> > > > > are
> > > > > > > >> really
> > > > > > > >> > > needed for the operation of the CLI. As for names,
> > anything
> > > > new
> > > > > > > needs
> > > > > > > >> to
> > > > > > > >> > > reflect Apex in the name and should follow common
> > > conventions,
> > > > > > > >> especially
> > > > > > > >> > > in Bigtop where there are many existing integrations to
> > look
> > > > at.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Thomas
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > > > > > > >> > > chinmay@datatorrent.com>
> > > > > > > >> > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Really good points Pradeep and Aniruddha.
> > > > > > > >> > > >
> > > > > > > >> > > > 1. I believe we won't need to change the dtcli
> > considering
> > > > it
> > > > > > > works
> > > > > > > >> > with
> > > > > > > >> > > DT
> > > > > > > >> > > > Community edition. We can keep the directory structure
> > > > similar
> > > > > > to
> > > > > > > >> that.
> > > > > > > >> > > > dt-env.sh has variables which contains information
> > > required
> > > > > for
> > > > > > > >> dtcli
> > > > > > > >> > to
> > > > > > > >> > > > launch.
> > > > > > > >> > > >
> > > > > > > >> > > > 2. Let me check with bigtop community that whether
> they
> > > > > > facilitate
> > > > > > > >> the
> > > > > > > >> > > > installation of rpms/debs before user is created. In
> > > either
> > > > > > case,
> > > > > > > >> > current
> > > > > > > >> > > > dtcli creates a .dt folder in home directory. Also
> > before
> > > > > > putting
> > > > > > > >> > > anything
> > > > > > > >> > > > in /etc/skel we need to define what are the default
> > > contents
> > > > > > that
> > > > > > > >> > should
> > > > > > > >> > > go
> > > > > > > >> > > > to ~/.dt/ folder. If there is no defaults, probably we
> > > > should
> > > > > > not
> > > > > > > >> > > > explicitly add it in /etc/skel.
> > > > > > > >> > > >
> > > > > > > >> > > > 3.  /etc/profile.d approach looks nice. This way
> > contents
> > > of
> > > > > > > >> dt-env.sh
> > > > > > > >> > > are
> > > > > > > >> > > > present as env variables. But I see a catch there.
> > Adding
> > > > > > > dt-env.sh
> > > > > > > >> to
> > > > > > > >> > > > /etc/profile.d would make all the variables available
> at
> > > > > runtime
> > > > > > > all
> > > > > > > >> > the
> > > > > > > >> > > > time. I feel a little skeptical about that. Maybe a
> > > possible
> > > > > > > >> collision
> > > > > > > >> > > can
> > > > > > > >> > > > occur with other application vars.
> > > > > > > >> > > > Moreover, current dtcli does source
> "../conf/dt-env.sh"
> > > and
> > > > > > > >> > > > "~/.dt/dt-env.sh". So those variables are anyway
> > > available.
> > > > > > > >> > > >
> > > > > > > >> > > > @Aniruddha, the Jira and PR effort is happening at
> > Bigtop
> > > > > > > >> > > Jira/repository.
> > > > > > > >> > > > Packaging related code usually goes there. (That's
> what
> > > all
> > > > > the
> > > > > > > >> > > components
> > > > > > > >> > > > in bigtop does).
> > > > > > > >> > > > Having said that, once a PR is created, I'll be
> sharing
> > > the
> > > > > link
> > > > > > > of
> > > > > > > >> PR
> > > > > > > >> > > > here, so that apex community as well can review it.
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks,
> > > > > > > >> > > > Chinmay.
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > > > > > >> > > > aniruddha@datatorrent.com> wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > +1 on suggestions and approach.
> > > > > > > >> > > > > We may need to iron out details about exact paths
> etc.
> > > > > > > >> > > > > Which can be done on jira / PR comments.
> > > > > > > >> > > > > Is that right @dev?
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > > > > > >> > apache@pradeepdalvi.com>
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Inline comments...
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
> Kolhatkar <
> > > > > > > >> > > > > > chinmay@datatorrent.com>
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > @Thomas, Not all the jar present in DT community
> > > > edition
> > > > > > > will
> > > > > > > >> be
> > > > > > > >> > > > > included
> > > > > > > >> > > > > > > there.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Here is the list of jars I found from common
> > > > > dependencies
> > > > > > > >> between
> > > > > > > >> > > of
> > > > > > > >> > > > > apex
> > > > > > > >> > > > > > > and DT Community edition:
> > > > > > > >> > > > > > > netlet-1.2.0.jar
> > > > > > > >> > > > > > > kryo-2.24.0.jar
> > > > > > > >> > > > > > > jackson-core-asl-1.9.2.jar
> > > > > > > >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > > > >> > > > > > > async-http-client-1.7.20.jar
> > > > > > > >> > > > > > > netty-3.6.6.Final.jar
> > > > > > > >> > > > > > > validation-api-1.1.0.Final.jar
> > > > > > > >> > > > > > > bval-jsr303-0.5.jar
> > > > > > > >> > > > > > > bval-core-0.5.jar
> > > > > > > >> > > > > > > commons-lang3-3.1.jar
> > > > > > > >> > > > > > > commons-beanutils-1.8.3.jar
> > > > > > > >> > > > > > > httpclient-4.3.5.jar
> > > > > > > >> > > > > > > commons-codec-1.10.jar
> > > > > > > >> > > > > > > zip4j-1.3.2.jar
> > > > > > > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > > >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > > >> > > > > > > jersey-apache-client4-1.9.jar
> > > > > > > >> > > > > > > jline-2.11.jar
> > > > > > > >> > > > > > > ant-1.9.2.jar
> > > > > > > >> > > > > > > ant-launcher-1.9.2.jar
> > > > > > > >> > > > > > > mbassador-1.1.9.jar
> > > > > > > >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > > >> > > > > > > jackson-xc-1.9.2.jar
> > > > > > > >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Ofcourse, I'll be running some tests do check
> that
> > > > dtcli
> > > > > > > works
> > > > > > > >> > > > properly
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > > launch+shutdown+kill of apps with only these
> > > libraries
> > > > > > > >> present in
> > > > > > > >> > > > > > isolation
> > > > > > > >> > > > > > > without dependency on local m2.
> > > > > > > >> > > > > > > I'm believe that there are unwanted jars which
> are
> > > > used
> > > > > > for
> > > > > > > >> > compile
> > > > > > > >> > > > > time
> > > > > > > >> > > > > > > dependency and not runtime in above list which I
> > can
> > > > > drop
> > > > > > to
> > > > > > > >> keep
> > > > > > > >> > > > > package
> > > > > > > >> > > > > > > size minimal.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > @Anniruddha,
> > > > > > > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
> > dt-env.sh
> > > > is
> > > > > > > >> required.
> > > > > > > >> > > > Only
> > > > > > > >> > > > > > > required values I see in it are classpath. and
> > dtcli
> > > > is
> > > > > > > >> capable
> > > > > > > >> > of
> > > > > > > >> > > > > > building
> > > > > > > >> > > > > > > it on the fly.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how
> dtcli
> > > can
> > > > > > find
> > > > > > > >> the
> > > > > > > >> > jar
> > > > > > > >> > > > > path,
> > > > > > > >> > > > > > > dt-env.sh and all such conf files. If we need a
> > > change
> > > > > in
> > > > > > > >> dtcli,
> > > > > > > >> > > then
> > > > > > > >> > > > > > > community's opinion is required for whether
> dtcli
> > > > should
> > > > > > > >> change
> > > > > > > >> > in
> > > > > > > >> > > > our
> > > > > > > >> > > > > > repo
> > > > > > > >> > > > > > > or a copy of that with required changes exist in
> > > > bigtop
> > > > > > repo
> > > > > > > >> > until
> > > > > > > >> > > we
> > > > > > > >> > > > > > make
> > > > > > > >> > > > > > > the dtcli generic enough.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Can we have these file paths set using environment
> > > > > variables
> > > > > > > >> with
> > > > > > > >> > > some
> > > > > > > >> > > > > > default values in dtcli?
> > > > > > > >> > > > > > Then we can set such params in dt-env.sh.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > 3. dt-sited.xml -> How is this file different
> from
> > > > > > > >> dt-site.xml?
> > > > > > > >> > Not
> > > > > > > >> > > > > sure
> > > > > > > >> > > > > > if
> > > > > > > >> > > > > > > adding in /etc/skel would help. Files & Dirs
> from
> > > > > > /etc/skel
> > > > > > > >> are
> > > > > > > >> > > > copied
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > home of new user when useradd program is called.
> > But
> > > > for
> > > > > > > >> existing
> > > > > > > >> > > > users
> > > > > > > >> > > > > > > this won't be of any use. Moreover, dtcli
> creates
> > > > ~/.dt/
> > > > > > on
> > > > > > > >> the
> > > > > > > >> > fly
> > > > > > > >> > > > if
> > > > > > > >> > > > > > not
> > > > > > > >> > > > > > > encountered for the first time. Again correct me
> > if
> > > > I'm
> > > > > > > wrong.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Yes, you are right. However this would probably be
> > > > > necessary
> > > > > > > >> step
> > > > > > > >> > for
> > > > > > > >> > > > > > rpm/deb, as they may also get installed during OS
> > > > install
> > > > > > and
> > > > > > > >> > before
> > > > > > > >> > > > user
> > > > > > > >> > > > > > accounts were created.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not
> sure
> > > its
> > > > > the
> > > > > > > best
> > > > > > > >> > > idea.
> > > > > > > >> > > > > > We'll
> > > > > > > >> > > > > > > anyway put dtcli in location which is by default
> > > > present
> > > > > > in
> > > > > > > >> path
> > > > > > > >> > > i.e.
> > > > > > > >> > > > > > > /usr/bin.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > For other env variables specific to apex, I'll
> > prefer
> > > to
> > > > > use
> > > > > > > >> > > dt-env.sh
> > > > > > > >> > > > > and
> > > > > > > >> > > > > > > source it in dtcli rather than changing bashrc
> > > etc...
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall
> > be
> > > > the
> > > > > > > >> cleaner
> > > > > > > >> > way
> > > > > > > >> > > > to
> > > > > > > >> > > > > > achieve the same.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > > > Chinmay.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
> > Thombare
> > > <
> > > > > > > >> > > > > > > aniruddha@datatorrent.com> wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > Hi,
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > @Chinmay,
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > We need to consider following:
> > > > > > > >> > > > > > > > System wide default config files can be
> located
> > at
> > > > > > > following
> > > > > > > >> > > > > locations:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change
> > dtcli
> > > > > > > behaviour
> > > > > > > >> on
> > > > > > > >> > > how
> > > > > > > >> > > > it
> > > > > > > >> > > > > > > finds
> > > > > > > >> > > > > > > > those locations)
> > > > > > > >> > > > > > > > /etc/apex/dt-site.xml (we may have to change
> > dtcli
> > > > > > > >> behaviour on
> > > > > > > >> > > how
> > > > > > > >> > > > > it
> > > > > > > >> > > > > > > > finds those locations)
> > > > > > > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files
> (for
> > > new
> > > > > > users
> > > > > > > >> that
> > > > > > > >> > > will
> > > > > > > >> > > > > be
> > > > > > > >> > > > > > > > created in system in future.)
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > We may also have to modify bashrc /
> bashprofile
> > > for
> > > > > > > >> population
> > > > > > > >> > > any
> > > > > > > >> > > > > > > > variables that are required.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > @dev, please put in your comments /
> suggestion.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Thanks,
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Aniruddha
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise
> <
> > > > > > > >> > > > > thomas@datatorrent.com>
> > > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > -->
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> > > Kolhatkar
> > > > <
> > > > > > > >> > > > > > > > chinmay@datatorrent.com
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > Hello Everyone!!
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Continuing with packaging effort (rpm+deb)
> > of
> > > > > apex,
> > > > > > > here
> > > > > > > >> > are
> > > > > > > >> > > > some
> > > > > > > >> > > > > > > > > proposals
> > > > > > > >> > > > > > > > > > about package structure etc..
> > > > > > > >> > > > > > > > > > Before posting it on bbigtop mailing
> list, I
> > > > have
> > > > > > some
> > > > > > > >> > > question
> > > > > > > >> > > > > for
> > > > > > > >> > > > > > > > apex
> > > > > > > >> > > > > > > > > > community.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Proposed Directory structure of apex
> package
> > > for
> > > > > > both
> > > > > > > >> deb &
> > > > > > > >> > > > rpm:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > >> > > > > > > > > >
> /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > >> > > > > > > > > >
> > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > >> > > > > > > > > >
> /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > >> > > > > > > > > >
> > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > >> > > > > > > > > > <other dependent jars> = All the 3rd party
> > > jars
> > > > > > which
> > > > > > > >> are
> > > > > > > >> > > > > required
> > > > > > > >> > > > > > > for
> > > > > > > >> > > > > > > > > apex
> > > > > > > >> > > > > > > > > > to run. Usually the dependencies are
> > packaged
> > > as
> > > > > > part
> > > > > > > of
> > > > > > > >> > > > rpm/deb
> > > > > > > >> > > > > by
> > > > > > > >> > > > > > > any
> > > > > > > >> > > > > > > > > > software in bigtop.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Can you specify what those jars are in
> Bigtop
> > > > > context?
> > > > > > > >> Same
> > > > > > > >> > as
> > > > > > > >> > > > > > shipped
> > > > > > > >> > > > > > > > with
> > > > > > > >> > > > > > > > > DT community addition under lib/ ?
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > <package LICENSE> = Licenses of
> > corresponding
> > > of
> > > > > 3rd
> > > > > > > >> party
> > > > > > > >> > > jars
> > > > > > > >> > > > > > which
> > > > > > > >> > > > > > > > > needs
> > > > > > > >> > > > > > > > > > to included while packaging.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Questions related to this:
> > > > > > > >> > > > > > > > > > 1. Should we call the cli of apex as
> "apex"
> > > > > instead
> > > > > > of
> > > > > > > >> > > "dtcli"
> > > > > > > >> > > > in
> > > > > > > >> > > > > > > > bigtop
> > > > > > > >> > > > > > > > > > package?
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > I think we should keep the name until we are
> > > able
> > > > to
> > > > > > > >> change
> > > > > > > >> > it
> > > > > > > >> > > in
> > > > > > > >> > > > > > Apex.
> > > > > > > >> > > > > > > > > Otherwise this may get confusing.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > 2. I see that all softwares in bigtop have
> > man
> > > > > page
> > > > > > > for
> > > > > > > >> > their
> > > > > > > >> > > > > > > > > executables.
> > > > > > > >> > > > > > > > > > I think we should have it too for dtcli.
> Is
> > > > there
> > > > > > any
> > > > > > > >> > > > > documentation
> > > > > > > >> > > > > > > > > which I
> > > > > > > >> > > > > > > > > > can convert to man page? or can I use
> output
> > > of
> > > > > > "dtcli
> > > > > > > >> > > --help"?
> > > > > > > >> > > > > > > > > > 3. Do we want to call version of apex in
> > > Bigtop
> > > > as
> > > > > > > >> 3.3.0 OR
> > > > > > > >> > > > > > > > > > 3.3.0-incubating?
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Has to be -incubating
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > 4. Is it ok for apex package of bigtop to
> > > depend
> > > > > on
> > > > > > > >> 2.7.1
> > > > > > > >> > > > > version
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > > > bigtop hadoop? Any problems that we see
> with
> > > > this
> > > > > > > >> > dependency?
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Should work without changed. I thought we
> > > > certified
> > > > > > > >> against
> > > > > > > >> > > > 2.7.0?
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > 5. Following is the apache mirror from
> which
> > > > > bigtop
> > > > > > > will
> > > > > > > >> > pick
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > apex
> > > > > > > >> > > > > > > > > > source for compilation and packaging.
> Please
> > > > > correct
> > > > > > > if
> > > > > > > >> > > > > incorrect:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > >
> > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror
> > > used
> > > > by
> > > > > > all
> > > > > > > >> the
> > > > > > > >> > > > > > softwares
> > > > > > > >> > > > > > > in
> > > > > > > >> > > > > > > > > > bigtop.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Please share your opinion.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > > > Chinmay.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> > > > > Kolhatkar <
> > > > > > > >> > > > > > > > > > chinmay@datatorrent.com>
> > > > > > > >> > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > I've already started discussion on
> bigtop
> > > dev
> > > > > > > mailing
> > > > > > > >> > list
> > > > > > > >> > > > for
> > > > > > > >> > > > > > > > > > > integration. Also created a Jira for the
> > > same.
> > > > > For
> > > > > > > >> this
> > > > > > > >> > > > > > communities
> > > > > > > >> > > > > > > > > > > reference, here is the bigtop Jira:
> > > > BIGTOP-2313.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas
> > > Weise <
> > > > > > > >> > > > > > > thomas@datatorrent.com
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >> Chinmay,
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >> I don't see anything under
> prerequisites
> > > that
> > > > > > would
> > > > > > > >> be a
> > > > > > > >> > > > > > problem.
> > > > > > > >> > > > > > > We
> > > > > > > >> > > > > > > > > > >> looked
> > > > > > > >> > > > > > > > > > >> at the ASF licencing compatibility as
> > part
> > > of
> > > > > > > >> becoming
> > > > > > > >> > an
> > > > > > > >> > > > > > > incubator
> > > > > > > >> > > > > > > > > > >> project.
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >> Please focus on the packaging during
> the
> > > next
> > > > > > > weeks.
> > > > > > > >> > Since
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > work
> > > > > > > >> > > > > > > > > will
> > > > > > > >> > > > > > > > > > >> be
> > > > > > > >> > > > > > > > > > >> part of Bigtop, related discussions and
> > > JIRAs
> > > > > > > should
> > > > > > > >> > also
> > > > > > > >> > > be
> > > > > > > >> > > > > > > there.
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >> Would be good to have the packaging in
> > > place
> > > > by
> > > > > > end
> > > > > > > >> Feb.
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >> Thanks,
> > > > > > > >> > > > > > > > > > >> Thomas
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM,
> Chinmay
> > > > > > Kolhatkar
> > > > > > > <
> > > > > > > >> > > > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > >> > > > > > > > > > >> wrote:
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >> > Hi All,
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > We're planning a work on adding
> Apache
> > > Apex
> > > > > as
> > > > > > a
> > > > > > > >> > > component
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > > Apache
> > > > > > > >> > > > > > > > > > >> > Bigtop.
> > > > > > > >> > > > > > > > > > >> > Bigtop is the packaging system for
> the
> > > > Apache
> > > > > > big
> > > > > > > >> data
> > > > > > > >> > > > > > > ecosystem.
> > > > > > > >> > > > > > > > > > >> Several
> > > > > > > >> > > > > > > > > > >> > Hadoop distros use it, most recently
> > EMR.
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > Here is the tracking Jira task in
> > > APEXCORE
> > > > > for
> > > > > > > the
> > > > > > > >> > same:
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > Proposed plan of execution is as
> > follows:
> > > > > > > >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> > > > > > expectation
> > > > > > > >> for
> > > > > > > >> > new
> > > > > > > >> > > > > > > > components
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > >> get
> > > > > > > >> > > > > > > > > > >> > integrated into Bigtop.
> > > > > > > >> > > > > > > > > > >> > Here is the list of it:
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > Most of them seems to be standard ASF
> > > based
> > > > > > > >> > > requirements,
> > > > > > > >> > > > > but
> > > > > > > >> > > > > > > few
> > > > > > > >> > > > > > > > > need
> > > > > > > >> > > > > > > > > > >> to
> > > > > > > >> > > > > > > > > > >> > be checked for:
> > > > > > > >> > > > > > > > > > >> > 1. Software projects are expected to
> be
> > > > > > Licensed
> > > > > > > >> under
> > > > > > > >> > > > > Apache
> > > > > > > >> > > > > > > > > License,
> > > > > > > >> > > > > > > > > > >> > Version 2.0 (and their dependencies
> are
> > > > > > expected
> > > > > > > >> to be
> > > > > > > >> > > > > > > compatible
> > > > > > > >> > > > > > > > > with
> > > > > > > >> > > > > > > > > > >> this
> > > > > > > >> > > > > > > > > > >> > license)
> > > > > > > >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need
> to
> > > > check
> > > > > > if
> > > > > > > >> > > > > dependencies
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > > Apex
> > > > > > > >> > > > > > > > > > >> are
> > > > > > > >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess
> > > would
> > > > > be
> > > > > > a
> > > > > > > >> > > > > verification
> > > > > > > >> > > > > > > > check.
> > > > > > > >> > > > > > > > > > >> > 2. Software projects are expected to
> be
> > > > > > > compatible
> > > > > > > >> > with
> > > > > > > >> > > > all
> > > > > > > >> > > > > of
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > >> > supported platforms that Bigtop
> > > > distribution
> > > > > is
> > > > > > > >> > > targeting
> > > > > > > >> > > > > > > > > > >> >     - This needs verifying whether
> our
> > > > > software
> > > > > > > >> runs
> > > > > > > >> > > fine
> > > > > > > >> > > > in
> > > > > > > >> > > > > > > > > centos-6
> > > > > > > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
> > debian-8
> > > > > > > >> > opensuse-13.2.
> > > > > > > >> > > > > > > > > > >> > 3. What smoke tests that should be
> > added
> > > > for
> > > > > > > >> > deployment.
> > > > > > > >> > > > > > > > > > >> > 4. Identifying the test artifacts
> which
> > > > goes
> > > > > > > beyond
> > > > > > > >> > > smoke
> > > > > > > >> > > > > test
> > > > > > > >> > > > > > > > > > >> >     - These are basically the
> > integration
> > > > > tests
> > > > > > > for
> > > > > > > >> > > > > > verification
> > > > > > > >> > > > > > > > > after
> > > > > > > >> > > > > > > > > > >> the
> > > > > > > >> > > > > > > > > > >> > deployment. This is a soft
> requirement,
> > > but
> > > > > aim
> > > > > > > is
> > > > > > > >> to
> > > > > > > >> > > > > achieve
> > > > > > > >> > > > > > > this
> > > > > > > >> > > > > > > > > as
> > > > > > > >> > > > > > > > > > >> well
> > > > > > > >> > > > > > > > > > >> > or at least have explanation why not
> to
> > > > > > include.
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > If there are any from the link which
> > > > > explicitly
> > > > > > > >> needs
> > > > > > > >> > to
> > > > > > > >> > > > be
> > > > > > > >> > > > > > > > checked
> > > > > > > >> > > > > > > > > > >> other
> > > > > > > >> > > > > > > > > > >> > than above 4, please let us know.
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > *Step 2) Adding Apex as component to
> > > > Bigtop*
> > > > > > > >> > > > > > > > > > >> > From one of the mail archive of
> Bigtop,
> > > it
> > > > > was
> > > > > > > >> learnt
> > > > > > > >> > > that
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > > > bigtop
> > > > > > > >> > > > > > > > > > >> > community want to see the addition of
> > new
> > > > > > > >> components
> > > > > > > >> > in
> > > > > > > >> > > > > > phases.
> > > > > > > >> > > > > > > > Here
> > > > > > > >> > > > > > > > > > are
> > > > > > > >> > > > > > > > > > >> > the phases:
> > > > > > > >> > > > > > > > > > >> > 1. Packaging
> > > > > > > >> > > > > > > > > > >> >     - This needs creating of package
> > i.e.
> > > > > rpm &
> > > > > > > deb
> > > > > > > >> > > files.
> > > > > > > >> > > > > > > > > > >> >     - documentations/READMEs,
> LICENSE,
> > > > > > DISCLAMER,
> > > > > > > >> > NOTES
> > > > > > > >> > > > etc
> > > > > > > >> > > > > if
> > > > > > > >> > > > > > > any
> > > > > > > >> > > > > > > > > > >> needed.
> > > > > > > >> > > > > > > > > > >> >     - Any documentation that need to
> be
> > > > added
> > > > > > to
> > > > > > > >> > > > > distribution
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > our
> > > > > > > >> > > > > > > > > > >> > software.
> > > > > > > >> > > > > > > > > > >> >     - Any license information of
> > > > dependencies
> > > > > > > >> required
> > > > > > > >> > > to
> > > > > > > >> > > > be
> > > > > > > >> > > > > > > added
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > >> > package
> > > > > > > >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > >> > > > > > > > > > >> >     - Adding smoke test for
> packaging.
> > > > > > > >> > > > > > > > > > >> > 3. Puppet recipes for automatic
> > > deployment
> > > > > and
> > > > > > > >> > > > configuration
> > > > > > > >> > > > > > > > > > >> >     - Add puppet recipes for
> automatic
> > > > > > deployment
> > > > > > > >> > > > > > > > > > >> > 4. Integration tests
> > > > > > > >> > > > > > > > > > >> >     - For verification of
> deployments.
> > > > > > > >> > > > > > > > > > >> > 5. license clearance:
> > > > > > > >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all
> > new
> > > > > stuff
> > > > > > > is
> > > > > > > >> > > > compliant
> > > > > > > >> > > > > > > with
> > > > > > > >> > > > > > > > > ASF
> > > > > > > >> > > > > > > > > > >> > license requirements. If you add code
> > > > > licenses
> > > > > > > >> under
> > > > > > > >> > > > > different
> > > > > > > >> > > > > > > > > > licenses,
> > > > > > > >> > > > > > > > > > >> > those would need to be listed in the
> > > > NOTICE.
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > Please share your thoughts on the
> > > approach.
> > > > > > > >> > > > > > > > > > >> > We'll start corresponding
> communication
> > > on
> > > > > > bigtop
> > > > > > > >> > > mailing
> > > > > > > >> > > > > list
> > > > > > > >> > > > > > > as
> > > > > > > >> > > > > > > > > > well.
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > We have some specific
> > > questions/suggestions
> > > > > > > >> related to
> > > > > > > >> > > > what
> > > > > > > >> > > > > > > should
> > > > > > > >> > > > > > > > > be
> > > > > > > >> > > > > > > > > > >> the
> > > > > > > >> > > > > > > > > > >> > content of the package and what
> should
> > be
> > > > the
> > > > > > > smoke
> > > > > > > >> > > tests,
> > > > > > > >> > > > > but
> > > > > > > >> > > > > > > in
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > > >> > interest of not having too much
> content
> > > > here,
> > > > > > > we'll
> > > > > > > >> > put
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > > > questions
> > > > > > > >> > > > > > > > > > >> as a
> > > > > > > >> > > > > > > > > > >> > separate mail in this mailthread.
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > Thanks,
> > > > > > > >> > > > > > > > > > >> > Chinmay.
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

/usr/lib/apex/bin contains executables related to apex. dtcli, even though
its a symlink, for current users its still an executable. Hence for the
sake of completion, added dtcli link under /usr/lib/apex/bin.

But if you think otherwise, please let me know.

Thanks,
Chinmay.


On Mon, Feb 15, 2016 at 10:42 AM, Thomas Weise <th...@datatorrent.com>
wrote:

> Why the dtcli link under lib?
>
>
> On Sun, Feb 14, 2016 at 9:09 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Hmmm... I get your point. In development mode, it tries to find the
> hadoop
> > executable like following:
> >
> >
> >
> >
> HADOOP_SEARCH_PATH="${HADOOP_PREFIX}/bin:${HADOOP_HOME}/bin:${PATH}:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:."
> >   export DT_HADOOP=`PATH=${HADOOP_SEARCH_PATH} && command -v hadoop
> > 2>/dev/null`
> >
> > I will check with bigtop community that what is the best way to ensure
> > during installation that hadoop binary will be present in above search
> path
> > search.
> >
> > Also, I checked as you suggested, i.e. removed dt-env.sh from package and
> > tried to run the dtcli... It works!!!
> >
> > Thank you for pointing that out Thomas.
> >
> > Now I agree with you that, we can get rid of conf/dt-env.sh.. As this was
> > the only file in conf/, unless someone has anything to include in conf/
> > folder, I'll remove that too then.
> >
> > Also, this would mean, only dt* that remains in package are dtcli links
> to
> > apex script added for backward compatibility.
> > i.e following:
> >
> > /
> > `-- usr
> >     |-- bin
> >     |   |-- apex -> ../lib/apex/bin/apex
> >     |   `-- dtcli -> ../lib/apex/bin/apex
> >     |-- lib
> >     |   `-- apex
> >     |       |-- bin
> >     |       |   |-- apex   // This is the real script
> >     |       |   `-- dtcli -> apex
> >
> >
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> > On Mon, Feb 15, 2016 at 10:22 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> > > I'm not convinced dt-env.sh is even needed. In development mode, the
> file
> > > doesn't exist but it will still find Hadoop.
> > >
> > > The absolute path to hadoop script was introduced to deal with the
> layout
> > > of specific distros. You will know whether that applies to Bigtop or
> not.
> > >
> > > Thomas
> > >
> > > On Sun, Feb 14, 2016 at 8:16 PM, Chinmay Kolhatkar <chinmay@apache.org
> >
> > > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > There are only 2 things that are required from dt-env.sh:
> > > > 1. DT_HADOOP variable which is exported by dt-env.sh This variable
> has
> > > > absolute path for hadoop binary
> > > > 2. A find_hadoop() method which find the path of hadoop binary.
> > > >
> > > > JAVA_HOME is not required for this.
> > > >
> > > > The idea is not to change the dtcli. Current dtcli (in apex
> > > > 3.3.0-incubating src code), sources dt-env.sh for above 2
> information.
> > > > Hence included dt-env.sh
> > > >
> > > > If the community agrees, we can make dtcli more generic so that in
> > future
> > > > packing of bigtop, dt-env.sh can be completely excluded OR at least
> > > change
> > > > the name to apex-env.sh.
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > >
> > > > On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise <
> thomas@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Chinmay,
> > > > >
> > > > > Can you please specify what settings in dt-env.sh are required in a
> > > > bigtop
> > > > > environment?
> > > > >
> > > > > Location of hadoop and JAVA_HOME should be provided by the stack
> > > already?
> > > > >
> > > > > Thanks,
> > > > > Thomas
> > > > >
> > > > >
> > > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> > chinmay@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Here is a suggestion I have related to name change:
> > > > > >
> > > > > > 1. While packaging we change the name of "dtcli" script to
> "apex".
> > > This
> > > > > is
> > > > > > done so that in future when we change the name to "apex", users
> of
> > > > bigtop
> > > > > > apex don't have to transition much.
> > > > > > 2. We also keep a symlink named dtcli which points to apex
> script.
> > > This
> > > > > is
> > > > > > for backward compatibility.
> > > > > > 3. We still keep conf/dt-env.sh for first integration in bigtop.
> > This
> > > > is
> > > > > in
> > > > > > the interest of not changing the content of dtcli or apex script.
> > > > > > 4. In apex 3.4.0, we take care of all these naming related
> changes
> > > and
> > > > > > update bigtop repository later to remove references to dtcli all
> > > > > together.
> > > > > >
> > > > > > Please share your thoughts on above approach.
> > > > > >
> > > > > > Also, please share what could be the man page content for "apex".
> > > > > >
> > > > > > Thanks,
> > > > > > Chinmay.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> > > chinmay@apache.org
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Yes.. It'll be build from source tar downloaded from one of the
> > > > apache
> > > > > > > mirror: apache.osuosl.org
> > > > > > >
> > > > > > > The reason for dt-env.sh is it's sourced from dtcli. This
> leaves
> > > us 3
> > > > > > > options:
> > > > > > > If we should not have any dt-env.sh, dtcli would need a change
> > > while
> > > > > > > packaging.
> > > > > > >
> > > > > > > But if we want to use dtcli as it is, then we would need
> > dt-env.sh
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> > > > thomas@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Looks good overall, though there shouldn't be any dt-env.sh
> > > > > > >>
> > > > > > >> What will the bigtop package be built from, the source tar?
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > > > > chinmay@apache.org>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Thomas,
> > > > > > >> >
> > > > > > >> > Thanks for the feedback.
> > > > > > >> >
> > > > > > >> > First, we're not changing the name anywhere. We'll follow
> what
> > > > > > >> currently is
> > > > > > >> > in source tarball for 3.3.0-incubating version.
> > > > > > >> >
> > > > > > >> > Secondly, I've mentioned a directory structure below which
> is
> > > > inline
> > > > > > >> with
> > > > > > >> > other existing integrations with bigtop.
> > > > > > >> >
> > > > > > >> > The need for each file is as follows:
> > > > > > >> > 1. bin/dtcli -> CLI for apex picked from
> > > engine/src/main/scripts/
> > > > of
> > > > > > >> source
> > > > > > >> > code apex 3.3.0-incubating.
> > > > > > >> >
> > > > > > >> > 2. conf/dt-env.sh -> This is source from dtcli. This
> searches
> > > for
> > > > > > hadoop
> > > > > > >> > binary path and exports an env variable for dtcli to use it.
> > > > > > >> >
> > > > > > >> > 3. lib/ -> Libraries for CLI. This list is extracted in a
> > > similar
> > > > > way
> > > > > > to
> > > > > > >> > how DT Community Edition finds dependency jars.
> > > > > > >> > Please note that this is the first iteration list of jars.
> I'm
> > > > > trying
> > > > > > to
> > > > > > >> > narrow this down to only those which are really required.
> > > > > > >> > The test that I'm running to check if dtcli runs fine with
> > given
> > > > > > >> dependency
> > > > > > >> > is to launch+shutdown+kill+status for pi demo.
> > > > > > >> >
> > > > > > >> > Above is the directory structure which is required for CLI
> to
> > > > work.
> > > > > > >> > What aniruddha and pradeep has mentioned are some additional
> > > files
> > > > > > which
> > > > > > >> > makes the like of administrator easier. For eg. /etc/skel
> > > > > > >> >
> > > > > > >> > Please let me know if this is inline with what you're
> > thinking.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Chinmay.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > .
> > > > > > >> > |-- bin
> > > > > > >> > |   `-- dtcli
> > > > > > >> > |-- conf
> > > > > > >> > |   `-- dt-env.sh
> > > > > > >> > `-- lib
> > > > > > >> >     |-- ant-1.9.2.jar
> > > > > > >> >     |-- ant-launcher-1.9.2.jar
> > > > > > >> >     |-- apex-api-3.3.0-incubating.jar
> > > > > > >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > > > > > >> >     |-- apex-common-3.3.0-incubating.jar
> > > > > > >> >     |-- apex-engine.jar
> > > > > > >> >     |-- apex-engine-tests.jar
> > > > > > >> >     |-- async-http-client-1.7.20.jar
> > > > > > >> >     |-- bval-core-0.5.jar
> > > > > > >> >     |-- bval-jsr303-0.5.jar
> > > > > > >> >     |-- commons-beanutils-1.8.3.jar
> > > > > > >> >     |-- commons-codec-1.10.jar
> > > > > > >> >     |-- commons-lang3-3.1.jar
> > > > > > >> >     |-- commons-logging-1.1.3.jar
> > > > > > >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > > > > >> >     |-- hadoop-common-2.2.0-tests.jar
> > > > > > >> >     |-- httpclient-4.3.5.jar
> > > > > > >> >     |-- httpcore-4.3.2.jar
> > > > > > >> >     |-- jackson-core-asl-1.9.2.jar
> > > > > > >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > > > > >> >     |-- javax.servlet-3.1.jar
> > > > > > >> >     |-- javax.servlet-api-3.0.1.jar
> > > > > > >> >     |-- jersey-apache-client4-1.9.jar
> > > > > > >> >     |-- jersey-client-1.9.jar
> > > > > > >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > > > > >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > > > > >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > > > > >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > > > > > >> >     |-- jline-2.11.jar
> > > > > > >> >     |-- kryo-2.24.0.jar
> > > > > > >> >     |-- mbassador-1.1.9.jar
> > > > > > >> >     |-- minlog-1.2.jar
> > > > > > >> >     |-- netlet-1.2.0.jar
> > > > > > >> >     |-- netty-3.6.6.Final.jar
> > > > > > >> >     |-- objenesis-2.1.jar
> > > > > > >> >     |-- validation-api-1.1.0.Final.jar
> > > > > > >> >     |-- xbean-asm5-shaded-4.3.jar
> > > > > > >> >     `-- zip4j-1.3.2.jar
> > > > > > >> >
> > > > > > >> > 3 directories, 40 files
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > > > > thomas@datatorrent.com
> > > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Chinmay,
> > > > > > >> > >
> > > > > > >> > > Before discussing where to put the files, let's make sure
> > they
> > > > are
> > > > > > >> really
> > > > > > >> > > needed for the operation of the CLI. As for names,
> anything
> > > new
> > > > > > needs
> > > > > > >> to
> > > > > > >> > > reflect Apex in the name and should follow common
> > conventions,
> > > > > > >> especially
> > > > > > >> > > in Bigtop where there are many existing integrations to
> look
> > > at.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Thomas
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > > > > > >> > > chinmay@datatorrent.com>
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Really good points Pradeep and Aniruddha.
> > > > > > >> > > >
> > > > > > >> > > > 1. I believe we won't need to change the dtcli
> considering
> > > it
> > > > > > works
> > > > > > >> > with
> > > > > > >> > > DT
> > > > > > >> > > > Community edition. We can keep the directory structure
> > > similar
> > > > > to
> > > > > > >> that.
> > > > > > >> > > > dt-env.sh has variables which contains information
> > required
> > > > for
> > > > > > >> dtcli
> > > > > > >> > to
> > > > > > >> > > > launch.
> > > > > > >> > > >
> > > > > > >> > > > 2. Let me check with bigtop community that whether they
> > > > > facilitate
> > > > > > >> the
> > > > > > >> > > > installation of rpms/debs before user is created. In
> > either
> > > > > case,
> > > > > > >> > current
> > > > > > >> > > > dtcli creates a .dt folder in home directory. Also
> before
> > > > > putting
> > > > > > >> > > anything
> > > > > > >> > > > in /etc/skel we need to define what are the default
> > contents
> > > > > that
> > > > > > >> > should
> > > > > > >> > > go
> > > > > > >> > > > to ~/.dt/ folder. If there is no defaults, probably we
> > > should
> > > > > not
> > > > > > >> > > > explicitly add it in /etc/skel.
> > > > > > >> > > >
> > > > > > >> > > > 3.  /etc/profile.d approach looks nice. This way
> contents
> > of
> > > > > > >> dt-env.sh
> > > > > > >> > > are
> > > > > > >> > > > present as env variables. But I see a catch there.
> Adding
> > > > > > dt-env.sh
> > > > > > >> to
> > > > > > >> > > > /etc/profile.d would make all the variables available at
> > > > runtime
> > > > > > all
> > > > > > >> > the
> > > > > > >> > > > time. I feel a little skeptical about that. Maybe a
> > possible
> > > > > > >> collision
> > > > > > >> > > can
> > > > > > >> > > > occur with other application vars.
> > > > > > >> > > > Moreover, current dtcli does source "../conf/dt-env.sh"
> > and
> > > > > > >> > > > "~/.dt/dt-env.sh". So those variables are anyway
> > available.
> > > > > > >> > > >
> > > > > > >> > > > @Aniruddha, the Jira and PR effort is happening at
> Bigtop
> > > > > > >> > > Jira/repository.
> > > > > > >> > > > Packaging related code usually goes there. (That's what
> > all
> > > > the
> > > > > > >> > > components
> > > > > > >> > > > in bigtop does).
> > > > > > >> > > > Having said that, once a PR is created, I'll be sharing
> > the
> > > > link
> > > > > > of
> > > > > > >> PR
> > > > > > >> > > > here, so that apex community as well can review it.
> > > > > > >> > > >
> > > > > > >> > > > Thanks,
> > > > > > >> > > > Chinmay.
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > > > > >> > > > aniruddha@datatorrent.com> wrote:
> > > > > > >> > > >
> > > > > > >> > > > > +1 on suggestions and approach.
> > > > > > >> > > > > We may need to iron out details about exact paths etc.
> > > > > > >> > > > > Which can be done on jira / PR comments.
> > > > > > >> > > > > Is that right @dev?
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > > > > >> > apache@pradeepdalvi.com>
> > > > > > >> > > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Inline comments...
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > > > >> > > > > > chinmay@datatorrent.com>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > @Thomas, Not all the jar present in DT community
> > > edition
> > > > > > will
> > > > > > >> be
> > > > > > >> > > > > included
> > > > > > >> > > > > > > there.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Here is the list of jars I found from common
> > > > dependencies
> > > > > > >> between
> > > > > > >> > > of
> > > > > > >> > > > > apex
> > > > > > >> > > > > > > and DT Community edition:
> > > > > > >> > > > > > > netlet-1.2.0.jar
> > > > > > >> > > > > > > kryo-2.24.0.jar
> > > > > > >> > > > > > > jackson-core-asl-1.9.2.jar
> > > > > > >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > > >> > > > > > > async-http-client-1.7.20.jar
> > > > > > >> > > > > > > netty-3.6.6.Final.jar
> > > > > > >> > > > > > > validation-api-1.1.0.Final.jar
> > > > > > >> > > > > > > bval-jsr303-0.5.jar
> > > > > > >> > > > > > > bval-core-0.5.jar
> > > > > > >> > > > > > > commons-lang3-3.1.jar
> > > > > > >> > > > > > > commons-beanutils-1.8.3.jar
> > > > > > >> > > > > > > httpclient-4.3.5.jar
> > > > > > >> > > > > > > commons-codec-1.10.jar
> > > > > > >> > > > > > > zip4j-1.3.2.jar
> > > > > > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > >> > > > > > > jersey-apache-client4-1.9.jar
> > > > > > >> > > > > > > jline-2.11.jar
> > > > > > >> > > > > > > ant-1.9.2.jar
> > > > > > >> > > > > > > ant-launcher-1.9.2.jar
> > > > > > >> > > > > > > mbassador-1.1.9.jar
> > > > > > >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > >> > > > > > > jackson-xc-1.9.2.jar
> > > > > > >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Ofcourse, I'll be running some tests do check that
> > > dtcli
> > > > > > works
> > > > > > >> > > > properly
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > launch+shutdown+kill of apps with only these
> > libraries
> > > > > > >> present in
> > > > > > >> > > > > > isolation
> > > > > > >> > > > > > > without dependency on local m2.
> > > > > > >> > > > > > > I'm believe that there are unwanted jars which are
> > > used
> > > > > for
> > > > > > >> > compile
> > > > > > >> > > > > time
> > > > > > >> > > > > > > dependency and not runtime in above list which I
> can
> > > > drop
> > > > > to
> > > > > > >> keep
> > > > > > >> > > > > package
> > > > > > >> > > > > > > size minimal.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > @Anniruddha,
> > > > > > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
> dt-env.sh
> > > is
> > > > > > >> required.
> > > > > > >> > > > Only
> > > > > > >> > > > > > > required values I see in it are classpath. and
> dtcli
> > > is
> > > > > > >> capable
> > > > > > >> > of
> > > > > > >> > > > > > building
> > > > > > >> > > > > > > it on the fly.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli
> > can
> > > > > find
> > > > > > >> the
> > > > > > >> > jar
> > > > > > >> > > > > path,
> > > > > > >> > > > > > > dt-env.sh and all such conf files. If we need a
> > change
> > > > in
> > > > > > >> dtcli,
> > > > > > >> > > then
> > > > > > >> > > > > > > community's opinion is required for whether dtcli
> > > should
> > > > > > >> change
> > > > > > >> > in
> > > > > > >> > > > our
> > > > > > >> > > > > > repo
> > > > > > >> > > > > > > or a copy of that with required changes exist in
> > > bigtop
> > > > > repo
> > > > > > >> > until
> > > > > > >> > > we
> > > > > > >> > > > > > make
> > > > > > >> > > > > > > the dtcli generic enough.
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > Can we have these file paths set using environment
> > > > variables
> > > > > > >> with
> > > > > > >> > > some
> > > > > > >> > > > > > default values in dtcli?
> > > > > > >> > > > > > Then we can set such params in dt-env.sh.
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > > 3. dt-sited.xml -> How is this file different from
> > > > > > >> dt-site.xml?
> > > > > > >> > Not
> > > > > > >> > > > > sure
> > > > > > >> > > > > > if
> > > > > > >> > > > > > > adding in /etc/skel would help. Files & Dirs from
> > > > > /etc/skel
> > > > > > >> are
> > > > > > >> > > > copied
> > > > > > >> > > > > to
> > > > > > >> > > > > > > home of new user when useradd program is called.
> But
> > > for
> > > > > > >> existing
> > > > > > >> > > > users
> > > > > > >> > > > > > > this won't be of any use. Moreover, dtcli creates
> > > ~/.dt/
> > > > > on
> > > > > > >> the
> > > > > > >> > fly
> > > > > > >> > > > if
> > > > > > >> > > > > > not
> > > > > > >> > > > > > > encountered for the first time. Again correct me
> if
> > > I'm
> > > > > > wrong.
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > Yes, you are right. However this would probably be
> > > > necessary
> > > > > > >> step
> > > > > > >> > for
> > > > > > >> > > > > > rpm/deb, as they may also get installed during OS
> > > install
> > > > > and
> > > > > > >> > before
> > > > > > >> > > > user
> > > > > > >> > > > > > accounts were created.
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure
> > its
> > > > the
> > > > > > best
> > > > > > >> > > idea.
> > > > > > >> > > > > > We'll
> > > > > > >> > > > > > > anyway put dtcli in location which is by default
> > > present
> > > > > in
> > > > > > >> path
> > > > > > >> > > i.e.
> > > > > > >> > > > > > > /usr/bin.
> > > > > > >> > > > > >
> > > > > > >> > > > > > For other env variables specific to apex, I'll
> prefer
> > to
> > > > use
> > > > > > >> > > dt-env.sh
> > > > > > >> > > > > and
> > > > > > >> > > > > > > source it in dtcli rather than changing bashrc
> > etc...
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall
> be
> > > the
> > > > > > >> cleaner
> > > > > > >> > way
> > > > > > >> > > > to
> > > > > > >> > > > > > achieve the same.
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > Thanks,
> > > > > > >> > > > > > > Chinmay.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
> Thombare
> > <
> > > > > > >> > > > > > > aniruddha@datatorrent.com> wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hi,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > @Chinmay,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > We need to consider following:
> > > > > > >> > > > > > > > System wide default config files can be located
> at
> > > > > > following
> > > > > > >> > > > > locations:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change
> dtcli
> > > > > > behaviour
> > > > > > >> on
> > > > > > >> > > how
> > > > > > >> > > > it
> > > > > > >> > > > > > > finds
> > > > > > >> > > > > > > > those locations)
> > > > > > >> > > > > > > > /etc/apex/dt-site.xml (we may have to change
> dtcli
> > > > > > >> behaviour on
> > > > > > >> > > how
> > > > > > >> > > > > it
> > > > > > >> > > > > > > > finds those locations)
> > > > > > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for
> > new
> > > > > users
> > > > > > >> that
> > > > > > >> > > will
> > > > > > >> > > > > be
> > > > > > >> > > > > > > > created in system in future.)
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > We may also have to modify bashrc / bashprofile
> > for
> > > > > > >> population
> > > > > > >> > > any
> > > > > > >> > > > > > > > variables that are required.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > @dev, please put in your comments / suggestion.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Thanks,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Aniruddha
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > > > > >> > > > > thomas@datatorrent.com>
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > -->
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> > Kolhatkar
> > > <
> > > > > > >> > > > > > > > chinmay@datatorrent.com
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > Hello Everyone!!
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Continuing with packaging effort (rpm+deb)
> of
> > > > apex,
> > > > > > here
> > > > > > >> > are
> > > > > > >> > > > some
> > > > > > >> > > > > > > > > proposals
> > > > > > >> > > > > > > > > > about package structure etc..
> > > > > > >> > > > > > > > > > Before posting it on bbigtop mailing list, I
> > > have
> > > > > some
> > > > > > >> > > question
> > > > > > >> > > > > for
> > > > > > >> > > > > > > > apex
> > > > > > >> > > > > > > > > > community.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Proposed Directory structure of apex package
> > for
> > > > > both
> > > > > > >> deb &
> > > > > > >> > > > rpm:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > >> > > > > > > > > >
> > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > >> > > > > > > > > >
> > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > <version> = 3.3.0-incubating.
> > > > > > >> > > > > > > > > > <other dependent jars> = All the 3rd party
> > jars
> > > > > which
> > > > > > >> are
> > > > > > >> > > > > required
> > > > > > >> > > > > > > for
> > > > > > >> > > > > > > > > apex
> > > > > > >> > > > > > > > > > to run. Usually the dependencies are
> packaged
> > as
> > > > > part
> > > > > > of
> > > > > > >> > > > rpm/deb
> > > > > > >> > > > > by
> > > > > > >> > > > > > > any
> > > > > > >> > > > > > > > > > software in bigtop.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Can you specify what those jars are in Bigtop
> > > > context?
> > > > > > >> Same
> > > > > > >> > as
> > > > > > >> > > > > > shipped
> > > > > > >> > > > > > > > with
> > > > > > >> > > > > > > > > DT community addition under lib/ ?
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > <package LICENSE> = Licenses of
> corresponding
> > of
> > > > 3rd
> > > > > > >> party
> > > > > > >> > > jars
> > > > > > >> > > > > > which
> > > > > > >> > > > > > > > > needs
> > > > > > >> > > > > > > > > > to included while packaging.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Questions related to this:
> > > > > > >> > > > > > > > > > 1. Should we call the cli of apex as "apex"
> > > > instead
> > > > > of
> > > > > > >> > > "dtcli"
> > > > > > >> > > > in
> > > > > > >> > > > > > > > bigtop
> > > > > > >> > > > > > > > > > package?
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I think we should keep the name until we are
> > able
> > > to
> > > > > > >> change
> > > > > > >> > it
> > > > > > >> > > in
> > > > > > >> > > > > > Apex.
> > > > > > >> > > > > > > > > Otherwise this may get confusing.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > 2. I see that all softwares in bigtop have
> man
> > > > page
> > > > > > for
> > > > > > >> > their
> > > > > > >> > > > > > > > > executables.
> > > > > > >> > > > > > > > > > I think we should have it too for dtcli. Is
> > > there
> > > > > any
> > > > > > >> > > > > documentation
> > > > > > >> > > > > > > > > which I
> > > > > > >> > > > > > > > > > can convert to man page? or can I use output
> > of
> > > > > "dtcli
> > > > > > >> > > --help"?
> > > > > > >> > > > > > > > > > 3. Do we want to call version of apex in
> > Bigtop
> > > as
> > > > > > >> 3.3.0 OR
> > > > > > >> > > > > > > > > > 3.3.0-incubating?
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Has to be -incubating
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > 4. Is it ok for apex package of bigtop to
> > depend
> > > > on
> > > > > > >> 2.7.1
> > > > > > >> > > > > version
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > > > bigtop hadoop? Any problems that we see with
> > > this
> > > > > > >> > dependency?
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Should work without changed. I thought we
> > > certified
> > > > > > >> against
> > > > > > >> > > > 2.7.0?
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > 5. Following is the apache mirror from which
> > > > bigtop
> > > > > > will
> > > > > > >> > pick
> > > > > > >> > > > the
> > > > > > >> > > > > > > apex
> > > > > > >> > > > > > > > > > source for compilation and packaging. Please
> > > > correct
> > > > > > if
> > > > > > >> > > > > incorrect:
> > > > > > >> > > > > > > > > >
> > > > > > >> > >
> http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror
> > used
> > > by
> > > > > all
> > > > > > >> the
> > > > > > >> > > > > > softwares
> > > > > > >> > > > > > > in
> > > > > > >> > > > > > > > > > bigtop.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Please share your opinion.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > > Chinmay.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> > > > Kolhatkar <
> > > > > > >> > > > > > > > > > chinmay@datatorrent.com>
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > I've already started discussion on bigtop
> > dev
> > > > > > mailing
> > > > > > >> > list
> > > > > > >> > > > for
> > > > > > >> > > > > > > > > > > integration. Also created a Jira for the
> > same.
> > > > For
> > > > > > >> this
> > > > > > >> > > > > > communities
> > > > > > >> > > > > > > > > > > reference, here is the bigtop Jira:
> > > BIGTOP-2313.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas
> > Weise <
> > > > > > >> > > > > > > thomas@datatorrent.com
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >> Chinmay,
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >> I don't see anything under prerequisites
> > that
> > > > > would
> > > > > > >> be a
> > > > > > >> > > > > > problem.
> > > > > > >> > > > > > > We
> > > > > > >> > > > > > > > > > >> looked
> > > > > > >> > > > > > > > > > >> at the ASF licencing compatibility as
> part
> > of
> > > > > > >> becoming
> > > > > > >> > an
> > > > > > >> > > > > > > incubator
> > > > > > >> > > > > > > > > > >> project.
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >> Please focus on the packaging during the
> > next
> > > > > > weeks.
> > > > > > >> > Since
> > > > > > >> > > > the
> > > > > > >> > > > > > > work
> > > > > > >> > > > > > > > > will
> > > > > > >> > > > > > > > > > >> be
> > > > > > >> > > > > > > > > > >> part of Bigtop, related discussions and
> > JIRAs
> > > > > > should
> > > > > > >> > also
> > > > > > >> > > be
> > > > > > >> > > > > > > there.
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >> Would be good to have the packaging in
> > place
> > > by
> > > > > end
> > > > > > >> Feb.
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >> Thanks,
> > > > > > >> > > > > > > > > > >> Thomas
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> > > > > Kolhatkar
> > > > > > <
> > > > > > >> > > > > > > > > > >> chinmay@datatorrent.com>
> > > > > > >> > > > > > > > > > >> wrote:
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >> > Hi All,
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > We're planning a work on adding Apache
> > Apex
> > > > as
> > > > > a
> > > > > > >> > > component
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > Apache
> > > > > > >> > > > > > > > > > >> > Bigtop.
> > > > > > >> > > > > > > > > > >> > Bigtop is the packaging system for the
> > > Apache
> > > > > big
> > > > > > >> data
> > > > > > >> > > > > > > ecosystem.
> > > > > > >> > > > > > > > > > >> Several
> > > > > > >> > > > > > > > > > >> > Hadoop distros use it, most recently
> EMR.
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > Here is the tracking Jira task in
> > APEXCORE
> > > > for
> > > > > > the
> > > > > > >> > same:
> > > > > > >> > > > > > > > > > >> >
> > > > > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > Proposed plan of execution is as
> follows:
> > > > > > >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> > > > > expectation
> > > > > > >> for
> > > > > > >> > new
> > > > > > >> > > > > > > > components
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > >> get
> > > > > > >> > > > > > > > > > >> > integrated into Bigtop.
> > > > > > >> > > > > > > > > > >> > Here is the list of it:
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > Most of them seems to be standard ASF
> > based
> > > > > > >> > > requirements,
> > > > > > >> > > > > but
> > > > > > >> > > > > > > few
> > > > > > >> > > > > > > > > need
> > > > > > >> > > > > > > > > > >> to
> > > > > > >> > > > > > > > > > >> > be checked for:
> > > > > > >> > > > > > > > > > >> > 1. Software projects are expected to be
> > > > > Licensed
> > > > > > >> under
> > > > > > >> > > > > Apache
> > > > > > >> > > > > > > > > License,
> > > > > > >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
> > > > > expected
> > > > > > >> to be
> > > > > > >> > > > > > > compatible
> > > > > > >> > > > > > > > > with
> > > > > > >> > > > > > > > > > >> this
> > > > > > >> > > > > > > > > > >> > license)
> > > > > > >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to
> > > check
> > > > > if
> > > > > > >> > > > > dependencies
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > > Apex
> > > > > > >> > > > > > > > > > >> are
> > > > > > >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess
> > would
> > > > be
> > > > > a
> > > > > > >> > > > > verification
> > > > > > >> > > > > > > > check.
> > > > > > >> > > > > > > > > > >> > 2. Software projects are expected to be
> > > > > > compatible
> > > > > > >> > with
> > > > > > >> > > > all
> > > > > > >> > > > > of
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > >> > supported platforms that Bigtop
> > > distribution
> > > > is
> > > > > > >> > > targeting
> > > > > > >> > > > > > > > > > >> >     - This needs verifying whether our
> > > > software
> > > > > > >> runs
> > > > > > >> > > fine
> > > > > > >> > > > in
> > > > > > >> > > > > > > > > centos-6
> > > > > > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
> debian-8
> > > > > > >> > opensuse-13.2.
> > > > > > >> > > > > > > > > > >> > 3. What smoke tests that should be
> added
> > > for
> > > > > > >> > deployment.
> > > > > > >> > > > > > > > > > >> > 4. Identifying the test artifacts which
> > > goes
> > > > > > beyond
> > > > > > >> > > smoke
> > > > > > >> > > > > test
> > > > > > >> > > > > > > > > > >> >     - These are basically the
> integration
> > > > tests
> > > > > > for
> > > > > > >> > > > > > verification
> > > > > > >> > > > > > > > > after
> > > > > > >> > > > > > > > > > >> the
> > > > > > >> > > > > > > > > > >> > deployment. This is a soft requirement,
> > but
> > > > aim
> > > > > > is
> > > > > > >> to
> > > > > > >> > > > > achieve
> > > > > > >> > > > > > > this
> > > > > > >> > > > > > > > > as
> > > > > > >> > > > > > > > > > >> well
> > > > > > >> > > > > > > > > > >> > or at least have explanation why not to
> > > > > include.
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > If there are any from the link which
> > > > explicitly
> > > > > > >> needs
> > > > > > >> > to
> > > > > > >> > > > be
> > > > > > >> > > > > > > > checked
> > > > > > >> > > > > > > > > > >> other
> > > > > > >> > > > > > > > > > >> > than above 4, please let us know.
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > *Step 2) Adding Apex as component to
> > > Bigtop*
> > > > > > >> > > > > > > > > > >> > From one of the mail archive of Bigtop,
> > it
> > > > was
> > > > > > >> learnt
> > > > > > >> > > that
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > > bigtop
> > > > > > >> > > > > > > > > > >> > community want to see the addition of
> new
> > > > > > >> components
> > > > > > >> > in
> > > > > > >> > > > > > phases.
> > > > > > >> > > > > > > > Here
> > > > > > >> > > > > > > > > > are
> > > > > > >> > > > > > > > > > >> > the phases:
> > > > > > >> > > > > > > > > > >> > 1. Packaging
> > > > > > >> > > > > > > > > > >> >     - This needs creating of package
> i.e.
> > > > rpm &
> > > > > > deb
> > > > > > >> > > files.
> > > > > > >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> > > > > DISCLAMER,
> > > > > > >> > NOTES
> > > > > > >> > > > etc
> > > > > > >> > > > > if
> > > > > > >> > > > > > > any
> > > > > > >> > > > > > > > > > >> needed.
> > > > > > >> > > > > > > > > > >> >     - Any documentation that need to be
> > > added
> > > > > to
> > > > > > >> > > > > distribution
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > our
> > > > > > >> > > > > > > > > > >> > software.
> > > > > > >> > > > > > > > > > >> >     - Any license information of
> > > dependencies
> > > > > > >> required
> > > > > > >> > > to
> > > > > > >> > > > be
> > > > > > >> > > > > > > added
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > >> > package
> > > > > > >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > >> > > > > > > > > > >> > 3. Puppet recipes for automatic
> > deployment
> > > > and
> > > > > > >> > > > configuration
> > > > > > >> > > > > > > > > > >> >     - Add puppet recipes for automatic
> > > > > deployment
> > > > > > >> > > > > > > > > > >> > 4. Integration tests
> > > > > > >> > > > > > > > > > >> >     - For verification of deployments.
> > > > > > >> > > > > > > > > > >> > 5. license clearance:
> > > > > > >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all
> new
> > > > stuff
> > > > > > is
> > > > > > >> > > > compliant
> > > > > > >> > > > > > > with
> > > > > > >> > > > > > > > > ASF
> > > > > > >> > > > > > > > > > >> > license requirements. If you add code
> > > > licenses
> > > > > > >> under
> > > > > > >> > > > > different
> > > > > > >> > > > > > > > > > licenses,
> > > > > > >> > > > > > > > > > >> > those would need to be listed in the
> > > NOTICE.
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > Please share your thoughts on the
> > approach.
> > > > > > >> > > > > > > > > > >> > We'll start corresponding communication
> > on
> > > > > bigtop
> > > > > > >> > > mailing
> > > > > > >> > > > > list
> > > > > > >> > > > > > > as
> > > > > > >> > > > > > > > > > well.
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > We have some specific
> > questions/suggestions
> > > > > > >> related to
> > > > > > >> > > > what
> > > > > > >> > > > > > > should
> > > > > > >> > > > > > > > > be
> > > > > > >> > > > > > > > > > >> the
> > > > > > >> > > > > > > > > > >> > content of the package and what should
> be
> > > the
> > > > > > smoke
> > > > > > >> > > tests,
> > > > > > >> > > > > but
> > > > > > >> > > > > > > in
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > >> > interest of not having too much content
> > > here,
> > > > > > we'll
> > > > > > >> > put
> > > > > > >> > > > the
> > > > > > >> > > > > > > > > questions
> > > > > > >> > > > > > > > > > >> as a
> > > > > > >> > > > > > > > > > >> > separate mail in this mailthread.
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > Thanks,
> > > > > > >> > > > > > > > > > >> > Chinmay.
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

Why the dtcli link under lib?


On Sun, Feb 14, 2016 at 9:09 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Hmmm... I get your point. In development mode, it tries to find the hadoop
> executable like following:
>
>
>
> HADOOP_SEARCH_PATH="${HADOOP_PREFIX}/bin:${HADOOP_HOME}/bin:${PATH}:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:."
>   export DT_HADOOP=`PATH=${HADOOP_SEARCH_PATH} && command -v hadoop
> 2>/dev/null`
>
> I will check with bigtop community that what is the best way to ensure
> during installation that hadoop binary will be present in above search path
> search.
>
> Also, I checked as you suggested, i.e. removed dt-env.sh from package and
> tried to run the dtcli... It works!!!
>
> Thank you for pointing that out Thomas.
>
> Now I agree with you that, we can get rid of conf/dt-env.sh.. As this was
> the only file in conf/, unless someone has anything to include in conf/
> folder, I'll remove that too then.
>
> Also, this would mean, only dt* that remains in package are dtcli links to
> apex script added for backward compatibility.
> i.e following:
>
> /
> `-- usr
>     |-- bin
>     |   |-- apex -> ../lib/apex/bin/apex
>     |   `-- dtcli -> ../lib/apex/bin/apex
>     |-- lib
>     |   `-- apex
>     |       |-- bin
>     |       |   |-- apex   // This is the real script
>     |       |   `-- dtcli -> apex
>
>
>
> Thanks,
> Chinmay.
>
>
>
> On Mon, Feb 15, 2016 at 10:22 AM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
> > I'm not convinced dt-env.sh is even needed. In development mode, the file
> > doesn't exist but it will still find Hadoop.
> >
> > The absolute path to hadoop script was introduced to deal with the layout
> > of specific distros. You will know whether that applies to Bigtop or not.
> >
> > Thomas
> >
> > On Sun, Feb 14, 2016 at 8:16 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > There are only 2 things that are required from dt-env.sh:
> > > 1. DT_HADOOP variable which is exported by dt-env.sh This variable has
> > > absolute path for hadoop binary
> > > 2. A find_hadoop() method which find the path of hadoop binary.
> > >
> > > JAVA_HOME is not required for this.
> > >
> > > The idea is not to change the dtcli. Current dtcli (in apex
> > > 3.3.0-incubating src code), sources dt-env.sh for above 2 information.
> > > Hence included dt-env.sh
> > >
> > > If the community agrees, we can make dtcli more generic so that in
> future
> > > packing of bigtop, dt-env.sh can be completely excluded OR at least
> > change
> > > the name to apex-env.sh.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > >
> > > On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise <th...@datatorrent.com>
> > > wrote:
> > >
> > > > Chinmay,
> > > >
> > > > Can you please specify what settings in dt-env.sh are required in a
> > > bigtop
> > > > environment?
> > > >
> > > > Location of hadoop and JAVA_HOME should be provided by the stack
> > already?
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > >
> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> chinmay@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Here is a suggestion I have related to name change:
> > > > >
> > > > > 1. While packaging we change the name of "dtcli" script to "apex".
> > This
> > > > is
> > > > > done so that in future when we change the name to "apex", users of
> > > bigtop
> > > > > apex don't have to transition much.
> > > > > 2. We also keep a symlink named dtcli which points to apex script.
> > This
> > > > is
> > > > > for backward compatibility.
> > > > > 3. We still keep conf/dt-env.sh for first integration in bigtop.
> This
> > > is
> > > > in
> > > > > the interest of not changing the content of dtcli or apex script.
> > > > > 4. In apex 3.4.0, we take care of all these naming related changes
> > and
> > > > > update bigtop repository later to remove references to dtcli all
> > > > together.
> > > > >
> > > > > Please share your thoughts on above approach.
> > > > >
> > > > > Also, please share what could be the man page content for "apex".
> > > > >
> > > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> > chinmay@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Yes.. It'll be build from source tar downloaded from one of the
> > > apache
> > > > > > mirror: apache.osuosl.org
> > > > > >
> > > > > > The reason for dt-env.sh is it's sourced from dtcli. This leaves
> > us 3
> > > > > > options:
> > > > > > If we should not have any dt-env.sh, dtcli would need a change
> > while
> > > > > > packaging.
> > > > > >
> > > > > > But if we want to use dtcli as it is, then we would need
> dt-env.sh
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> > > thomas@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Looks good overall, though there shouldn't be any dt-env.sh
> > > > > >>
> > > > > >> What will the bigtop package be built from, the source tar?
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > > > chinmay@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Thomas,
> > > > > >> >
> > > > > >> > Thanks for the feedback.
> > > > > >> >
> > > > > >> > First, we're not changing the name anywhere. We'll follow what
> > > > > >> currently is
> > > > > >> > in source tarball for 3.3.0-incubating version.
> > > > > >> >
> > > > > >> > Secondly, I've mentioned a directory structure below which is
> > > inline
> > > > > >> with
> > > > > >> > other existing integrations with bigtop.
> > > > > >> >
> > > > > >> > The need for each file is as follows:
> > > > > >> > 1. bin/dtcli -> CLI for apex picked from
> > engine/src/main/scripts/
> > > of
> > > > > >> source
> > > > > >> > code apex 3.3.0-incubating.
> > > > > >> >
> > > > > >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches
> > for
> > > > > hadoop
> > > > > >> > binary path and exports an env variable for dtcli to use it.
> > > > > >> >
> > > > > >> > 3. lib/ -> Libraries for CLI. This list is extracted in a
> > similar
> > > > way
> > > > > to
> > > > > >> > how DT Community Edition finds dependency jars.
> > > > > >> > Please note that this is the first iteration list of jars. I'm
> > > > trying
> > > > > to
> > > > > >> > narrow this down to only those which are really required.
> > > > > >> > The test that I'm running to check if dtcli runs fine with
> given
> > > > > >> dependency
> > > > > >> > is to launch+shutdown+kill+status for pi demo.
> > > > > >> >
> > > > > >> > Above is the directory structure which is required for CLI to
> > > work.
> > > > > >> > What aniruddha and pradeep has mentioned are some additional
> > files
> > > > > which
> > > > > >> > makes the like of administrator easier. For eg. /etc/skel
> > > > > >> >
> > > > > >> > Please let me know if this is inline with what you're
> thinking.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Chinmay.
> > > > > >> >
> > > > > >> >
> > > > > >> > .
> > > > > >> > |-- bin
> > > > > >> > |   `-- dtcli
> > > > > >> > |-- conf
> > > > > >> > |   `-- dt-env.sh
> > > > > >> > `-- lib
> > > > > >> >     |-- ant-1.9.2.jar
> > > > > >> >     |-- ant-launcher-1.9.2.jar
> > > > > >> >     |-- apex-api-3.3.0-incubating.jar
> > > > > >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > > > > >> >     |-- apex-common-3.3.0-incubating.jar
> > > > > >> >     |-- apex-engine.jar
> > > > > >> >     |-- apex-engine-tests.jar
> > > > > >> >     |-- async-http-client-1.7.20.jar
> > > > > >> >     |-- bval-core-0.5.jar
> > > > > >> >     |-- bval-jsr303-0.5.jar
> > > > > >> >     |-- commons-beanutils-1.8.3.jar
> > > > > >> >     |-- commons-codec-1.10.jar
> > > > > >> >     |-- commons-lang3-3.1.jar
> > > > > >> >     |-- commons-logging-1.1.3.jar
> > > > > >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > > > >> >     |-- hadoop-common-2.2.0-tests.jar
> > > > > >> >     |-- httpclient-4.3.5.jar
> > > > > >> >     |-- httpcore-4.3.2.jar
> > > > > >> >     |-- jackson-core-asl-1.9.2.jar
> > > > > >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > > > >> >     |-- javax.servlet-3.1.jar
> > > > > >> >     |-- javax.servlet-api-3.0.1.jar
> > > > > >> >     |-- jersey-apache-client4-1.9.jar
> > > > > >> >     |-- jersey-client-1.9.jar
> > > > > >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > > > >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > > > >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > > > >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > > > > >> >     |-- jline-2.11.jar
> > > > > >> >     |-- kryo-2.24.0.jar
> > > > > >> >     |-- mbassador-1.1.9.jar
> > > > > >> >     |-- minlog-1.2.jar
> > > > > >> >     |-- netlet-1.2.0.jar
> > > > > >> >     |-- netty-3.6.6.Final.jar
> > > > > >> >     |-- objenesis-2.1.jar
> > > > > >> >     |-- validation-api-1.1.0.Final.jar
> > > > > >> >     |-- xbean-asm5-shaded-4.3.jar
> > > > > >> >     `-- zip4j-1.3.2.jar
> > > > > >> >
> > > > > >> > 3 directories, 40 files
> > > > > >> >
> > > > > >> >
> > > > > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > > > thomas@datatorrent.com
> > > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Chinmay,
> > > > > >> > >
> > > > > >> > > Before discussing where to put the files, let's make sure
> they
> > > are
> > > > > >> really
> > > > > >> > > needed for the operation of the CLI. As for names, anything
> > new
> > > > > needs
> > > > > >> to
> > > > > >> > > reflect Apex in the name and should follow common
> conventions,
> > > > > >> especially
> > > > > >> > > in Bigtop where there are many existing integrations to look
> > at.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Thomas
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > > > > >> > > chinmay@datatorrent.com>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Really good points Pradeep and Aniruddha.
> > > > > >> > > >
> > > > > >> > > > 1. I believe we won't need to change the dtcli considering
> > it
> > > > > works
> > > > > >> > with
> > > > > >> > > DT
> > > > > >> > > > Community edition. We can keep the directory structure
> > similar
> > > > to
> > > > > >> that.
> > > > > >> > > > dt-env.sh has variables which contains information
> required
> > > for
> > > > > >> dtcli
> > > > > >> > to
> > > > > >> > > > launch.
> > > > > >> > > >
> > > > > >> > > > 2. Let me check with bigtop community that whether they
> > > > facilitate
> > > > > >> the
> > > > > >> > > > installation of rpms/debs before user is created. In
> either
> > > > case,
> > > > > >> > current
> > > > > >> > > > dtcli creates a .dt folder in home directory. Also before
> > > > putting
> > > > > >> > > anything
> > > > > >> > > > in /etc/skel we need to define what are the default
> contents
> > > > that
> > > > > >> > should
> > > > > >> > > go
> > > > > >> > > > to ~/.dt/ folder. If there is no defaults, probably we
> > should
> > > > not
> > > > > >> > > > explicitly add it in /etc/skel.
> > > > > >> > > >
> > > > > >> > > > 3.  /etc/profile.d approach looks nice. This way contents
> of
> > > > > >> dt-env.sh
> > > > > >> > > are
> > > > > >> > > > present as env variables. But I see a catch there. Adding
> > > > > dt-env.sh
> > > > > >> to
> > > > > >> > > > /etc/profile.d would make all the variables available at
> > > runtime
> > > > > all
> > > > > >> > the
> > > > > >> > > > time. I feel a little skeptical about that. Maybe a
> possible
> > > > > >> collision
> > > > > >> > > can
> > > > > >> > > > occur with other application vars.
> > > > > >> > > > Moreover, current dtcli does source "../conf/dt-env.sh"
> and
> > > > > >> > > > "~/.dt/dt-env.sh". So those variables are anyway
> available.
> > > > > >> > > >
> > > > > >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > > > > >> > > Jira/repository.
> > > > > >> > > > Packaging related code usually goes there. (That's what
> all
> > > the
> > > > > >> > > components
> > > > > >> > > > in bigtop does).
> > > > > >> > > > Having said that, once a PR is created, I'll be sharing
> the
> > > link
> > > > > of
> > > > > >> PR
> > > > > >> > > > here, so that apex community as well can review it.
> > > > > >> > > >
> > > > > >> > > > Thanks,
> > > > > >> > > > Chinmay.
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > > > >> > > > aniruddha@datatorrent.com> wrote:
> > > > > >> > > >
> > > > > >> > > > > +1 on suggestions and approach.
> > > > > >> > > > > We may need to iron out details about exact paths etc.
> > > > > >> > > > > Which can be done on jira / PR comments.
> > > > > >> > > > > Is that right @dev?
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > > > >> > apache@pradeepdalvi.com>
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Inline comments...
> > > > > >> > > > > >
> > > > > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > > >> > > > > > chinmay@datatorrent.com>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > @Thomas, Not all the jar present in DT community
> > edition
> > > > > will
> > > > > >> be
> > > > > >> > > > > included
> > > > > >> > > > > > > there.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Here is the list of jars I found from common
> > > dependencies
> > > > > >> between
> > > > > >> > > of
> > > > > >> > > > > apex
> > > > > >> > > > > > > and DT Community edition:
> > > > > >> > > > > > > netlet-1.2.0.jar
> > > > > >> > > > > > > kryo-2.24.0.jar
> > > > > >> > > > > > > jackson-core-asl-1.9.2.jar
> > > > > >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > >> > > > > > > async-http-client-1.7.20.jar
> > > > > >> > > > > > > netty-3.6.6.Final.jar
> > > > > >> > > > > > > validation-api-1.1.0.Final.jar
> > > > > >> > > > > > > bval-jsr303-0.5.jar
> > > > > >> > > > > > > bval-core-0.5.jar
> > > > > >> > > > > > > commons-lang3-3.1.jar
> > > > > >> > > > > > > commons-beanutils-1.8.3.jar
> > > > > >> > > > > > > httpclient-4.3.5.jar
> > > > > >> > > > > > > commons-codec-1.10.jar
> > > > > >> > > > > > > zip4j-1.3.2.jar
> > > > > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > >> > > > > > > jersey-apache-client4-1.9.jar
> > > > > >> > > > > > > jline-2.11.jar
> > > > > >> > > > > > > ant-1.9.2.jar
> > > > > >> > > > > > > ant-launcher-1.9.2.jar
> > > > > >> > > > > > > mbassador-1.1.9.jar
> > > > > >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > >> > > > > > > jackson-xc-1.9.2.jar
> > > > > >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > > > >> > > > > > >
> > > > > >> > > > > > > Ofcourse, I'll be running some tests do check that
> > dtcli
> > > > > works
> > > > > >> > > > properly
> > > > > >> > > > > > for
> > > > > >> > > > > > > launch+shutdown+kill of apps with only these
> libraries
> > > > > >> present in
> > > > > >> > > > > > isolation
> > > > > >> > > > > > > without dependency on local m2.
> > > > > >> > > > > > > I'm believe that there are unwanted jars which are
> > used
> > > > for
> > > > > >> > compile
> > > > > >> > > > > time
> > > > > >> > > > > > > dependency and not runtime in above list which I can
> > > drop
> > > > to
> > > > > >> keep
> > > > > >> > > > > package
> > > > > >> > > > > > > size minimal.
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > @Anniruddha,
> > > > > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh
> > is
> > > > > >> required.
> > > > > >> > > > Only
> > > > > >> > > > > > > required values I see in it are classpath. and dtcli
> > is
> > > > > >> capable
> > > > > >> > of
> > > > > >> > > > > > building
> > > > > >> > > > > > > it on the fly.
> > > > > >> > > > > > >
> > > > > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli
> can
> > > > find
> > > > > >> the
> > > > > >> > jar
> > > > > >> > > > > path,
> > > > > >> > > > > > > dt-env.sh and all such conf files. If we need a
> change
> > > in
> > > > > >> dtcli,
> > > > > >> > > then
> > > > > >> > > > > > > community's opinion is required for whether dtcli
> > should
> > > > > >> change
> > > > > >> > in
> > > > > >> > > > our
> > > > > >> > > > > > repo
> > > > > >> > > > > > > or a copy of that with required changes exist in
> > bigtop
> > > > repo
> > > > > >> > until
> > > > > >> > > we
> > > > > >> > > > > > make
> > > > > >> > > > > > > the dtcli generic enough.
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Can we have these file paths set using environment
> > > variables
> > > > > >> with
> > > > > >> > > some
> > > > > >> > > > > > default values in dtcli?
> > > > > >> > > > > > Then we can set such params in dt-env.sh.
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > > 3. dt-sited.xml -> How is this file different from
> > > > > >> dt-site.xml?
> > > > > >> > Not
> > > > > >> > > > > sure
> > > > > >> > > > > > if
> > > > > >> > > > > > > adding in /etc/skel would help. Files & Dirs from
> > > > /etc/skel
> > > > > >> are
> > > > > >> > > > copied
> > > > > >> > > > > to
> > > > > >> > > > > > > home of new user when useradd program is called. But
> > for
> > > > > >> existing
> > > > > >> > > > users
> > > > > >> > > > > > > this won't be of any use. Moreover, dtcli creates
> > ~/.dt/
> > > > on
> > > > > >> the
> > > > > >> > fly
> > > > > >> > > > if
> > > > > >> > > > > > not
> > > > > >> > > > > > > encountered for the first time. Again correct me if
> > I'm
> > > > > wrong.
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Yes, you are right. However this would probably be
> > > necessary
> > > > > >> step
> > > > > >> > for
> > > > > >> > > > > > rpm/deb, as they may also get installed during OS
> > install
> > > > and
> > > > > >> > before
> > > > > >> > > > user
> > > > > >> > > > > > accounts were created.
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure
> its
> > > the
> > > > > best
> > > > > >> > > idea.
> > > > > >> > > > > > We'll
> > > > > >> > > > > > > anyway put dtcli in location which is by default
> > present
> > > > in
> > > > > >> path
> > > > > >> > > i.e.
> > > > > >> > > > > > > /usr/bin.
> > > > > >> > > > > >
> > > > > >> > > > > > For other env variables specific to apex, I'll prefer
> to
> > > use
> > > > > >> > > dt-env.sh
> > > > > >> > > > > and
> > > > > >> > > > > > > source it in dtcli rather than changing bashrc
> etc...
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be
> > the
> > > > > >> cleaner
> > > > > >> > way
> > > > > >> > > > to
> > > > > >> > > > > > achieve the same.
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > Thanks,
> > > > > >> > > > > > > Chinmay.
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare
> <
> > > > > >> > > > > > > aniruddha@datatorrent.com> wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > @Chinmay,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > We need to consider following:
> > > > > >> > > > > > > > System wide default config files can be located at
> > > > > following
> > > > > >> > > > > locations:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> > > > > behaviour
> > > > > >> on
> > > > > >> > > how
> > > > > >> > > > it
> > > > > >> > > > > > > finds
> > > > > >> > > > > > > > those locations)
> > > > > >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> > > > > >> behaviour on
> > > > > >> > > how
> > > > > >> > > > > it
> > > > > >> > > > > > > > finds those locations)
> > > > > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for
> new
> > > > users
> > > > > >> that
> > > > > >> > > will
> > > > > >> > > > > be
> > > > > >> > > > > > > > created in system in future.)
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > We may also have to modify bashrc / bashprofile
> for
> > > > > >> population
> > > > > >> > > any
> > > > > >> > > > > > > > variables that are required.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > @dev, please put in your comments / suggestion.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Thanks,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Aniruddha
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > > > >> > > > > thomas@datatorrent.com>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > -->
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> Kolhatkar
> > <
> > > > > >> > > > > > > > chinmay@datatorrent.com
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Hello Everyone!!
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of
> > > apex,
> > > > > here
> > > > > >> > are
> > > > > >> > > > some
> > > > > >> > > > > > > > > proposals
> > > > > >> > > > > > > > > > about package structure etc..
> > > > > >> > > > > > > > > > Before posting it on bbigtop mailing list, I
> > have
> > > > some
> > > > > >> > > question
> > > > > >> > > > > for
> > > > > >> > > > > > > > apex
> > > > > >> > > > > > > > > > community.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Proposed Directory structure of apex package
> for
> > > > both
> > > > > >> deb &
> > > > > >> > > > rpm:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > >> > > > > > > > > >
> > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > >> > > > > > > > > >
> > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > <version> = 3.3.0-incubating.
> > > > > >> > > > > > > > > > <other dependent jars> = All the 3rd party
> jars
> > > > which
> > > > > >> are
> > > > > >> > > > > required
> > > > > >> > > > > > > for
> > > > > >> > > > > > > > > apex
> > > > > >> > > > > > > > > > to run. Usually the dependencies are packaged
> as
> > > > part
> > > > > of
> > > > > >> > > > rpm/deb
> > > > > >> > > > > by
> > > > > >> > > > > > > any
> > > > > >> > > > > > > > > > software in bigtop.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Can you specify what those jars are in Bigtop
> > > context?
> > > > > >> Same
> > > > > >> > as
> > > > > >> > > > > > shipped
> > > > > >> > > > > > > > with
> > > > > >> > > > > > > > > DT community addition under lib/ ?
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > <package LICENSE> = Licenses of corresponding
> of
> > > 3rd
> > > > > >> party
> > > > > >> > > jars
> > > > > >> > > > > > which
> > > > > >> > > > > > > > > needs
> > > > > >> > > > > > > > > > to included while packaging.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Questions related to this:
> > > > > >> > > > > > > > > > 1. Should we call the cli of apex as "apex"
> > > instead
> > > > of
> > > > > >> > > "dtcli"
> > > > > >> > > > in
> > > > > >> > > > > > > > bigtop
> > > > > >> > > > > > > > > > package?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I think we should keep the name until we are
> able
> > to
> > > > > >> change
> > > > > >> > it
> > > > > >> > > in
> > > > > >> > > > > > Apex.
> > > > > >> > > > > > > > > Otherwise this may get confusing.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > 2. I see that all softwares in bigtop have man
> > > page
> > > > > for
> > > > > >> > their
> > > > > >> > > > > > > > > executables.
> > > > > >> > > > > > > > > > I think we should have it too for dtcli. Is
> > there
> > > > any
> > > > > >> > > > > documentation
> > > > > >> > > > > > > > > which I
> > > > > >> > > > > > > > > > can convert to man page? or can I use output
> of
> > > > "dtcli
> > > > > >> > > --help"?
> > > > > >> > > > > > > > > > 3. Do we want to call version of apex in
> Bigtop
> > as
> > > > > >> 3.3.0 OR
> > > > > >> > > > > > > > > > 3.3.0-incubating?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Has to be -incubating
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > 4. Is it ok for apex package of bigtop to
> depend
> > > on
> > > > > >> 2.7.1
> > > > > >> > > > > version
> > > > > >> > > > > > of
> > > > > >> > > > > > > > > > bigtop hadoop? Any problems that we see with
> > this
> > > > > >> > dependency?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Should work without changed. I thought we
> > certified
> > > > > >> against
> > > > > >> > > > 2.7.0?
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > 5. Following is the apache mirror from which
> > > bigtop
> > > > > will
> > > > > >> > pick
> > > > > >> > > > the
> > > > > >> > > > > > > apex
> > > > > >> > > > > > > > > > source for compilation and packaging. Please
> > > correct
> > > > > if
> > > > > >> > > > > incorrect:
> > > > > >> > > > > > > > > >
> > > > > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror
> used
> > by
> > > > all
> > > > > >> the
> > > > > >> > > > > > softwares
> > > > > >> > > > > > > in
> > > > > >> > > > > > > > > > bigtop.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Please share your opinion.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > Chinmay.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> > > Kolhatkar <
> > > > > >> > > > > > > > > > chinmay@datatorrent.com>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > I've already started discussion on bigtop
> dev
> > > > > mailing
> > > > > >> > list
> > > > > >> > > > for
> > > > > >> > > > > > > > > > > integration. Also created a Jira for the
> same.
> > > For
> > > > > >> this
> > > > > >> > > > > > communities
> > > > > >> > > > > > > > > > > reference, here is the bigtop Jira:
> > BIGTOP-2313.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas
> Weise <
> > > > > >> > > > > > > thomas@datatorrent.com
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >> Chinmay,
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >> I don't see anything under prerequisites
> that
> > > > would
> > > > > >> be a
> > > > > >> > > > > > problem.
> > > > > >> > > > > > > We
> > > > > >> > > > > > > > > > >> looked
> > > > > >> > > > > > > > > > >> at the ASF licencing compatibility as part
> of
> > > > > >> becoming
> > > > > >> > an
> > > > > >> > > > > > > incubator
> > > > > >> > > > > > > > > > >> project.
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >> Please focus on the packaging during the
> next
> > > > > weeks.
> > > > > >> > Since
> > > > > >> > > > the
> > > > > >> > > > > > > work
> > > > > >> > > > > > > > > will
> > > > > >> > > > > > > > > > >> be
> > > > > >> > > > > > > > > > >> part of Bigtop, related discussions and
> JIRAs
> > > > > should
> > > > > >> > also
> > > > > >> > > be
> > > > > >> > > > > > > there.
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >> Would be good to have the packaging in
> place
> > by
> > > > end
> > > > > >> Feb.
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >> Thanks,
> > > > > >> > > > > > > > > > >> Thomas
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> > > > Kolhatkar
> > > > > <
> > > > > >> > > > > > > > > > >> chinmay@datatorrent.com>
> > > > > >> > > > > > > > > > >> wrote:
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >> > Hi All,
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > We're planning a work on adding Apache
> Apex
> > > as
> > > > a
> > > > > >> > > component
> > > > > >> > > > > to
> > > > > >> > > > > > > > Apache
> > > > > >> > > > > > > > > > >> > Bigtop.
> > > > > >> > > > > > > > > > >> > Bigtop is the packaging system for the
> > Apache
> > > > big
> > > > > >> data
> > > > > >> > > > > > > ecosystem.
> > > > > >> > > > > > > > > > >> Several
> > > > > >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > Here is the tracking Jira task in
> APEXCORE
> > > for
> > > > > the
> > > > > >> > same:
> > > > > >> > > > > > > > > > >> >
> > > > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> > > > expectation
> > > > > >> for
> > > > > >> > new
> > > > > >> > > > > > > > components
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > >> get
> > > > > >> > > > > > > > > > >> > integrated into Bigtop.
> > > > > >> > > > > > > > > > >> > Here is the list of it:
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > Most of them seems to be standard ASF
> based
> > > > > >> > > requirements,
> > > > > >> > > > > but
> > > > > >> > > > > > > few
> > > > > >> > > > > > > > > need
> > > > > >> > > > > > > > > > >> to
> > > > > >> > > > > > > > > > >> > be checked for:
> > > > > >> > > > > > > > > > >> > 1. Software projects are expected to be
> > > > Licensed
> > > > > >> under
> > > > > >> > > > > Apache
> > > > > >> > > > > > > > > License,
> > > > > >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
> > > > expected
> > > > > >> to be
> > > > > >> > > > > > > compatible
> > > > > >> > > > > > > > > with
> > > > > >> > > > > > > > > > >> this
> > > > > >> > > > > > > > > > >> > license)
> > > > > >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to
> > check
> > > > if
> > > > > >> > > > > dependencies
> > > > > >> > > > > > of
> > > > > >> > > > > > > > > Apex
> > > > > >> > > > > > > > > > >> are
> > > > > >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess
> would
> > > be
> > > > a
> > > > > >> > > > > verification
> > > > > >> > > > > > > > check.
> > > > > >> > > > > > > > > > >> > 2. Software projects are expected to be
> > > > > compatible
> > > > > >> > with
> > > > > >> > > > all
> > > > > >> > > > > of
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > >> > supported platforms that Bigtop
> > distribution
> > > is
> > > > > >> > > targeting
> > > > > >> > > > > > > > > > >> >     - This needs verifying whether our
> > > software
> > > > > >> runs
> > > > > >> > > fine
> > > > > >> > > > in
> > > > > >> > > > > > > > > centos-6
> > > > > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > > > > >> > opensuse-13.2.
> > > > > >> > > > > > > > > > >> > 3. What smoke tests that should be added
> > for
> > > > > >> > deployment.
> > > > > >> > > > > > > > > > >> > 4. Identifying the test artifacts which
> > goes
> > > > > beyond
> > > > > >> > > smoke
> > > > > >> > > > > test
> > > > > >> > > > > > > > > > >> >     - These are basically the integration
> > > tests
> > > > > for
> > > > > >> > > > > > verification
> > > > > >> > > > > > > > > after
> > > > > >> > > > > > > > > > >> the
> > > > > >> > > > > > > > > > >> > deployment. This is a soft requirement,
> but
> > > aim
> > > > > is
> > > > > >> to
> > > > > >> > > > > achieve
> > > > > >> > > > > > > this
> > > > > >> > > > > > > > > as
> > > > > >> > > > > > > > > > >> well
> > > > > >> > > > > > > > > > >> > or at least have explanation why not to
> > > > include.
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > If there are any from the link which
> > > explicitly
> > > > > >> needs
> > > > > >> > to
> > > > > >> > > > be
> > > > > >> > > > > > > > checked
> > > > > >> > > > > > > > > > >> other
> > > > > >> > > > > > > > > > >> > than above 4, please let us know.
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > *Step 2) Adding Apex as component to
> > Bigtop*
> > > > > >> > > > > > > > > > >> > From one of the mail archive of Bigtop,
> it
> > > was
> > > > > >> learnt
> > > > > >> > > that
> > > > > >> > > > > the
> > > > > >> > > > > > > > > bigtop
> > > > > >> > > > > > > > > > >> > community want to see the addition of new
> > > > > >> components
> > > > > >> > in
> > > > > >> > > > > > phases.
> > > > > >> > > > > > > > Here
> > > > > >> > > > > > > > > > are
> > > > > >> > > > > > > > > > >> > the phases:
> > > > > >> > > > > > > > > > >> > 1. Packaging
> > > > > >> > > > > > > > > > >> >     - This needs creating of package i.e.
> > > rpm &
> > > > > deb
> > > > > >> > > files.
> > > > > >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> > > > DISCLAMER,
> > > > > >> > NOTES
> > > > > >> > > > etc
> > > > > >> > > > > if
> > > > > >> > > > > > > any
> > > > > >> > > > > > > > > > >> needed.
> > > > > >> > > > > > > > > > >> >     - Any documentation that need to be
> > added
> > > > to
> > > > > >> > > > > distribution
> > > > > >> > > > > > of
> > > > > >> > > > > > > > our
> > > > > >> > > > > > > > > > >> > software.
> > > > > >> > > > > > > > > > >> >     - Any license information of
> > dependencies
> > > > > >> required
> > > > > >> > > to
> > > > > >> > > > be
> > > > > >> > > > > > > added
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > >> > package
> > > > > >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > >> > > > > > > > > > >> > 3. Puppet recipes for automatic
> deployment
> > > and
> > > > > >> > > > configuration
> > > > > >> > > > > > > > > > >> >     - Add puppet recipes for automatic
> > > > deployment
> > > > > >> > > > > > > > > > >> > 4. Integration tests
> > > > > >> > > > > > > > > > >> >     - For verification of deployments.
> > > > > >> > > > > > > > > > >> > 5. license clearance:
> > > > > >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new
> > > stuff
> > > > > is
> > > > > >> > > > compliant
> > > > > >> > > > > > > with
> > > > > >> > > > > > > > > ASF
> > > > > >> > > > > > > > > > >> > license requirements. If you add code
> > > licenses
> > > > > >> under
> > > > > >> > > > > different
> > > > > >> > > > > > > > > > licenses,
> > > > > >> > > > > > > > > > >> > those would need to be listed in the
> > NOTICE.
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > Please share your thoughts on the
> approach.
> > > > > >> > > > > > > > > > >> > We'll start corresponding communication
> on
> > > > bigtop
> > > > > >> > > mailing
> > > > > >> > > > > list
> > > > > >> > > > > > > as
> > > > > >> > > > > > > > > > well.
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > We have some specific
> questions/suggestions
> > > > > >> related to
> > > > > >> > > > what
> > > > > >> > > > > > > should
> > > > > >> > > > > > > > > be
> > > > > >> > > > > > > > > > >> the
> > > > > >> > > > > > > > > > >> > content of the package and what should be
> > the
> > > > > smoke
> > > > > >> > > tests,
> > > > > >> > > > > but
> > > > > >> > > > > > > in
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > >> > interest of not having too much content
> > here,
> > > > > we'll
> > > > > >> > put
> > > > > >> > > > the
> > > > > >> > > > > > > > > questions
> > > > > >> > > > > > > > > > >> as a
> > > > > >> > > > > > > > > > >> > separate mail in this mailthread.
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > Thanks,
> > > > > >> > > > > > > > > > >> > Chinmay.
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Hmmm... I get your point. In development mode, it tries to find the hadoop
executable like following:


HADOOP_SEARCH_PATH="${HADOOP_PREFIX}/bin:${HADOOP_HOME}/bin:${PATH}:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:."
  export DT_HADOOP=`PATH=${HADOOP_SEARCH_PATH} && command -v hadoop
2>/dev/null`

I will check with bigtop community that what is the best way to ensure
during installation that hadoop binary will be present in above search path
search.

Also, I checked as you suggested, i.e. removed dt-env.sh from package and
tried to run the dtcli... It works!!!

Thank you for pointing that out Thomas.

Now I agree with you that, we can get rid of conf/dt-env.sh.. As this was
the only file in conf/, unless someone has anything to include in conf/
folder, I'll remove that too then.

Also, this would mean, only dt* that remains in package are dtcli links to
apex script added for backward compatibility.
i.e following:

/
`-- usr
    |-- bin
    |   |-- apex -> ../lib/apex/bin/apex
    |   `-- dtcli -> ../lib/apex/bin/apex
    |-- lib
    |   `-- apex
    |       |-- bin
    |       |   |-- apex   // This is the real script
    |       |   `-- dtcli -> apex



Thanks,
Chinmay.



On Mon, Feb 15, 2016 at 10:22 AM, Thomas Weise <th...@datatorrent.com>
wrote:

> I'm not convinced dt-env.sh is even needed. In development mode, the file
> doesn't exist but it will still find Hadoop.
>
> The absolute path to hadoop script was introduced to deal with the layout
> of specific distros. You will know whether that applies to Bigtop or not.
>
> Thomas
>
> On Sun, Feb 14, 2016 at 8:16 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Hi Thomas,
> >
> > There are only 2 things that are required from dt-env.sh:
> > 1. DT_HADOOP variable which is exported by dt-env.sh This variable has
> > absolute path for hadoop binary
> > 2. A find_hadoop() method which find the path of hadoop binary.
> >
> > JAVA_HOME is not required for this.
> >
> > The idea is not to change the dtcli. Current dtcli (in apex
> > 3.3.0-incubating src code), sources dt-env.sh for above 2 information.
> > Hence included dt-env.sh
> >
> > If the community agrees, we can make dtcli more generic so that in future
> > packing of bigtop, dt-env.sh can be completely excluded OR at least
> change
> > the name to apex-env.sh.
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> > On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> > > Chinmay,
> > >
> > > Can you please specify what settings in dt-env.sh are required in a
> > bigtop
> > > environment?
> > >
> > > Location of hadoop and JAVA_HOME should be provided by the stack
> already?
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <chinmay@apache.org
> >
> > > wrote:
> > >
> > > > Here is a suggestion I have related to name change:
> > > >
> > > > 1. While packaging we change the name of "dtcli" script to "apex".
> This
> > > is
> > > > done so that in future when we change the name to "apex", users of
> > bigtop
> > > > apex don't have to transition much.
> > > > 2. We also keep a symlink named dtcli which points to apex script.
> This
> > > is
> > > > for backward compatibility.
> > > > 3. We still keep conf/dt-env.sh for first integration in bigtop. This
> > is
> > > in
> > > > the interest of not changing the content of dtcli or apex script.
> > > > 4. In apex 3.4.0, we take care of all these naming related changes
> and
> > > > update bigtop repository later to remove references to dtcli all
> > > together.
> > > >
> > > > Please share your thoughts on above approach.
> > > >
> > > > Also, please share what could be the man page content for "apex".
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > >
> > > > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> chinmay@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Yes.. It'll be build from source tar downloaded from one of the
> > apache
> > > > > mirror: apache.osuosl.org
> > > > >
> > > > > The reason for dt-env.sh is it's sourced from dtcli. This leaves
> us 3
> > > > > options:
> > > > > If we should not have any dt-env.sh, dtcli would need a change
> while
> > > > > packaging.
> > > > >
> > > > > But if we want to use dtcli as it is, then we would need dt-env.sh
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> > thomas@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > >> Looks good overall, though there shouldn't be any dt-env.sh
> > > > >>
> > > > >> What will the bigtop package be built from, the source tar?
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > > chinmay@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Thomas,
> > > > >> >
> > > > >> > Thanks for the feedback.
> > > > >> >
> > > > >> > First, we're not changing the name anywhere. We'll follow what
> > > > >> currently is
> > > > >> > in source tarball for 3.3.0-incubating version.
> > > > >> >
> > > > >> > Secondly, I've mentioned a directory structure below which is
> > inline
> > > > >> with
> > > > >> > other existing integrations with bigtop.
> > > > >> >
> > > > >> > The need for each file is as follows:
> > > > >> > 1. bin/dtcli -> CLI for apex picked from
> engine/src/main/scripts/
> > of
> > > > >> source
> > > > >> > code apex 3.3.0-incubating.
> > > > >> >
> > > > >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches
> for
> > > > hadoop
> > > > >> > binary path and exports an env variable for dtcli to use it.
> > > > >> >
> > > > >> > 3. lib/ -> Libraries for CLI. This list is extracted in a
> similar
> > > way
> > > > to
> > > > >> > how DT Community Edition finds dependency jars.
> > > > >> > Please note that this is the first iteration list of jars. I'm
> > > trying
> > > > to
> > > > >> > narrow this down to only those which are really required.
> > > > >> > The test that I'm running to check if dtcli runs fine with given
> > > > >> dependency
> > > > >> > is to launch+shutdown+kill+status for pi demo.
> > > > >> >
> > > > >> > Above is the directory structure which is required for CLI to
> > work.
> > > > >> > What aniruddha and pradeep has mentioned are some additional
> files
> > > > which
> > > > >> > makes the like of administrator easier. For eg. /etc/skel
> > > > >> >
> > > > >> > Please let me know if this is inline with what you're thinking.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Chinmay.
> > > > >> >
> > > > >> >
> > > > >> > .
> > > > >> > |-- bin
> > > > >> > |   `-- dtcli
> > > > >> > |-- conf
> > > > >> > |   `-- dt-env.sh
> > > > >> > `-- lib
> > > > >> >     |-- ant-1.9.2.jar
> > > > >> >     |-- ant-launcher-1.9.2.jar
> > > > >> >     |-- apex-api-3.3.0-incubating.jar
> > > > >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > > > >> >     |-- apex-common-3.3.0-incubating.jar
> > > > >> >     |-- apex-engine.jar
> > > > >> >     |-- apex-engine-tests.jar
> > > > >> >     |-- async-http-client-1.7.20.jar
> > > > >> >     |-- bval-core-0.5.jar
> > > > >> >     |-- bval-jsr303-0.5.jar
> > > > >> >     |-- commons-beanutils-1.8.3.jar
> > > > >> >     |-- commons-codec-1.10.jar
> > > > >> >     |-- commons-lang3-3.1.jar
> > > > >> >     |-- commons-logging-1.1.3.jar
> > > > >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > > >> >     |-- hadoop-common-2.2.0-tests.jar
> > > > >> >     |-- httpclient-4.3.5.jar
> > > > >> >     |-- httpcore-4.3.2.jar
> > > > >> >     |-- jackson-core-asl-1.9.2.jar
> > > > >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > > >> >     |-- javax.servlet-3.1.jar
> > > > >> >     |-- javax.servlet-api-3.0.1.jar
> > > > >> >     |-- jersey-apache-client4-1.9.jar
> > > > >> >     |-- jersey-client-1.9.jar
> > > > >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > > >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > > >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > > >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > > > >> >     |-- jline-2.11.jar
> > > > >> >     |-- kryo-2.24.0.jar
> > > > >> >     |-- mbassador-1.1.9.jar
> > > > >> >     |-- minlog-1.2.jar
> > > > >> >     |-- netlet-1.2.0.jar
> > > > >> >     |-- netty-3.6.6.Final.jar
> > > > >> >     |-- objenesis-2.1.jar
> > > > >> >     |-- validation-api-1.1.0.Final.jar
> > > > >> >     |-- xbean-asm5-shaded-4.3.jar
> > > > >> >     `-- zip4j-1.3.2.jar
> > > > >> >
> > > > >> > 3 directories, 40 files
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > > thomas@datatorrent.com
> > > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Chinmay,
> > > > >> > >
> > > > >> > > Before discussing where to put the files, let's make sure they
> > are
> > > > >> really
> > > > >> > > needed for the operation of the CLI. As for names, anything
> new
> > > > needs
> > > > >> to
> > > > >> > > reflect Apex in the name and should follow common conventions,
> > > > >> especially
> > > > >> > > in Bigtop where there are many existing integrations to look
> at.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Thomas
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > > > >> > > chinmay@datatorrent.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Really good points Pradeep and Aniruddha.
> > > > >> > > >
> > > > >> > > > 1. I believe we won't need to change the dtcli considering
> it
> > > > works
> > > > >> > with
> > > > >> > > DT
> > > > >> > > > Community edition. We can keep the directory structure
> similar
> > > to
> > > > >> that.
> > > > >> > > > dt-env.sh has variables which contains information required
> > for
> > > > >> dtcli
> > > > >> > to
> > > > >> > > > launch.
> > > > >> > > >
> > > > >> > > > 2. Let me check with bigtop community that whether they
> > > facilitate
> > > > >> the
> > > > >> > > > installation of rpms/debs before user is created. In either
> > > case,
> > > > >> > current
> > > > >> > > > dtcli creates a .dt folder in home directory. Also before
> > > putting
> > > > >> > > anything
> > > > >> > > > in /etc/skel we need to define what are the default contents
> > > that
> > > > >> > should
> > > > >> > > go
> > > > >> > > > to ~/.dt/ folder. If there is no defaults, probably we
> should
> > > not
> > > > >> > > > explicitly add it in /etc/skel.
> > > > >> > > >
> > > > >> > > > 3.  /etc/profile.d approach looks nice. This way contents of
> > > > >> dt-env.sh
> > > > >> > > are
> > > > >> > > > present as env variables. But I see a catch there. Adding
> > > > dt-env.sh
> > > > >> to
> > > > >> > > > /etc/profile.d would make all the variables available at
> > runtime
> > > > all
> > > > >> > the
> > > > >> > > > time. I feel a little skeptical about that. Maybe a possible
> > > > >> collision
> > > > >> > > can
> > > > >> > > > occur with other application vars.
> > > > >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > > > >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > > > >> > > >
> > > > >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > > > >> > > Jira/repository.
> > > > >> > > > Packaging related code usually goes there. (That's what all
> > the
> > > > >> > > components
> > > > >> > > > in bigtop does).
> > > > >> > > > Having said that, once a PR is created, I'll be sharing the
> > link
> > > > of
> > > > >> PR
> > > > >> > > > here, so that apex community as well can review it.
> > > > >> > > >
> > > > >> > > > Thanks,
> > > > >> > > > Chinmay.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > > >> > > > aniruddha@datatorrent.com> wrote:
> > > > >> > > >
> > > > >> > > > > +1 on suggestions and approach.
> > > > >> > > > > We may need to iron out details about exact paths etc.
> > > > >> > > > > Which can be done on jira / PR comments.
> > > > >> > > > > Is that right @dev?
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > > >> > apache@pradeepdalvi.com>
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Inline comments...
> > > > >> > > > > >
> > > > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > >> > > > > > chinmay@datatorrent.com>
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > @Thomas, Not all the jar present in DT community
> edition
> > > > will
> > > > >> be
> > > > >> > > > > included
> > > > >> > > > > > > there.
> > > > >> > > > > > >
> > > > >> > > > > > > Here is the list of jars I found from common
> > dependencies
> > > > >> between
> > > > >> > > of
> > > > >> > > > > apex
> > > > >> > > > > > > and DT Community edition:
> > > > >> > > > > > > netlet-1.2.0.jar
> > > > >> > > > > > > kryo-2.24.0.jar
> > > > >> > > > > > > jackson-core-asl-1.9.2.jar
> > > > >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > >> > > > > > > async-http-client-1.7.20.jar
> > > > >> > > > > > > netty-3.6.6.Final.jar
> > > > >> > > > > > > validation-api-1.1.0.Final.jar
> > > > >> > > > > > > bval-jsr303-0.5.jar
> > > > >> > > > > > > bval-core-0.5.jar
> > > > >> > > > > > > commons-lang3-3.1.jar
> > > > >> > > > > > > commons-beanutils-1.8.3.jar
> > > > >> > > > > > > httpclient-4.3.5.jar
> > > > >> > > > > > > commons-codec-1.10.jar
> > > > >> > > > > > > zip4j-1.3.2.jar
> > > > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > >> > > > > > > jersey-apache-client4-1.9.jar
> > > > >> > > > > > > jline-2.11.jar
> > > > >> > > > > > > ant-1.9.2.jar
> > > > >> > > > > > > ant-launcher-1.9.2.jar
> > > > >> > > > > > > mbassador-1.1.9.jar
> > > > >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > >> > > > > > > jackson-xc-1.9.2.jar
> > > > >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > > >> > > > > > >
> > > > >> > > > > > > Ofcourse, I'll be running some tests do check that
> dtcli
> > > > works
> > > > >> > > > properly
> > > > >> > > > > > for
> > > > >> > > > > > > launch+shutdown+kill of apps with only these libraries
> > > > >> present in
> > > > >> > > > > > isolation
> > > > >> > > > > > > without dependency on local m2.
> > > > >> > > > > > > I'm believe that there are unwanted jars which are
> used
> > > for
> > > > >> > compile
> > > > >> > > > > time
> > > > >> > > > > > > dependency and not runtime in above list which I can
> > drop
> > > to
> > > > >> keep
> > > > >> > > > > package
> > > > >> > > > > > > size minimal.
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > @Anniruddha,
> > > > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh
> is
> > > > >> required.
> > > > >> > > > Only
> > > > >> > > > > > > required values I see in it are classpath. and dtcli
> is
> > > > >> capable
> > > > >> > of
> > > > >> > > > > > building
> > > > >> > > > > > > it on the fly.
> > > > >> > > > > > >
> > > > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can
> > > find
> > > > >> the
> > > > >> > jar
> > > > >> > > > > path,
> > > > >> > > > > > > dt-env.sh and all such conf files. If we need a change
> > in
> > > > >> dtcli,
> > > > >> > > then
> > > > >> > > > > > > community's opinion is required for whether dtcli
> should
> > > > >> change
> > > > >> > in
> > > > >> > > > our
> > > > >> > > > > > repo
> > > > >> > > > > > > or a copy of that with required changes exist in
> bigtop
> > > repo
> > > > >> > until
> > > > >> > > we
> > > > >> > > > > > make
> > > > >> > > > > > > the dtcli generic enough.
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Can we have these file paths set using environment
> > variables
> > > > >> with
> > > > >> > > some
> > > > >> > > > > > default values in dtcli?
> > > > >> > > > > > Then we can set such params in dt-env.sh.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > > 3. dt-sited.xml -> How is this file different from
> > > > >> dt-site.xml?
> > > > >> > Not
> > > > >> > > > > sure
> > > > >> > > > > > if
> > > > >> > > > > > > adding in /etc/skel would help. Files & Dirs from
> > > /etc/skel
> > > > >> are
> > > > >> > > > copied
> > > > >> > > > > to
> > > > >> > > > > > > home of new user when useradd program is called. But
> for
> > > > >> existing
> > > > >> > > > users
> > > > >> > > > > > > this won't be of any use. Moreover, dtcli creates
> ~/.dt/
> > > on
> > > > >> the
> > > > >> > fly
> > > > >> > > > if
> > > > >> > > > > > not
> > > > >> > > > > > > encountered for the first time. Again correct me if
> I'm
> > > > wrong.
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Yes, you are right. However this would probably be
> > necessary
> > > > >> step
> > > > >> > for
> > > > >> > > > > > rpm/deb, as they may also get installed during OS
> install
> > > and
> > > > >> > before
> > > > >> > > > user
> > > > >> > > > > > accounts were created.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its
> > the
> > > > best
> > > > >> > > idea.
> > > > >> > > > > > We'll
> > > > >> > > > > > > anyway put dtcli in location which is by default
> present
> > > in
> > > > >> path
> > > > >> > > i.e.
> > > > >> > > > > > > /usr/bin.
> > > > >> > > > > >
> > > > >> > > > > > For other env variables specific to apex, I'll prefer to
> > use
> > > > >> > > dt-env.sh
> > > > >> > > > > and
> > > > >> > > > > > > source it in dtcli rather than changing bashrc etc...
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be
> the
> > > > >> cleaner
> > > > >> > way
> > > > >> > > > to
> > > > >> > > > > > achieve the same.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > Thanks,
> > > > >> > > > > > > Chinmay.
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > >> > > > > > > aniruddha@datatorrent.com> wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi,
> > > > >> > > > > > > >
> > > > >> > > > > > > > @Chinmay,
> > > > >> > > > > > > >
> > > > >> > > > > > > > We need to consider following:
> > > > >> > > > > > > > System wide default config files can be located at
> > > > following
> > > > >> > > > > locations:
> > > > >> > > > > > > >
> > > > >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> > > > behaviour
> > > > >> on
> > > > >> > > how
> > > > >> > > > it
> > > > >> > > > > > > finds
> > > > >> > > > > > > > those locations)
> > > > >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> > > > >> behaviour on
> > > > >> > > how
> > > > >> > > > > it
> > > > >> > > > > > > > finds those locations)
> > > > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new
> > > users
> > > > >> that
> > > > >> > > will
> > > > >> > > > > be
> > > > >> > > > > > > > created in system in future.)
> > > > >> > > > > > > >
> > > > >> > > > > > > > We may also have to modify bashrc / bashprofile for
> > > > >> population
> > > > >> > > any
> > > > >> > > > > > > > variables that are required.
> > > > >> > > > > > > >
> > > > >> > > > > > > > @dev, please put in your comments / suggestion.
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thanks,
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > Aniruddha
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > > >> > > > > thomas@datatorrent.com>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > -->
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar
> <
> > > > >> > > > > > > > chinmay@datatorrent.com
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > Hello Everyone!!
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of
> > apex,
> > > > here
> > > > >> > are
> > > > >> > > > some
> > > > >> > > > > > > > > proposals
> > > > >> > > > > > > > > > about package structure etc..
> > > > >> > > > > > > > > > Before posting it on bbigtop mailing list, I
> have
> > > some
> > > > >> > > question
> > > > >> > > > > for
> > > > >> > > > > > > > apex
> > > > >> > > > > > > > > > community.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Proposed Directory structure of apex package for
> > > both
> > > > >> deb &
> > > > >> > > > rpm:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > >> > > > > > > > > >
> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > >> > > > > > > > > >
> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > <version> = 3.3.0-incubating.
> > > > >> > > > > > > > > > <other dependent jars> = All the 3rd party jars
> > > which
> > > > >> are
> > > > >> > > > > required
> > > > >> > > > > > > for
> > > > >> > > > > > > > > apex
> > > > >> > > > > > > > > > to run. Usually the dependencies are packaged as
> > > part
> > > > of
> > > > >> > > > rpm/deb
> > > > >> > > > > by
> > > > >> > > > > > > any
> > > > >> > > > > > > > > > software in bigtop.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Can you specify what those jars are in Bigtop
> > context?
> > > > >> Same
> > > > >> > as
> > > > >> > > > > > shipped
> > > > >> > > > > > > > with
> > > > >> > > > > > > > > DT community addition under lib/ ?
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > <package LICENSE> = Licenses of corresponding of
> > 3rd
> > > > >> party
> > > > >> > > jars
> > > > >> > > > > > which
> > > > >> > > > > > > > > needs
> > > > >> > > > > > > > > > to included while packaging.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Questions related to this:
> > > > >> > > > > > > > > > 1. Should we call the cli of apex as "apex"
> > instead
> > > of
> > > > >> > > "dtcli"
> > > > >> > > > in
> > > > >> > > > > > > > bigtop
> > > > >> > > > > > > > > > package?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I think we should keep the name until we are able
> to
> > > > >> change
> > > > >> > it
> > > > >> > > in
> > > > >> > > > > > Apex.
> > > > >> > > > > > > > > Otherwise this may get confusing.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > 2. I see that all softwares in bigtop have man
> > page
> > > > for
> > > > >> > their
> > > > >> > > > > > > > > executables.
> > > > >> > > > > > > > > > I think we should have it too for dtcli. Is
> there
> > > any
> > > > >> > > > > documentation
> > > > >> > > > > > > > > which I
> > > > >> > > > > > > > > > can convert to man page? or can I use output of
> > > "dtcli
> > > > >> > > --help"?
> > > > >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop
> as
> > > > >> 3.3.0 OR
> > > > >> > > > > > > > > > 3.3.0-incubating?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Has to be -incubating
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend
> > on
> > > > >> 2.7.1
> > > > >> > > > > version
> > > > >> > > > > > of
> > > > >> > > > > > > > > > bigtop hadoop? Any problems that we see with
> this
> > > > >> > dependency?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Should work without changed. I thought we
> certified
> > > > >> against
> > > > >> > > > 2.7.0?
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > 5. Following is the apache mirror from which
> > bigtop
> > > > will
> > > > >> > pick
> > > > >> > > > the
> > > > >> > > > > > > apex
> > > > >> > > > > > > > > > source for compilation and packaging. Please
> > correct
> > > > if
> > > > >> > > > > incorrect:
> > > > >> > > > > > > > > >
> > > > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used
> by
> > > all
> > > > >> the
> > > > >> > > > > > softwares
> > > > >> > > > > > > in
> > > > >> > > > > > > > > > bigtop.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Please share your opinion.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > Chinmay.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> > Kolhatkar <
> > > > >> > > > > > > > > > chinmay@datatorrent.com>
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > I've already started discussion on bigtop dev
> > > > mailing
> > > > >> > list
> > > > >> > > > for
> > > > >> > > > > > > > > > > integration. Also created a Jira for the same.
> > For
> > > > >> this
> > > > >> > > > > > communities
> > > > >> > > > > > > > > > > reference, here is the bigtop Jira:
> BIGTOP-2313.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > >> > > > > > > thomas@datatorrent.com
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >> Chinmay,
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >> I don't see anything under prerequisites that
> > > would
> > > > >> be a
> > > > >> > > > > > problem.
> > > > >> > > > > > > We
> > > > >> > > > > > > > > > >> looked
> > > > >> > > > > > > > > > >> at the ASF licencing compatibility as part of
> > > > >> becoming
> > > > >> > an
> > > > >> > > > > > > incubator
> > > > >> > > > > > > > > > >> project.
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >> Please focus on the packaging during the next
> > > > weeks.
> > > > >> > Since
> > > > >> > > > the
> > > > >> > > > > > > work
> > > > >> > > > > > > > > will
> > > > >> > > > > > > > > > >> be
> > > > >> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> > > > should
> > > > >> > also
> > > > >> > > be
> > > > >> > > > > > > there.
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >> Would be good to have the packaging in place
> by
> > > end
> > > > >> Feb.
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >> Thanks,
> > > > >> > > > > > > > > > >> Thomas
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> > > Kolhatkar
> > > > <
> > > > >> > > > > > > > > > >> chinmay@datatorrent.com>
> > > > >> > > > > > > > > > >> wrote:
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >> > Hi All,
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > We're planning a work on adding Apache Apex
> > as
> > > a
> > > > >> > > component
> > > > >> > > > > to
> > > > >> > > > > > > > Apache
> > > > >> > > > > > > > > > >> > Bigtop.
> > > > >> > > > > > > > > > >> > Bigtop is the packaging system for the
> Apache
> > > big
> > > > >> data
> > > > >> > > > > > > ecosystem.
> > > > >> > > > > > > > > > >> Several
> > > > >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE
> > for
> > > > the
> > > > >> > same:
> > > > >> > > > > > > > > > >> >
> > > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > Proposed plan of execution is as follows:
> > > > >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> > > expectation
> > > > >> for
> > > > >> > new
> > > > >> > > > > > > > components
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > >> get
> > > > >> > > > > > > > > > >> > integrated into Bigtop.
> > > > >> > > > > > > > > > >> > Here is the list of it:
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > Most of them seems to be standard ASF based
> > > > >> > > requirements,
> > > > >> > > > > but
> > > > >> > > > > > > few
> > > > >> > > > > > > > > need
> > > > >> > > > > > > > > > >> to
> > > > >> > > > > > > > > > >> > be checked for:
> > > > >> > > > > > > > > > >> > 1. Software projects are expected to be
> > > Licensed
> > > > >> under
> > > > >> > > > > Apache
> > > > >> > > > > > > > > License,
> > > > >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
> > > expected
> > > > >> to be
> > > > >> > > > > > > compatible
> > > > >> > > > > > > > > with
> > > > >> > > > > > > > > > >> this
> > > > >> > > > > > > > > > >> > license)
> > > > >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to
> check
> > > if
> > > > >> > > > > dependencies
> > > > >> > > > > > of
> > > > >> > > > > > > > > Apex
> > > > >> > > > > > > > > > >> are
> > > > >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would
> > be
> > > a
> > > > >> > > > > verification
> > > > >> > > > > > > > check.
> > > > >> > > > > > > > > > >> > 2. Software projects are expected to be
> > > > compatible
> > > > >> > with
> > > > >> > > > all
> > > > >> > > > > of
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > >> > supported platforms that Bigtop
> distribution
> > is
> > > > >> > > targeting
> > > > >> > > > > > > > > > >> >     - This needs verifying whether our
> > software
> > > > >> runs
> > > > >> > > fine
> > > > >> > > > in
> > > > >> > > > > > > > > centos-6
> > > > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > > > >> > opensuse-13.2.
> > > > >> > > > > > > > > > >> > 3. What smoke tests that should be added
> for
> > > > >> > deployment.
> > > > >> > > > > > > > > > >> > 4. Identifying the test artifacts which
> goes
> > > > beyond
> > > > >> > > smoke
> > > > >> > > > > test
> > > > >> > > > > > > > > > >> >     - These are basically the integration
> > tests
> > > > for
> > > > >> > > > > > verification
> > > > >> > > > > > > > > after
> > > > >> > > > > > > > > > >> the
> > > > >> > > > > > > > > > >> > deployment. This is a soft requirement, but
> > aim
> > > > is
> > > > >> to
> > > > >> > > > > achieve
> > > > >> > > > > > > this
> > > > >> > > > > > > > > as
> > > > >> > > > > > > > > > >> well
> > > > >> > > > > > > > > > >> > or at least have explanation why not to
> > > include.
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > If there are any from the link which
> > explicitly
> > > > >> needs
> > > > >> > to
> > > > >> > > > be
> > > > >> > > > > > > > checked
> > > > >> > > > > > > > > > >> other
> > > > >> > > > > > > > > > >> > than above 4, please let us know.
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > *Step 2) Adding Apex as component to
> Bigtop*
> > > > >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it
> > was
> > > > >> learnt
> > > > >> > > that
> > > > >> > > > > the
> > > > >> > > > > > > > > bigtop
> > > > >> > > > > > > > > > >> > community want to see the addition of new
> > > > >> components
> > > > >> > in
> > > > >> > > > > > phases.
> > > > >> > > > > > > > Here
> > > > >> > > > > > > > > > are
> > > > >> > > > > > > > > > >> > the phases:
> > > > >> > > > > > > > > > >> > 1. Packaging
> > > > >> > > > > > > > > > >> >     - This needs creating of package i.e.
> > rpm &
> > > > deb
> > > > >> > > files.
> > > > >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> > > DISCLAMER,
> > > > >> > NOTES
> > > > >> > > > etc
> > > > >> > > > > if
> > > > >> > > > > > > any
> > > > >> > > > > > > > > > >> needed.
> > > > >> > > > > > > > > > >> >     - Any documentation that need to be
> added
> > > to
> > > > >> > > > > distribution
> > > > >> > > > > > of
> > > > >> > > > > > > > our
> > > > >> > > > > > > > > > >> > software.
> > > > >> > > > > > > > > > >> >     - Any license information of
> dependencies
> > > > >> required
> > > > >> > > to
> > > > >> > > > be
> > > > >> > > > > > > added
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > >> > package
> > > > >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> > > > >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment
> > and
> > > > >> > > > configuration
> > > > >> > > > > > > > > > >> >     - Add puppet recipes for automatic
> > > deployment
> > > > >> > > > > > > > > > >> > 4. Integration tests
> > > > >> > > > > > > > > > >> >     - For verification of deployments.
> > > > >> > > > > > > > > > >> > 5. license clearance:
> > > > >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new
> > stuff
> > > > is
> > > > >> > > > compliant
> > > > >> > > > > > > with
> > > > >> > > > > > > > > ASF
> > > > >> > > > > > > > > > >> > license requirements. If you add code
> > licenses
> > > > >> under
> > > > >> > > > > different
> > > > >> > > > > > > > > > licenses,
> > > > >> > > > > > > > > > >> > those would need to be listed in the
> NOTICE.
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > Please share your thoughts on the approach.
> > > > >> > > > > > > > > > >> > We'll start corresponding communication on
> > > bigtop
> > > > >> > > mailing
> > > > >> > > > > list
> > > > >> > > > > > > as
> > > > >> > > > > > > > > > well.
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > We have some specific questions/suggestions
> > > > >> related to
> > > > >> > > > what
> > > > >> > > > > > > should
> > > > >> > > > > > > > > be
> > > > >> > > > > > > > > > >> the
> > > > >> > > > > > > > > > >> > content of the package and what should be
> the
> > > > smoke
> > > > >> > > tests,
> > > > >> > > > > but
> > > > >> > > > > > > in
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > >> > interest of not having too much content
> here,
> > > > we'll
> > > > >> > put
> > > > >> > > > the
> > > > >> > > > > > > > > questions
> > > > >> > > > > > > > > > >> as a
> > > > >> > > > > > > > > > >> > separate mail in this mailthread.
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > Thanks,
> > > > >> > > > > > > > > > >> > Chinmay.
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

I'm not convinced dt-env.sh is even needed. In development mode, the file
doesn't exist but it will still find Hadoop.

The absolute path to hadoop script was introduced to deal with the layout
of specific distros. You will know whether that applies to Bigtop or not.

Thomas

On Sun, Feb 14, 2016 at 8:16 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Hi Thomas,
>
> There are only 2 things that are required from dt-env.sh:
> 1. DT_HADOOP variable which is exported by dt-env.sh This variable has
> absolute path for hadoop binary
> 2. A find_hadoop() method which find the path of hadoop binary.
>
> JAVA_HOME is not required for this.
>
> The idea is not to change the dtcli. Current dtcli (in apex
> 3.3.0-incubating src code), sources dt-env.sh for above 2 information.
> Hence included dt-env.sh
>
> If the community agrees, we can make dtcli more generic so that in future
> packing of bigtop, dt-env.sh can be completely excluded OR at least change
> the name to apex-env.sh.
>
> Thanks,
> Chinmay.
>
>
>
> On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
> > Chinmay,
> >
> > Can you please specify what settings in dt-env.sh are required in a
> bigtop
> > environment?
> >
> > Location of hadoop and JAVA_HOME should be provided by the stack already?
> >
> > Thanks,
> > Thomas
> >
> >
> > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Here is a suggestion I have related to name change:
> > >
> > > 1. While packaging we change the name of "dtcli" script to "apex". This
> > is
> > > done so that in future when we change the name to "apex", users of
> bigtop
> > > apex don't have to transition much.
> > > 2. We also keep a symlink named dtcli which points to apex script. This
> > is
> > > for backward compatibility.
> > > 3. We still keep conf/dt-env.sh for first integration in bigtop. This
> is
> > in
> > > the interest of not changing the content of dtcli or apex script.
> > > 4. In apex 3.4.0, we take care of all these naming related changes and
> > > update bigtop repository later to remove references to dtcli all
> > together.
> > >
> > > Please share your thoughts on above approach.
> > >
> > > Also, please share what could be the man page content for "apex".
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > >
> > > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <chinmay@apache.org
> >
> > > wrote:
> > >
> > > > Yes.. It'll be build from source tar downloaded from one of the
> apache
> > > > mirror: apache.osuosl.org
> > > >
> > > > The reason for dt-env.sh is it's sourced from dtcli. This leaves us 3
> > > > options:
> > > > If we should not have any dt-env.sh, dtcli would need a change while
> > > > packaging.
> > > >
> > > > But if we want to use dtcli as it is, then we would need dt-env.sh
> > > >
> > > >
> > > >
> > > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> thomas@datatorrent.com>
> > > > wrote:
> > > >
> > > >> Looks good overall, though there shouldn't be any dt-env.sh
> > > >>
> > > >> What will the bigtop package be built from, the source tar?
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > chinmay@apache.org>
> > > >> wrote:
> > > >>
> > > >> > Hi Thomas,
> > > >> >
> > > >> > Thanks for the feedback.
> > > >> >
> > > >> > First, we're not changing the name anywhere. We'll follow what
> > > >> currently is
> > > >> > in source tarball for 3.3.0-incubating version.
> > > >> >
> > > >> > Secondly, I've mentioned a directory structure below which is
> inline
> > > >> with
> > > >> > other existing integrations with bigtop.
> > > >> >
> > > >> > The need for each file is as follows:
> > > >> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/
> of
> > > >> source
> > > >> > code apex 3.3.0-incubating.
> > > >> >
> > > >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for
> > > hadoop
> > > >> > binary path and exports an env variable for dtcli to use it.
> > > >> >
> > > >> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar
> > way
> > > to
> > > >> > how DT Community Edition finds dependency jars.
> > > >> > Please note that this is the first iteration list of jars. I'm
> > trying
> > > to
> > > >> > narrow this down to only those which are really required.
> > > >> > The test that I'm running to check if dtcli runs fine with given
> > > >> dependency
> > > >> > is to launch+shutdown+kill+status for pi demo.
> > > >> >
> > > >> > Above is the directory structure which is required for CLI to
> work.
> > > >> > What aniruddha and pradeep has mentioned are some additional files
> > > which
> > > >> > makes the like of administrator easier. For eg. /etc/skel
> > > >> >
> > > >> > Please let me know if this is inline with what you're thinking.
> > > >> >
> > > >> > Thanks,
> > > >> > Chinmay.
> > > >> >
> > > >> >
> > > >> > .
> > > >> > |-- bin
> > > >> > |   `-- dtcli
> > > >> > |-- conf
> > > >> > |   `-- dt-env.sh
> > > >> > `-- lib
> > > >> >     |-- ant-1.9.2.jar
> > > >> >     |-- ant-launcher-1.9.2.jar
> > > >> >     |-- apex-api-3.3.0-incubating.jar
> > > >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > > >> >     |-- apex-common-3.3.0-incubating.jar
> > > >> >     |-- apex-engine.jar
> > > >> >     |-- apex-engine-tests.jar
> > > >> >     |-- async-http-client-1.7.20.jar
> > > >> >     |-- bval-core-0.5.jar
> > > >> >     |-- bval-jsr303-0.5.jar
> > > >> >     |-- commons-beanutils-1.8.3.jar
> > > >> >     |-- commons-codec-1.10.jar
> > > >> >     |-- commons-lang3-3.1.jar
> > > >> >     |-- commons-logging-1.1.3.jar
> > > >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > >> >     |-- hadoop-common-2.2.0-tests.jar
> > > >> >     |-- httpclient-4.3.5.jar
> > > >> >     |-- httpcore-4.3.2.jar
> > > >> >     |-- jackson-core-asl-1.9.2.jar
> > > >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > >> >     |-- javax.servlet-3.1.jar
> > > >> >     |-- javax.servlet-api-3.0.1.jar
> > > >> >     |-- jersey-apache-client4-1.9.jar
> > > >> >     |-- jersey-client-1.9.jar
> > > >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > > >> >     |-- jline-2.11.jar
> > > >> >     |-- kryo-2.24.0.jar
> > > >> >     |-- mbassador-1.1.9.jar
> > > >> >     |-- minlog-1.2.jar
> > > >> >     |-- netlet-1.2.0.jar
> > > >> >     |-- netty-3.6.6.Final.jar
> > > >> >     |-- objenesis-2.1.jar
> > > >> >     |-- validation-api-1.1.0.Final.jar
> > > >> >     |-- xbean-asm5-shaded-4.3.jar
> > > >> >     `-- zip4j-1.3.2.jar
> > > >> >
> > > >> > 3 directories, 40 files
> > > >> >
> > > >> >
> > > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > thomas@datatorrent.com
> > > >
> > > >> > wrote:
> > > >> >
> > > >> > > Chinmay,
> > > >> > >
> > > >> > > Before discussing where to put the files, let's make sure they
> are
> > > >> really
> > > >> > > needed for the operation of the CLI. As for names, anything new
> > > needs
> > > >> to
> > > >> > > reflect Apex in the name and should follow common conventions,
> > > >> especially
> > > >> > > in Bigtop where there are many existing integrations to look at.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Thomas
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > > >> > > chinmay@datatorrent.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Really good points Pradeep and Aniruddha.
> > > >> > > >
> > > >> > > > 1. I believe we won't need to change the dtcli considering it
> > > works
> > > >> > with
> > > >> > > DT
> > > >> > > > Community edition. We can keep the directory structure similar
> > to
> > > >> that.
> > > >> > > > dt-env.sh has variables which contains information required
> for
> > > >> dtcli
> > > >> > to
> > > >> > > > launch.
> > > >> > > >
> > > >> > > > 2. Let me check with bigtop community that whether they
> > facilitate
> > > >> the
> > > >> > > > installation of rpms/debs before user is created. In either
> > case,
> > > >> > current
> > > >> > > > dtcli creates a .dt folder in home directory. Also before
> > putting
> > > >> > > anything
> > > >> > > > in /etc/skel we need to define what are the default contents
> > that
> > > >> > should
> > > >> > > go
> > > >> > > > to ~/.dt/ folder. If there is no defaults, probably we should
> > not
> > > >> > > > explicitly add it in /etc/skel.
> > > >> > > >
> > > >> > > > 3.  /etc/profile.d approach looks nice. This way contents of
> > > >> dt-env.sh
> > > >> > > are
> > > >> > > > present as env variables. But I see a catch there. Adding
> > > dt-env.sh
> > > >> to
> > > >> > > > /etc/profile.d would make all the variables available at
> runtime
> > > all
> > > >> > the
> > > >> > > > time. I feel a little skeptical about that. Maybe a possible
> > > >> collision
> > > >> > > can
> > > >> > > > occur with other application vars.
> > > >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > > >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > > >> > > >
> > > >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > > >> > > Jira/repository.
> > > >> > > > Packaging related code usually goes there. (That's what all
> the
> > > >> > > components
> > > >> > > > in bigtop does).
> > > >> > > > Having said that, once a PR is created, I'll be sharing the
> link
> > > of
> > > >> PR
> > > >> > > > here, so that apex community as well can review it.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Chinmay.
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > >> > > > aniruddha@datatorrent.com> wrote:
> > > >> > > >
> > > >> > > > > +1 on suggestions and approach.
> > > >> > > > > We may need to iron out details about exact paths etc.
> > > >> > > > > Which can be done on jira / PR comments.
> > > >> > > > > Is that right @dev?
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > >> > apache@pradeepdalvi.com>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Inline comments...
> > > >> > > > > >
> > > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > >> > > > > > chinmay@datatorrent.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > @Thomas, Not all the jar present in DT community edition
> > > will
> > > >> be
> > > >> > > > > included
> > > >> > > > > > > there.
> > > >> > > > > > >
> > > >> > > > > > > Here is the list of jars I found from common
> dependencies
> > > >> between
> > > >> > > of
> > > >> > > > > apex
> > > >> > > > > > > and DT Community edition:
> > > >> > > > > > > netlet-1.2.0.jar
> > > >> > > > > > > kryo-2.24.0.jar
> > > >> > > > > > > jackson-core-asl-1.9.2.jar
> > > >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > >> > > > > > > async-http-client-1.7.20.jar
> > > >> > > > > > > netty-3.6.6.Final.jar
> > > >> > > > > > > validation-api-1.1.0.Final.jar
> > > >> > > > > > > bval-jsr303-0.5.jar
> > > >> > > > > > > bval-core-0.5.jar
> > > >> > > > > > > commons-lang3-3.1.jar
> > > >> > > > > > > commons-beanutils-1.8.3.jar
> > > >> > > > > > > httpclient-4.3.5.jar
> > > >> > > > > > > commons-codec-1.10.jar
> > > >> > > > > > > zip4j-1.3.2.jar
> > > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > >> > > > > > > jersey-apache-client4-1.9.jar
> > > >> > > > > > > jline-2.11.jar
> > > >> > > > > > > ant-1.9.2.jar
> > > >> > > > > > > ant-launcher-1.9.2.jar
> > > >> > > > > > > mbassador-1.1.9.jar
> > > >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > >> > > > > > > jackson-xc-1.9.2.jar
> > > >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > >> > > > > > >
> > > >> > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> > > works
> > > >> > > > properly
> > > >> > > > > > for
> > > >> > > > > > > launch+shutdown+kill of apps with only these libraries
> > > >> present in
> > > >> > > > > > isolation
> > > >> > > > > > > without dependency on local m2.
> > > >> > > > > > > I'm believe that there are unwanted jars which are used
> > for
> > > >> > compile
> > > >> > > > > time
> > > >> > > > > > > dependency and not runtime in above list which I can
> drop
> > to
> > > >> keep
> > > >> > > > > package
> > > >> > > > > > > size minimal.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > @Anniruddha,
> > > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> > > >> required.
> > > >> > > > Only
> > > >> > > > > > > required values I see in it are classpath. and dtcli is
> > > >> capable
> > > >> > of
> > > >> > > > > > building
> > > >> > > > > > > it on the fly.
> > > >> > > > > > >
> > > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can
> > find
> > > >> the
> > > >> > jar
> > > >> > > > > path,
> > > >> > > > > > > dt-env.sh and all such conf files. If we need a change
> in
> > > >> dtcli,
> > > >> > > then
> > > >> > > > > > > community's opinion is required for whether dtcli should
> > > >> change
> > > >> > in
> > > >> > > > our
> > > >> > > > > > repo
> > > >> > > > > > > or a copy of that with required changes exist in bigtop
> > repo
> > > >> > until
> > > >> > > we
> > > >> > > > > > make
> > > >> > > > > > > the dtcli generic enough.
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > > > Can we have these file paths set using environment
> variables
> > > >> with
> > > >> > > some
> > > >> > > > > > default values in dtcli?
> > > >> > > > > > Then we can set such params in dt-env.sh.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > > 3. dt-sited.xml -> How is this file different from
> > > >> dt-site.xml?
> > > >> > Not
> > > >> > > > > sure
> > > >> > > > > > if
> > > >> > > > > > > adding in /etc/skel would help. Files & Dirs from
> > /etc/skel
> > > >> are
> > > >> > > > copied
> > > >> > > > > to
> > > >> > > > > > > home of new user when useradd program is called. But for
> > > >> existing
> > > >> > > > users
> > > >> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/
> > on
> > > >> the
> > > >> > fly
> > > >> > > > if
> > > >> > > > > > not
> > > >> > > > > > > encountered for the first time. Again correct me if I'm
> > > wrong.
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > > > Yes, you are right. However this would probably be
> necessary
> > > >> step
> > > >> > for
> > > >> > > > > > rpm/deb, as they may also get installed during OS install
> > and
> > > >> > before
> > > >> > > > user
> > > >> > > > > > accounts were created.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its
> the
> > > best
> > > >> > > idea.
> > > >> > > > > > We'll
> > > >> > > > > > > anyway put dtcli in location which is by default present
> > in
> > > >> path
> > > >> > > i.e.
> > > >> > > > > > > /usr/bin.
> > > >> > > > > >
> > > >> > > > > > For other env variables specific to apex, I'll prefer to
> use
> > > >> > > dt-env.sh
> > > >> > > > > and
> > > >> > > > > > > source it in dtcli rather than changing bashrc etc...
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> > > >> cleaner
> > > >> > way
> > > >> > > > to
> > > >> > > > > > achieve the same.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > > > Chinmay.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > >> > > > > > > aniruddha@datatorrent.com> wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi,
> > > >> > > > > > > >
> > > >> > > > > > > > @Chinmay,
> > > >> > > > > > > >
> > > >> > > > > > > > We need to consider following:
> > > >> > > > > > > > System wide default config files can be located at
> > > following
> > > >> > > > > locations:
> > > >> > > > > > > >
> > > >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> > > behaviour
> > > >> on
> > > >> > > how
> > > >> > > > it
> > > >> > > > > > > finds
> > > >> > > > > > > > those locations)
> > > >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> > > >> behaviour on
> > > >> > > how
> > > >> > > > > it
> > > >> > > > > > > > finds those locations)
> > > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new
> > users
> > > >> that
> > > >> > > will
> > > >> > > > > be
> > > >> > > > > > > > created in system in future.)
> > > >> > > > > > > >
> > > >> > > > > > > > We may also have to modify bashrc / bashprofile for
> > > >> population
> > > >> > > any
> > > >> > > > > > > > variables that are required.
> > > >> > > > > > > >
> > > >> > > > > > > > @dev, please put in your comments / suggestion.
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks,
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > Aniruddha
> > > >> > > > > > > >
> > > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > >> > > > > thomas@datatorrent.com>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > -->
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > >> > > > > > > > chinmay@datatorrent.com
> > > >> > > > > > > > > >
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Hello Everyone!!
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of
> apex,
> > > here
> > > >> > are
> > > >> > > > some
> > > >> > > > > > > > > proposals
> > > >> > > > > > > > > > about package structure etc..
> > > >> > > > > > > > > > Before posting it on bbigtop mailing list, I have
> > some
> > > >> > > question
> > > >> > > > > for
> > > >> > > > > > > > apex
> > > >> > > > > > > > > > community.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Proposed Directory structure of apex package for
> > both
> > > >> deb &
> > > >> > > > rpm:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > >> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > >> > > > > > > > >
> > > >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > >> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > <version> = 3.3.0-incubating.
> > > >> > > > > > > > > > <other dependent jars> = All the 3rd party jars
> > which
> > > >> are
> > > >> > > > > required
> > > >> > > > > > > for
> > > >> > > > > > > > > apex
> > > >> > > > > > > > > > to run. Usually the dependencies are packaged as
> > part
> > > of
> > > >> > > > rpm/deb
> > > >> > > > > by
> > > >> > > > > > > any
> > > >> > > > > > > > > > software in bigtop.
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > Can you specify what those jars are in Bigtop
> context?
> > > >> Same
> > > >> > as
> > > >> > > > > > shipped
> > > >> > > > > > > > with
> > > >> > > > > > > > > DT community addition under lib/ ?
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > > <package LICENSE> = Licenses of corresponding of
> 3rd
> > > >> party
> > > >> > > jars
> > > >> > > > > > which
> > > >> > > > > > > > > needs
> > > >> > > > > > > > > > to included while packaging.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Questions related to this:
> > > >> > > > > > > > > > 1. Should we call the cli of apex as "apex"
> instead
> > of
> > > >> > > "dtcli"
> > > >> > > > in
> > > >> > > > > > > > bigtop
> > > >> > > > > > > > > > package?
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > I think we should keep the name until we are able to
> > > >> change
> > > >> > it
> > > >> > > in
> > > >> > > > > > Apex.
> > > >> > > > > > > > > Otherwise this may get confusing.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > > 2. I see that all softwares in bigtop have man
> page
> > > for
> > > >> > their
> > > >> > > > > > > > > executables.
> > > >> > > > > > > > > > I think we should have it too for dtcli. Is there
> > any
> > > >> > > > > documentation
> > > >> > > > > > > > > which I
> > > >> > > > > > > > > > can convert to man page? or can I use output of
> > "dtcli
> > > >> > > --help"?
> > > >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> > > >> 3.3.0 OR
> > > >> > > > > > > > > > 3.3.0-incubating?
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > Has to be -incubating
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend
> on
> > > >> 2.7.1
> > > >> > > > > version
> > > >> > > > > > of
> > > >> > > > > > > > > > bigtop hadoop? Any problems that we see with this
> > > >> > dependency?
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > Should work without changed. I thought we certified
> > > >> against
> > > >> > > > 2.7.0?
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > > 5. Following is the apache mirror from which
> bigtop
> > > will
> > > >> > pick
> > > >> > > > the
> > > >> > > > > > > apex
> > > >> > > > > > > > > > source for compilation and packaging. Please
> correct
> > > if
> > > >> > > > > incorrect:
> > > >> > > > > > > > > >
> > > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
> > all
> > > >> the
> > > >> > > > > > softwares
> > > >> > > > > > > in
> > > >> > > > > > > > > > bigtop.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Please share your opinion.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks,
> > > >> > > > > > > > > > Chinmay.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> Kolhatkar <
> > > >> > > > > > > > > > chinmay@datatorrent.com>
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I've already started discussion on bigtop dev
> > > mailing
> > > >> > list
> > > >> > > > for
> > > >> > > > > > > > > > > integration. Also created a Jira for the same.
> For
> > > >> this
> > > >> > > > > > communities
> > > >> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > >> > > > > > > thomas@datatorrent.com
> > > >> > > > > > > > >
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >> Chinmay,
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >> I don't see anything under prerequisites that
> > would
> > > >> be a
> > > >> > > > > > problem.
> > > >> > > > > > > We
> > > >> > > > > > > > > > >> looked
> > > >> > > > > > > > > > >> at the ASF licencing compatibility as part of
> > > >> becoming
> > > >> > an
> > > >> > > > > > > incubator
> > > >> > > > > > > > > > >> project.
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >> Please focus on the packaging during the next
> > > weeks.
> > > >> > Since
> > > >> > > > the
> > > >> > > > > > > work
> > > >> > > > > > > > > will
> > > >> > > > > > > > > > >> be
> > > >> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> > > should
> > > >> > also
> > > >> > > be
> > > >> > > > > > > there.
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >> Would be good to have the packaging in place by
> > end
> > > >> Feb.
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >> Thanks,
> > > >> > > > > > > > > > >> Thomas
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> > Kolhatkar
> > > <
> > > >> > > > > > > > > > >> chinmay@datatorrent.com>
> > > >> > > > > > > > > > >> wrote:
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >> > Hi All,
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > We're planning a work on adding Apache Apex
> as
> > a
> > > >> > > component
> > > >> > > > > to
> > > >> > > > > > > > Apache
> > > >> > > > > > > > > > >> > Bigtop.
> > > >> > > > > > > > > > >> > Bigtop is the packaging system for the Apache
> > big
> > > >> data
> > > >> > > > > > > ecosystem.
> > > >> > > > > > > > > > >> Several
> > > >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE
> for
> > > the
> > > >> > same:
> > > >> > > > > > > > > > >> >
> > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > Proposed plan of execution is as follows:
> > > >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> > expectation
> > > >> for
> > > >> > new
> > > >> > > > > > > > components
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > >> get
> > > >> > > > > > > > > > >> > integrated into Bigtop.
> > > >> > > > > > > > > > >> > Here is the list of it:
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > Most of them seems to be standard ASF based
> > > >> > > requirements,
> > > >> > > > > but
> > > >> > > > > > > few
> > > >> > > > > > > > > need
> > > >> > > > > > > > > > >> to
> > > >> > > > > > > > > > >> > be checked for:
> > > >> > > > > > > > > > >> > 1. Software projects are expected to be
> > Licensed
> > > >> under
> > > >> > > > > Apache
> > > >> > > > > > > > > License,
> > > >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
> > expected
> > > >> to be
> > > >> > > > > > > compatible
> > > >> > > > > > > > > with
> > > >> > > > > > > > > > >> this
> > > >> > > > > > > > > > >> > license)
> > > >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check
> > if
> > > >> > > > > dependencies
> > > >> > > > > > of
> > > >> > > > > > > > > Apex
> > > >> > > > > > > > > > >> are
> > > >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would
> be
> > a
> > > >> > > > > verification
> > > >> > > > > > > > check.
> > > >> > > > > > > > > > >> > 2. Software projects are expected to be
> > > compatible
> > > >> > with
> > > >> > > > all
> > > >> > > > > of
> > > >> > > > > > > the
> > > >> > > > > > > > > > >> > supported platforms that Bigtop distribution
> is
> > > >> > > targeting
> > > >> > > > > > > > > > >> >     - This needs verifying whether our
> software
> > > >> runs
> > > >> > > fine
> > > >> > > > in
> > > >> > > > > > > > > centos-6
> > > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > > >> > opensuse-13.2.
> > > >> > > > > > > > > > >> > 3. What smoke tests that should be added for
> > > >> > deployment.
> > > >> > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> > > beyond
> > > >> > > smoke
> > > >> > > > > test
> > > >> > > > > > > > > > >> >     - These are basically the integration
> tests
> > > for
> > > >> > > > > > verification
> > > >> > > > > > > > > after
> > > >> > > > > > > > > > >> the
> > > >> > > > > > > > > > >> > deployment. This is a soft requirement, but
> aim
> > > is
> > > >> to
> > > >> > > > > achieve
> > > >> > > > > > > this
> > > >> > > > > > > > > as
> > > >> > > > > > > > > > >> well
> > > >> > > > > > > > > > >> > or at least have explanation why not to
> > include.
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > If there are any from the link which
> explicitly
> > > >> needs
> > > >> > to
> > > >> > > > be
> > > >> > > > > > > > checked
> > > >> > > > > > > > > > >> other
> > > >> > > > > > > > > > >> > than above 4, please let us know.
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it
> was
> > > >> learnt
> > > >> > > that
> > > >> > > > > the
> > > >> > > > > > > > > bigtop
> > > >> > > > > > > > > > >> > community want to see the addition of new
> > > >> components
> > > >> > in
> > > >> > > > > > phases.
> > > >> > > > > > > > Here
> > > >> > > > > > > > > > are
> > > >> > > > > > > > > > >> > the phases:
> > > >> > > > > > > > > > >> > 1. Packaging
> > > >> > > > > > > > > > >> >     - This needs creating of package i.e.
> rpm &
> > > deb
> > > >> > > files.
> > > >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> > DISCLAMER,
> > > >> > NOTES
> > > >> > > > etc
> > > >> > > > > if
> > > >> > > > > > > any
> > > >> > > > > > > > > > >> needed.
> > > >> > > > > > > > > > >> >     - Any documentation that need to be added
> > to
> > > >> > > > > distribution
> > > >> > > > > > of
> > > >> > > > > > > > our
> > > >> > > > > > > > > > >> > software.
> > > >> > > > > > > > > > >> >     - Any license information of dependencies
> > > >> required
> > > >> > > to
> > > >> > > > be
> > > >> > > > > > > added
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > >> > package
> > > >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> > > >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment
> and
> > > >> > > > configuration
> > > >> > > > > > > > > > >> >     - Add puppet recipes for automatic
> > deployment
> > > >> > > > > > > > > > >> > 4. Integration tests
> > > >> > > > > > > > > > >> >     - For verification of deployments.
> > > >> > > > > > > > > > >> > 5. license clearance:
> > > >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new
> stuff
> > > is
> > > >> > > > compliant
> > > >> > > > > > > with
> > > >> > > > > > > > > ASF
> > > >> > > > > > > > > > >> > license requirements. If you add code
> licenses
> > > >> under
> > > >> > > > > different
> > > >> > > > > > > > > > licenses,
> > > >> > > > > > > > > > >> > those would need to be listed in the NOTICE.
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > Please share your thoughts on the approach.
> > > >> > > > > > > > > > >> > We'll start corresponding communication on
> > bigtop
> > > >> > > mailing
> > > >> > > > > list
> > > >> > > > > > > as
> > > >> > > > > > > > > > well.
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > We have some specific questions/suggestions
> > > >> related to
> > > >> > > > what
> > > >> > > > > > > should
> > > >> > > > > > > > > be
> > > >> > > > > > > > > > >> the
> > > >> > > > > > > > > > >> > content of the package and what should be the
> > > smoke
> > > >> > > tests,
> > > >> > > > > but
> > > >> > > > > > > in
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > >> > interest of not having too much content here,
> > > we'll
> > > >> > put
> > > >> > > > the
> > > >> > > > > > > > > questions
> > > >> > > > > > > > > > >> as a
> > > >> > > > > > > > > > >> > separate mail in this mailthread.
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > Thanks,
> > > >> > > > > > > > > > >> > Chinmay.
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Hi Thomas,

There are only 2 things that are required from dt-env.sh:
1. DT_HADOOP variable which is exported by dt-env.sh This variable has
absolute path for hadoop binary
2. A find_hadoop() method which find the path of hadoop binary.

JAVA_HOME is not required for this.

The idea is not to change the dtcli. Current dtcli (in apex
3.3.0-incubating src code), sources dt-env.sh for above 2 information.
Hence included dt-env.sh

If the community agrees, we can make dtcli more generic so that in future
packing of bigtop, dt-env.sh can be completely excluded OR at least change
the name to apex-env.sh.

Thanks,
Chinmay.



On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise <th...@datatorrent.com>
wrote:

> Chinmay,
>
> Can you please specify what settings in dt-env.sh are required in a bigtop
> environment?
>
> Location of hadoop and JAVA_HOME should be provided by the stack already?
>
> Thanks,
> Thomas
>
>
> On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Here is a suggestion I have related to name change:
> >
> > 1. While packaging we change the name of "dtcli" script to "apex". This
> is
> > done so that in future when we change the name to "apex", users of bigtop
> > apex don't have to transition much.
> > 2. We also keep a symlink named dtcli which points to apex script. This
> is
> > for backward compatibility.
> > 3. We still keep conf/dt-env.sh for first integration in bigtop. This is
> in
> > the interest of not changing the content of dtcli or apex script.
> > 4. In apex 3.4.0, we take care of all these naming related changes and
> > update bigtop repository later to remove references to dtcli all
> together.
> >
> > Please share your thoughts on above approach.
> >
> > Also, please share what could be the man page content for "apex".
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Yes.. It'll be build from source tar downloaded from one of the apache
> > > mirror: apache.osuosl.org
> > >
> > > The reason for dt-env.sh is it's sourced from dtcli. This leaves us 3
> > > options:
> > > If we should not have any dt-env.sh, dtcli would need a change while
> > > packaging.
> > >
> > > But if we want to use dtcli as it is, then we would need dt-env.sh
> > >
> > >
> > >
> > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <th...@datatorrent.com>
> > > wrote:
> > >
> > >> Looks good overall, though there shouldn't be any dt-env.sh
> > >>
> > >> What will the bigtop package be built from, the source tar?
> > >>
> > >>
> > >>
> > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> chinmay@apache.org>
> > >> wrote:
> > >>
> > >> > Hi Thomas,
> > >> >
> > >> > Thanks for the feedback.
> > >> >
> > >> > First, we're not changing the name anywhere. We'll follow what
> > >> currently is
> > >> > in source tarball for 3.3.0-incubating version.
> > >> >
> > >> > Secondly, I've mentioned a directory structure below which is inline
> > >> with
> > >> > other existing integrations with bigtop.
> > >> >
> > >> > The need for each file is as follows:
> > >> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of
> > >> source
> > >> > code apex 3.3.0-incubating.
> > >> >
> > >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for
> > hadoop
> > >> > binary path and exports an env variable for dtcli to use it.
> > >> >
> > >> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar
> way
> > to
> > >> > how DT Community Edition finds dependency jars.
> > >> > Please note that this is the first iteration list of jars. I'm
> trying
> > to
> > >> > narrow this down to only those which are really required.
> > >> > The test that I'm running to check if dtcli runs fine with given
> > >> dependency
> > >> > is to launch+shutdown+kill+status for pi demo.
> > >> >
> > >> > Above is the directory structure which is required for CLI to work.
> > >> > What aniruddha and pradeep has mentioned are some additional files
> > which
> > >> > makes the like of administrator easier. For eg. /etc/skel
> > >> >
> > >> > Please let me know if this is inline with what you're thinking.
> > >> >
> > >> > Thanks,
> > >> > Chinmay.
> > >> >
> > >> >
> > >> > .
> > >> > |-- bin
> > >> > |   `-- dtcli
> > >> > |-- conf
> > >> > |   `-- dt-env.sh
> > >> > `-- lib
> > >> >     |-- ant-1.9.2.jar
> > >> >     |-- ant-launcher-1.9.2.jar
> > >> >     |-- apex-api-3.3.0-incubating.jar
> > >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > >> >     |-- apex-common-3.3.0-incubating.jar
> > >> >     |-- apex-engine.jar
> > >> >     |-- apex-engine-tests.jar
> > >> >     |-- async-http-client-1.7.20.jar
> > >> >     |-- bval-core-0.5.jar
> > >> >     |-- bval-jsr303-0.5.jar
> > >> >     |-- commons-beanutils-1.8.3.jar
> > >> >     |-- commons-codec-1.10.jar
> > >> >     |-- commons-lang3-3.1.jar
> > >> >     |-- commons-logging-1.1.3.jar
> > >> >     |-- grizzly-http-servlet-2.1.2.jar
> > >> >     |-- hadoop-common-2.2.0-tests.jar
> > >> >     |-- httpclient-4.3.5.jar
> > >> >     |-- httpcore-4.3.2.jar
> > >> >     |-- jackson-core-asl-1.9.2.jar
> > >> >     |-- jackson-mapper-asl-1.9.2.jar
> > >> >     |-- javax.servlet-3.1.jar
> > >> >     |-- javax.servlet-api-3.0.1.jar
> > >> >     |-- jersey-apache-client4-1.9.jar
> > >> >     |-- jersey-client-1.9.jar
> > >> >     |-- jetty-http-8.1.10.v20130312.jar
> > >> >     |-- jetty-io-8.1.10.v20130312.jar
> > >> >     |-- jetty-util-8.1.10.v20130312.jar
> > >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > >> >     |-- jline-2.11.jar
> > >> >     |-- kryo-2.24.0.jar
> > >> >     |-- mbassador-1.1.9.jar
> > >> >     |-- minlog-1.2.jar
> > >> >     |-- netlet-1.2.0.jar
> > >> >     |-- netty-3.6.6.Final.jar
> > >> >     |-- objenesis-2.1.jar
> > >> >     |-- validation-api-1.1.0.Final.jar
> > >> >     |-- xbean-asm5-shaded-4.3.jar
> > >> >     `-- zip4j-1.3.2.jar
> > >> >
> > >> > 3 directories, 40 files
> > >> >
> > >> >
> > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> thomas@datatorrent.com
> > >
> > >> > wrote:
> > >> >
> > >> > > Chinmay,
> > >> > >
> > >> > > Before discussing where to put the files, let's make sure they are
> > >> really
> > >> > > needed for the operation of the CLI. As for names, anything new
> > needs
> > >> to
> > >> > > reflect Apex in the name and should follow common conventions,
> > >> especially
> > >> > > in Bigtop where there are many existing integrations to look at.
> > >> > >
> > >> > > Thanks,
> > >> > > Thomas
> > >> > >
> > >> > >
> > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > >> > > chinmay@datatorrent.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Really good points Pradeep and Aniruddha.
> > >> > > >
> > >> > > > 1. I believe we won't need to change the dtcli considering it
> > works
> > >> > with
> > >> > > DT
> > >> > > > Community edition. We can keep the directory structure similar
> to
> > >> that.
> > >> > > > dt-env.sh has variables which contains information required for
> > >> dtcli
> > >> > to
> > >> > > > launch.
> > >> > > >
> > >> > > > 2. Let me check with bigtop community that whether they
> facilitate
> > >> the
> > >> > > > installation of rpms/debs before user is created. In either
> case,
> > >> > current
> > >> > > > dtcli creates a .dt folder in home directory. Also before
> putting
> > >> > > anything
> > >> > > > in /etc/skel we need to define what are the default contents
> that
> > >> > should
> > >> > > go
> > >> > > > to ~/.dt/ folder. If there is no defaults, probably we should
> not
> > >> > > > explicitly add it in /etc/skel.
> > >> > > >
> > >> > > > 3.  /etc/profile.d approach looks nice. This way contents of
> > >> dt-env.sh
> > >> > > are
> > >> > > > present as env variables. But I see a catch there. Adding
> > dt-env.sh
> > >> to
> > >> > > > /etc/profile.d would make all the variables available at runtime
> > all
> > >> > the
> > >> > > > time. I feel a little skeptical about that. Maybe a possible
> > >> collision
> > >> > > can
> > >> > > > occur with other application vars.
> > >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > >> > > >
> > >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > >> > > Jira/repository.
> > >> > > > Packaging related code usually goes there. (That's what all the
> > >> > > components
> > >> > > > in bigtop does).
> > >> > > > Having said that, once a PR is created, I'll be sharing the link
> > of
> > >> PR
> > >> > > > here, so that apex community as well can review it.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Chinmay.
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > >> > > > aniruddha@datatorrent.com> wrote:
> > >> > > >
> > >> > > > > +1 on suggestions and approach.
> > >> > > > > We may need to iron out details about exact paths etc.
> > >> > > > > Which can be done on jira / PR comments.
> > >> > > > > Is that right @dev?
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > >> > apache@pradeepdalvi.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Inline comments...
> > >> > > > > >
> > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > >> > > > > > chinmay@datatorrent.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > @Thomas, Not all the jar present in DT community edition
> > will
> > >> be
> > >> > > > > included
> > >> > > > > > > there.
> > >> > > > > > >
> > >> > > > > > > Here is the list of jars I found from common dependencies
> > >> between
> > >> > > of
> > >> > > > > apex
> > >> > > > > > > and DT Community edition:
> > >> > > > > > > netlet-1.2.0.jar
> > >> > > > > > > kryo-2.24.0.jar
> > >> > > > > > > jackson-core-asl-1.9.2.jar
> > >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > >> > > > > > > async-http-client-1.7.20.jar
> > >> > > > > > > netty-3.6.6.Final.jar
> > >> > > > > > > validation-api-1.1.0.Final.jar
> > >> > > > > > > bval-jsr303-0.5.jar
> > >> > > > > > > bval-core-0.5.jar
> > >> > > > > > > commons-lang3-3.1.jar
> > >> > > > > > > commons-beanutils-1.8.3.jar
> > >> > > > > > > httpclient-4.3.5.jar
> > >> > > > > > > commons-codec-1.10.jar
> > >> > > > > > > zip4j-1.3.2.jar
> > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > >> > > > > > > xbean-asm5-shaded-4.3.jar
> > >> > > > > > > jersey-apache-client4-1.9.jar
> > >> > > > > > > jline-2.11.jar
> > >> > > > > > > ant-1.9.2.jar
> > >> > > > > > > ant-launcher-1.9.2.jar
> > >> > > > > > > mbassador-1.1.9.jar
> > >> > > > > > > jackson-jaxrs-1.9.2.jar
> > >> > > > > > > jackson-xc-1.9.2.jar
> > >> > > > > > > hadoop-common-2.2.0-tests.jar
> > >> > > > > > >
> > >> > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> > works
> > >> > > > properly
> > >> > > > > > for
> > >> > > > > > > launch+shutdown+kill of apps with only these libraries
> > >> present in
> > >> > > > > > isolation
> > >> > > > > > > without dependency on local m2.
> > >> > > > > > > I'm believe that there are unwanted jars which are used
> for
> > >> > compile
> > >> > > > > time
> > >> > > > > > > dependency and not runtime in above list which I can drop
> to
> > >> keep
> > >> > > > > package
> > >> > > > > > > size minimal.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > @Anniruddha,
> > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> > >> required.
> > >> > > > Only
> > >> > > > > > > required values I see in it are classpath. and dtcli is
> > >> capable
> > >> > of
> > >> > > > > > building
> > >> > > > > > > it on the fly.
> > >> > > > > > >
> > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can
> find
> > >> the
> > >> > jar
> > >> > > > > path,
> > >> > > > > > > dt-env.sh and all such conf files. If we need a change in
> > >> dtcli,
> > >> > > then
> > >> > > > > > > community's opinion is required for whether dtcli should
> > >> change
> > >> > in
> > >> > > > our
> > >> > > > > > repo
> > >> > > > > > > or a copy of that with required changes exist in bigtop
> repo
> > >> > until
> > >> > > we
> > >> > > > > > make
> > >> > > > > > > the dtcli generic enough.
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > > Can we have these file paths set using environment variables
> > >> with
> > >> > > some
> > >> > > > > > default values in dtcli?
> > >> > > > > > Then we can set such params in dt-env.sh.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > 3. dt-sited.xml -> How is this file different from
> > >> dt-site.xml?
> > >> > Not
> > >> > > > > sure
> > >> > > > > > if
> > >> > > > > > > adding in /etc/skel would help. Files & Dirs from
> /etc/skel
> > >> are
> > >> > > > copied
> > >> > > > > to
> > >> > > > > > > home of new user when useradd program is called. But for
> > >> existing
> > >> > > > users
> > >> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/
> on
> > >> the
> > >> > fly
> > >> > > > if
> > >> > > > > > not
> > >> > > > > > > encountered for the first time. Again correct me if I'm
> > wrong.
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > > Yes, you are right. However this would probably be necessary
> > >> step
> > >> > for
> > >> > > > > > rpm/deb, as they may also get installed during OS install
> and
> > >> > before
> > >> > > > user
> > >> > > > > > accounts were created.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
> > best
> > >> > > idea.
> > >> > > > > > We'll
> > >> > > > > > > anyway put dtcli in location which is by default present
> in
> > >> path
> > >> > > i.e.
> > >> > > > > > > /usr/bin.
> > >> > > > > >
> > >> > > > > > For other env variables specific to apex, I'll prefer to use
> > >> > > dt-env.sh
> > >> > > > > and
> > >> > > > > > > source it in dtcli rather than changing bashrc etc...
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> > >> cleaner
> > >> > way
> > >> > > > to
> > >> > > > > > achieve the same.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > > Chinmay.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > >> > > > > > > aniruddha@datatorrent.com> wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi,
> > >> > > > > > > >
> > >> > > > > > > > @Chinmay,
> > >> > > > > > > >
> > >> > > > > > > > We need to consider following:
> > >> > > > > > > > System wide default config files can be located at
> > following
> > >> > > > > locations:
> > >> > > > > > > >
> > >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> > behaviour
> > >> on
> > >> > > how
> > >> > > > it
> > >> > > > > > > finds
> > >> > > > > > > > those locations)
> > >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> > >> behaviour on
> > >> > > how
> > >> > > > > it
> > >> > > > > > > > finds those locations)
> > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new
> users
> > >> that
> > >> > > will
> > >> > > > > be
> > >> > > > > > > > created in system in future.)
> > >> > > > > > > >
> > >> > > > > > > > We may also have to modify bashrc / bashprofile for
> > >> population
> > >> > > any
> > >> > > > > > > > variables that are required.
> > >> > > > > > > >
> > >> > > > > > > > @dev, please put in your comments / suggestion.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Aniruddha
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > >> > > > > thomas@datatorrent.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > -->
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > >> > > > > > > > chinmay@datatorrent.com
> > >> > > > > > > > > >
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hello Everyone!!
> > >> > > > > > > > > >
> > >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
> > here
> > >> > are
> > >> > > > some
> > >> > > > > > > > > proposals
> > >> > > > > > > > > > about package structure etc..
> > >> > > > > > > > > > Before posting it on bbigtop mailing list, I have
> some
> > >> > > question
> > >> > > > > for
> > >> > > > > > > > apex
> > >> > > > > > > > > > community.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Proposed Directory structure of apex package for
> both
> > >> deb &
> > >> > > > rpm:
> > >> > > > > > > > > >
> > >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > >> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > >> > > > > > > > >
> > >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > >> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > <version> = 3.3.0-incubating.
> > >> > > > > > > > > > <other dependent jars> = All the 3rd party jars
> which
> > >> are
> > >> > > > > required
> > >> > > > > > > for
> > >> > > > > > > > > apex
> > >> > > > > > > > > > to run. Usually the dependencies are packaged as
> part
> > of
> > >> > > > rpm/deb
> > >> > > > > by
> > >> > > > > > > any
> > >> > > > > > > > > > software in bigtop.
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > Can you specify what those jars are in Bigtop context?
> > >> Same
> > >> > as
> > >> > > > > > shipped
> > >> > > > > > > > with
> > >> > > > > > > > > DT community addition under lib/ ?
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
> > >> party
> > >> > > jars
> > >> > > > > > which
> > >> > > > > > > > > needs
> > >> > > > > > > > > > to included while packaging.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Questions related to this:
> > >> > > > > > > > > > 1. Should we call the cli of apex as "apex" instead
> of
> > >> > > "dtcli"
> > >> > > > in
> > >> > > > > > > > bigtop
> > >> > > > > > > > > > package?
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > I think we should keep the name until we are able to
> > >> change
> > >> > it
> > >> > > in
> > >> > > > > > Apex.
> > >> > > > > > > > > Otherwise this may get confusing.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > > 2. I see that all softwares in bigtop have man page
> > for
> > >> > their
> > >> > > > > > > > > executables.
> > >> > > > > > > > > > I think we should have it too for dtcli. Is there
> any
> > >> > > > > documentation
> > >> > > > > > > > > which I
> > >> > > > > > > > > > can convert to man page? or can I use output of
> "dtcli
> > >> > > --help"?
> > >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> > >> 3.3.0 OR
> > >> > > > > > > > > > 3.3.0-incubating?
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > Has to be -incubating
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
> > >> 2.7.1
> > >> > > > > version
> > >> > > > > > of
> > >> > > > > > > > > > bigtop hadoop? Any problems that we see with this
> > >> > dependency?
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > Should work without changed. I thought we certified
> > >> against
> > >> > > > 2.7.0?
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > > 5. Following is the apache mirror from which bigtop
> > will
> > >> > pick
> > >> > > > the
> > >> > > > > > > apex
> > >> > > > > > > > > > source for compilation and packaging. Please correct
> > if
> > >> > > > > incorrect:
> > >> > > > > > > > > >
> > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
> all
> > >> the
> > >> > > > > > softwares
> > >> > > > > > > in
> > >> > > > > > > > > > bigtop.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > Please share your opinion.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Chinmay.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > >> > > > > > > > > > chinmay@datatorrent.com>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I've already started discussion on bigtop dev
> > mailing
> > >> > list
> > >> > > > for
> > >> > > > > > > > > > > integration. Also created a Jira for the same. For
> > >> this
> > >> > > > > > communities
> > >> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > >> > > > > > > thomas@datatorrent.com
> > >> > > > > > > > >
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >> Chinmay,
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> I don't see anything under prerequisites that
> would
> > >> be a
> > >> > > > > > problem.
> > >> > > > > > > We
> > >> > > > > > > > > > >> looked
> > >> > > > > > > > > > >> at the ASF licencing compatibility as part of
> > >> becoming
> > >> > an
> > >> > > > > > > incubator
> > >> > > > > > > > > > >> project.
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> Please focus on the packaging during the next
> > weeks.
> > >> > Since
> > >> > > > the
> > >> > > > > > > work
> > >> > > > > > > > > will
> > >> > > > > > > > > > >> be
> > >> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> > should
> > >> > also
> > >> > > be
> > >> > > > > > > there.
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> Would be good to have the packaging in place by
> end
> > >> Feb.
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> Thanks,
> > >> > > > > > > > > > >> Thomas
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> Kolhatkar
> > <
> > >> > > > > > > > > > >> chinmay@datatorrent.com>
> > >> > > > > > > > > > >> wrote:
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> > Hi All,
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > We're planning a work on adding Apache Apex as
> a
> > >> > > component
> > >> > > > > to
> > >> > > > > > > > Apache
> > >> > > > > > > > > > >> > Bigtop.
> > >> > > > > > > > > > >> > Bigtop is the packaging system for the Apache
> big
> > >> data
> > >> > > > > > > ecosystem.
> > >> > > > > > > > > > >> Several
> > >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for
> > the
> > >> > same:
> > >> > > > > > > > > > >> >
> > https://issues.apache.org/jira/browse/APEXCORE-331
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Proposed plan of execution is as follows:
> > >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> expectation
> > >> for
> > >> > new
> > >> > > > > > > > components
> > >> > > > > > > > > to
> > >> > > > > > > > > > >> get
> > >> > > > > > > > > > >> > integrated into Bigtop.
> > >> > > > > > > > > > >> > Here is the list of it:
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Most of them seems to be standard ASF based
> > >> > > requirements,
> > >> > > > > but
> > >> > > > > > > few
> > >> > > > > > > > > need
> > >> > > > > > > > > > >> to
> > >> > > > > > > > > > >> > be checked for:
> > >> > > > > > > > > > >> > 1. Software projects are expected to be
> Licensed
> > >> under
> > >> > > > > Apache
> > >> > > > > > > > > License,
> > >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
> expected
> > >> to be
> > >> > > > > > > compatible
> > >> > > > > > > > > with
> > >> > > > > > > > > > >> this
> > >> > > > > > > > > > >> > license)
> > >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check
> if
> > >> > > > > dependencies
> > >> > > > > > of
> > >> > > > > > > > > Apex
> > >> > > > > > > > > > >> are
> > >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be
> a
> > >> > > > > verification
> > >> > > > > > > > check.
> > >> > > > > > > > > > >> > 2. Software projects are expected to be
> > compatible
> > >> > with
> > >> > > > all
> > >> > > > > of
> > >> > > > > > > the
> > >> > > > > > > > > > >> > supported platforms that Bigtop distribution is
> > >> > > targeting
> > >> > > > > > > > > > >> >     - This needs verifying whether our software
> > >> runs
> > >> > > fine
> > >> > > > in
> > >> > > > > > > > > centos-6
> > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > >> > opensuse-13.2.
> > >> > > > > > > > > > >> > 3. What smoke tests that should be added for
> > >> > deployment.
> > >> > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> > beyond
> > >> > > smoke
> > >> > > > > test
> > >> > > > > > > > > > >> >     - These are basically the integration tests
> > for
> > >> > > > > > verification
> > >> > > > > > > > > after
> > >> > > > > > > > > > >> the
> > >> > > > > > > > > > >> > deployment. This is a soft requirement, but aim
> > is
> > >> to
> > >> > > > > achieve
> > >> > > > > > > this
> > >> > > > > > > > > as
> > >> > > > > > > > > > >> well
> > >> > > > > > > > > > >> > or at least have explanation why not to
> include.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > If there are any from the link which explicitly
> > >> needs
> > >> > to
> > >> > > > be
> > >> > > > > > > > checked
> > >> > > > > > > > > > >> other
> > >> > > > > > > > > > >> > than above 4, please let us know.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
> > >> learnt
> > >> > > that
> > >> > > > > the
> > >> > > > > > > > > bigtop
> > >> > > > > > > > > > >> > community want to see the addition of new
> > >> components
> > >> > in
> > >> > > > > > phases.
> > >> > > > > > > > Here
> > >> > > > > > > > > > are
> > >> > > > > > > > > > >> > the phases:
> > >> > > > > > > > > > >> > 1. Packaging
> > >> > > > > > > > > > >> >     - This needs creating of package i.e. rpm &
> > deb
> > >> > > files.
> > >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> DISCLAMER,
> > >> > NOTES
> > >> > > > etc
> > >> > > > > if
> > >> > > > > > > any
> > >> > > > > > > > > > >> needed.
> > >> > > > > > > > > > >> >     - Any documentation that need to be added
> to
> > >> > > > > distribution
> > >> > > > > > of
> > >> > > > > > > > our
> > >> > > > > > > > > > >> > software.
> > >> > > > > > > > > > >> >     - Any license information of dependencies
> > >> required
> > >> > > to
> > >> > > > be
> > >> > > > > > > added
> > >> > > > > > > > > to
> > >> > > > > > > > > > >> > package
> > >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> > >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > >> > > > configuration
> > >> > > > > > > > > > >> >     - Add puppet recipes for automatic
> deployment
> > >> > > > > > > > > > >> > 4. Integration tests
> > >> > > > > > > > > > >> >     - For verification of deployments.
> > >> > > > > > > > > > >> > 5. license clearance:
> > >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff
> > is
> > >> > > > compliant
> > >> > > > > > > with
> > >> > > > > > > > > ASF
> > >> > > > > > > > > > >> > license requirements. If you add code licenses
> > >> under
> > >> > > > > different
> > >> > > > > > > > > > licenses,
> > >> > > > > > > > > > >> > those would need to be listed in the NOTICE.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Please share your thoughts on the approach.
> > >> > > > > > > > > > >> > We'll start corresponding communication on
> bigtop
> > >> > > mailing
> > >> > > > > list
> > >> > > > > > > as
> > >> > > > > > > > > > well.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > We have some specific questions/suggestions
> > >> related to
> > >> > > > what
> > >> > > > > > > should
> > >> > > > > > > > > be
> > >> > > > > > > > > > >> the
> > >> > > > > > > > > > >> > content of the package and what should be the
> > smoke
> > >> > > tests,
> > >> > > > > but
> > >> > > > > > > in
> > >> > > > > > > > > the
> > >> > > > > > > > > > >> > interest of not having too much content here,
> > we'll
> > >> > put
> > >> > > > the
> > >> > > > > > > > > questions
> > >> > > > > > > > > > >> as a
> > >> > > > > > > > > > >> > separate mail in this mailthread.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Thanks,
> > >> > > > > > > > > > >> > Chinmay.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

Chinmay,

Can you please specify what settings in dt-env.sh are required in a bigtop
environment?

Location of hadoop and JAVA_HOME should be provided by the stack already?

Thanks,
Thomas


On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Here is a suggestion I have related to name change:
>
> 1. While packaging we change the name of "dtcli" script to "apex". This is
> done so that in future when we change the name to "apex", users of bigtop
> apex don't have to transition much.
> 2. We also keep a symlink named dtcli which points to apex script. This is
> for backward compatibility.
> 3. We still keep conf/dt-env.sh for first integration in bigtop. This is in
> the interest of not changing the content of dtcli or apex script.
> 4. In apex 3.4.0, we take care of all these naming related changes and
> update bigtop repository later to remove references to dtcli all together.
>
> Please share your thoughts on above approach.
>
> Also, please share what could be the man page content for "apex".
>
> Thanks,
> Chinmay.
>
>
>
> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Yes.. It'll be build from source tar downloaded from one of the apache
> > mirror: apache.osuosl.org
> >
> > The reason for dt-env.sh is it's sourced from dtcli. This leaves us 3
> > options:
> > If we should not have any dt-env.sh, dtcli would need a change while
> > packaging.
> >
> > But if we want to use dtcli as it is, then we would need dt-env.sh
> >
> >
> >
> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> >> Looks good overall, though there shouldn't be any dt-env.sh
> >>
> >> What will the bigtop package be built from, the source tar?
> >>
> >>
> >>
> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <ch...@apache.org>
> >> wrote:
> >>
> >> > Hi Thomas,
> >> >
> >> > Thanks for the feedback.
> >> >
> >> > First, we're not changing the name anywhere. We'll follow what
> >> currently is
> >> > in source tarball for 3.3.0-incubating version.
> >> >
> >> > Secondly, I've mentioned a directory structure below which is inline
> >> with
> >> > other existing integrations with bigtop.
> >> >
> >> > The need for each file is as follows:
> >> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of
> >> source
> >> > code apex 3.3.0-incubating.
> >> >
> >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for
> hadoop
> >> > binary path and exports an env variable for dtcli to use it.
> >> >
> >> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar way
> to
> >> > how DT Community Edition finds dependency jars.
> >> > Please note that this is the first iteration list of jars. I'm trying
> to
> >> > narrow this down to only those which are really required.
> >> > The test that I'm running to check if dtcli runs fine with given
> >> dependency
> >> > is to launch+shutdown+kill+status for pi demo.
> >> >
> >> > Above is the directory structure which is required for CLI to work.
> >> > What aniruddha and pradeep has mentioned are some additional files
> which
> >> > makes the like of administrator easier. For eg. /etc/skel
> >> >
> >> > Please let me know if this is inline with what you're thinking.
> >> >
> >> > Thanks,
> >> > Chinmay.
> >> >
> >> >
> >> > .
> >> > |-- bin
> >> > |   `-- dtcli
> >> > |-- conf
> >> > |   `-- dt-env.sh
> >> > `-- lib
> >> >     |-- ant-1.9.2.jar
> >> >     |-- ant-launcher-1.9.2.jar
> >> >     |-- apex-api-3.3.0-incubating.jar
> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> >> >     |-- apex-common-3.3.0-incubating.jar
> >> >     |-- apex-engine.jar
> >> >     |-- apex-engine-tests.jar
> >> >     |-- async-http-client-1.7.20.jar
> >> >     |-- bval-core-0.5.jar
> >> >     |-- bval-jsr303-0.5.jar
> >> >     |-- commons-beanutils-1.8.3.jar
> >> >     |-- commons-codec-1.10.jar
> >> >     |-- commons-lang3-3.1.jar
> >> >     |-- commons-logging-1.1.3.jar
> >> >     |-- grizzly-http-servlet-2.1.2.jar
> >> >     |-- hadoop-common-2.2.0-tests.jar
> >> >     |-- httpclient-4.3.5.jar
> >> >     |-- httpcore-4.3.2.jar
> >> >     |-- jackson-core-asl-1.9.2.jar
> >> >     |-- jackson-mapper-asl-1.9.2.jar
> >> >     |-- javax.servlet-3.1.jar
> >> >     |-- javax.servlet-api-3.0.1.jar
> >> >     |-- jersey-apache-client4-1.9.jar
> >> >     |-- jersey-client-1.9.jar
> >> >     |-- jetty-http-8.1.10.v20130312.jar
> >> >     |-- jetty-io-8.1.10.v20130312.jar
> >> >     |-- jetty-util-8.1.10.v20130312.jar
> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> >> >     |-- jline-2.11.jar
> >> >     |-- kryo-2.24.0.jar
> >> >     |-- mbassador-1.1.9.jar
> >> >     |-- minlog-1.2.jar
> >> >     |-- netlet-1.2.0.jar
> >> >     |-- netty-3.6.6.Final.jar
> >> >     |-- objenesis-2.1.jar
> >> >     |-- validation-api-1.1.0.Final.jar
> >> >     |-- xbean-asm5-shaded-4.3.jar
> >> >     `-- zip4j-1.3.2.jar
> >> >
> >> > 3 directories, 40 files
> >> >
> >> >
> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <thomas@datatorrent.com
> >
> >> > wrote:
> >> >
> >> > > Chinmay,
> >> > >
> >> > > Before discussing where to put the files, let's make sure they are
> >> really
> >> > > needed for the operation of the CLI. As for names, anything new
> needs
> >> to
> >> > > reflect Apex in the name and should follow common conventions,
> >> especially
> >> > > in Bigtop where there are many existing integrations to look at.
> >> > >
> >> > > Thanks,
> >> > > Thomas
> >> > >
> >> > >
> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> >> > > chinmay@datatorrent.com>
> >> > > wrote:
> >> > >
> >> > > > Really good points Pradeep and Aniruddha.
> >> > > >
> >> > > > 1. I believe we won't need to change the dtcli considering it
> works
> >> > with
> >> > > DT
> >> > > > Community edition. We can keep the directory structure similar to
> >> that.
> >> > > > dt-env.sh has variables which contains information required for
> >> dtcli
> >> > to
> >> > > > launch.
> >> > > >
> >> > > > 2. Let me check with bigtop community that whether they facilitate
> >> the
> >> > > > installation of rpms/debs before user is created. In either case,
> >> > current
> >> > > > dtcli creates a .dt folder in home directory. Also before putting
> >> > > anything
> >> > > > in /etc/skel we need to define what are the default contents that
> >> > should
> >> > > go
> >> > > > to ~/.dt/ folder. If there is no defaults, probably we should not
> >> > > > explicitly add it in /etc/skel.
> >> > > >
> >> > > > 3.  /etc/profile.d approach looks nice. This way contents of
> >> dt-env.sh
> >> > > are
> >> > > > present as env variables. But I see a catch there. Adding
> dt-env.sh
> >> to
> >> > > > /etc/profile.d would make all the variables available at runtime
> all
> >> > the
> >> > > > time. I feel a little skeptical about that. Maybe a possible
> >> collision
> >> > > can
> >> > > > occur with other application vars.
> >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> >> > > >
> >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> >> > > Jira/repository.
> >> > > > Packaging related code usually goes there. (That's what all the
> >> > > components
> >> > > > in bigtop does).
> >> > > > Having said that, once a PR is created, I'll be sharing the link
> of
> >> PR
> >> > > > here, so that apex community as well can review it.
> >> > > >
> >> > > > Thanks,
> >> > > > Chinmay.
> >> > > >
> >> > > >
> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> >> > > > aniruddha@datatorrent.com> wrote:
> >> > > >
> >> > > > > +1 on suggestions and approach.
> >> > > > > We may need to iron out details about exact paths etc.
> >> > > > > Which can be done on jira / PR comments.
> >> > > > > Is that right @dev?
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> >> > apache@pradeepdalvi.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Inline comments...
> >> > > > > >
> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> >> > > > > > chinmay@datatorrent.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > @Thomas, Not all the jar present in DT community edition
> will
> >> be
> >> > > > > included
> >> > > > > > > there.
> >> > > > > > >
> >> > > > > > > Here is the list of jars I found from common dependencies
> >> between
> >> > > of
> >> > > > > apex
> >> > > > > > > and DT Community edition:
> >> > > > > > > netlet-1.2.0.jar
> >> > > > > > > kryo-2.24.0.jar
> >> > > > > > > jackson-core-asl-1.9.2.jar
> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> >> > > > > > > async-http-client-1.7.20.jar
> >> > > > > > > netty-3.6.6.Final.jar
> >> > > > > > > validation-api-1.1.0.Final.jar
> >> > > > > > > bval-jsr303-0.5.jar
> >> > > > > > > bval-core-0.5.jar
> >> > > > > > > commons-lang3-3.1.jar
> >> > > > > > > commons-beanutils-1.8.3.jar
> >> > > > > > > httpclient-4.3.5.jar
> >> > > > > > > commons-codec-1.10.jar
> >> > > > > > > zip4j-1.3.2.jar
> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> >> > > > > > > xbean-asm5-shaded-4.3.jar
> >> > > > > > > jersey-apache-client4-1.9.jar
> >> > > > > > > jline-2.11.jar
> >> > > > > > > ant-1.9.2.jar
> >> > > > > > > ant-launcher-1.9.2.jar
> >> > > > > > > mbassador-1.1.9.jar
> >> > > > > > > jackson-jaxrs-1.9.2.jar
> >> > > > > > > jackson-xc-1.9.2.jar
> >> > > > > > > hadoop-common-2.2.0-tests.jar
> >> > > > > > >
> >> > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> works
> >> > > > properly
> >> > > > > > for
> >> > > > > > > launch+shutdown+kill of apps with only these libraries
> >> present in
> >> > > > > > isolation
> >> > > > > > > without dependency on local m2.
> >> > > > > > > I'm believe that there are unwanted jars which are used for
> >> > compile
> >> > > > > time
> >> > > > > > > dependency and not runtime in above list which I can drop to
> >> keep
> >> > > > > package
> >> > > > > > > size minimal.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > @Anniruddha,
> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> >> required.
> >> > > > Only
> >> > > > > > > required values I see in it are classpath. and dtcli is
> >> capable
> >> > of
> >> > > > > > building
> >> > > > > > > it on the fly.
> >> > > > > > >
> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find
> >> the
> >> > jar
> >> > > > > path,
> >> > > > > > > dt-env.sh and all such conf files. If we need a change in
> >> dtcli,
> >> > > then
> >> > > > > > > community's opinion is required for whether dtcli should
> >> change
> >> > in
> >> > > > our
> >> > > > > > repo
> >> > > > > > > or a copy of that with required changes exist in bigtop repo
> >> > until
> >> > > we
> >> > > > > > make
> >> > > > > > > the dtcli generic enough.
> >> > > > > > >
> >> > > > > >
> >> > > > > > Can we have these file paths set using environment variables
> >> with
> >> > > some
> >> > > > > > default values in dtcli?
> >> > > > > > Then we can set such params in dt-env.sh.
> >> > > > > >
> >> > > > > >
> >> > > > > > > 3. dt-sited.xml -> How is this file different from
> >> dt-site.xml?
> >> > Not
> >> > > > > sure
> >> > > > > > if
> >> > > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel
> >> are
> >> > > > copied
> >> > > > > to
> >> > > > > > > home of new user when useradd program is called. But for
> >> existing
> >> > > > users
> >> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on
> >> the
> >> > fly
> >> > > > if
> >> > > > > > not
> >> > > > > > > encountered for the first time. Again correct me if I'm
> wrong.
> >> > > > > > >
> >> > > > > >
> >> > > > > > Yes, you are right. However this would probably be necessary
> >> step
> >> > for
> >> > > > > > rpm/deb, as they may also get installed during OS install and
> >> > before
> >> > > > user
> >> > > > > > accounts were created.
> >> > > > > >
> >> > > > > >
> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
> best
> >> > > idea.
> >> > > > > > We'll
> >> > > > > > > anyway put dtcli in location which is by default present in
> >> path
> >> > > i.e.
> >> > > > > > > /usr/bin.
> >> > > > > >
> >> > > > > > For other env variables specific to apex, I'll prefer to use
> >> > > dt-env.sh
> >> > > > > and
> >> > > > > > > source it in dtcli rather than changing bashrc etc...
> >> > > > > >
> >> > > > > >
> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> >> cleaner
> >> > way
> >> > > > to
> >> > > > > > achieve the same.
> >> > > > > >
> >> > > > > >
> >> > > > > > >
> >> > > > > > Thanks,
> >> > > > > > > Chinmay.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> >> > > > > > > aniruddha@datatorrent.com> wrote:
> >> > > > > > >
> >> > > > > > > > Hi,
> >> > > > > > > >
> >> > > > > > > > @Chinmay,
> >> > > > > > > >
> >> > > > > > > > We need to consider following:
> >> > > > > > > > System wide default config files can be located at
> following
> >> > > > > locations:
> >> > > > > > > >
> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> behaviour
> >> on
> >> > > how
> >> > > > it
> >> > > > > > > finds
> >> > > > > > > > those locations)
> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> >> behaviour on
> >> > > how
> >> > > > > it
> >> > > > > > > > finds those locations)
> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users
> >> that
> >> > > will
> >> > > > > be
> >> > > > > > > > created in system in future.)
> >> > > > > > > >
> >> > > > > > > > We may also have to modify bashrc / bashprofile for
> >> population
> >> > > any
> >> > > > > > > > variables that are required.
> >> > > > > > > >
> >> > > > > > > > @dev, please put in your comments / suggestion.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Aniruddha
> >> > > > > > > >
> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> >> > > > > thomas@datatorrent.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > -->
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> >> > > > > > > > chinmay@datatorrent.com
> >> > > > > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hello Everyone!!
> >> > > > > > > > > >
> >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
> here
> >> > are
> >> > > > some
> >> > > > > > > > > proposals
> >> > > > > > > > > > about package structure etc..
> >> > > > > > > > > > Before posting it on bbigtop mailing list, I have some
> >> > > question
> >> > > > > for
> >> > > > > > > > apex
> >> > > > > > > > > > community.
> >> > > > > > > > > >
> >> > > > > > > > > > Proposed Directory structure of apex package for both
> >> deb &
> >> > > > rpm:
> >> > > > > > > > > >
> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> >> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> >> > > > > > > > >
> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> >> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > <version> = 3.3.0-incubating.
> >> > > > > > > > > > <other dependent jars> = All the 3rd party jars which
> >> are
> >> > > > > required
> >> > > > > > > for
> >> > > > > > > > > apex
> >> > > > > > > > > > to run. Usually the dependencies are packaged as part
> of
> >> > > > rpm/deb
> >> > > > > by
> >> > > > > > > any
> >> > > > > > > > > > software in bigtop.
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Can you specify what those jars are in Bigtop context?
> >> Same
> >> > as
> >> > > > > > shipped
> >> > > > > > > > with
> >> > > > > > > > > DT community addition under lib/ ?
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
> >> party
> >> > > jars
> >> > > > > > which
> >> > > > > > > > > needs
> >> > > > > > > > > > to included while packaging.
> >> > > > > > > > > >
> >> > > > > > > > > > Questions related to this:
> >> > > > > > > > > > 1. Should we call the cli of apex as "apex" instead of
> >> > > "dtcli"
> >> > > > in
> >> > > > > > > > bigtop
> >> > > > > > > > > > package?
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > I think we should keep the name until we are able to
> >> change
> >> > it
> >> > > in
> >> > > > > > Apex.
> >> > > > > > > > > Otherwise this may get confusing.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > > 2. I see that all softwares in bigtop have man page
> for
> >> > their
> >> > > > > > > > > executables.
> >> > > > > > > > > > I think we should have it too for dtcli. Is there any
> >> > > > > documentation
> >> > > > > > > > > which I
> >> > > > > > > > > > can convert to man page? or can I use output of "dtcli
> >> > > --help"?
> >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> >> 3.3.0 OR
> >> > > > > > > > > > 3.3.0-incubating?
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Has to be -incubating
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
> >> 2.7.1
> >> > > > > version
> >> > > > > > of
> >> > > > > > > > > > bigtop hadoop? Any problems that we see with this
> >> > dependency?
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Should work without changed. I thought we certified
> >> against
> >> > > > 2.7.0?
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > > 5. Following is the apache mirror from which bigtop
> will
> >> > pick
> >> > > > the
> >> > > > > > > apex
> >> > > > > > > > > > source for compilation and packaging. Please correct
> if
> >> > > > > incorrect:
> >> > > > > > > > > >
> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by all
> >> the
> >> > > > > > softwares
> >> > > > > > > in
> >> > > > > > > > > > bigtop.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Please share your opinion.
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Chinmay.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> >> > > > > > > > > > chinmay@datatorrent.com>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> >> > > > > > > > > > >
> >> > > > > > > > > > > I've already started discussion on bigtop dev
> mailing
> >> > list
> >> > > > for
> >> > > > > > > > > > > integration. Also created a Jira for the same. For
> >> this
> >> > > > > > communities
> >> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> >> > > > > > > thomas@datatorrent.com
> >> > > > > > > > >
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > >> Chinmay,
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> I don't see anything under prerequisites that would
> >> be a
> >> > > > > > problem.
> >> > > > > > > We
> >> > > > > > > > > > >> looked
> >> > > > > > > > > > >> at the ASF licencing compatibility as part of
> >> becoming
> >> > an
> >> > > > > > > incubator
> >> > > > > > > > > > >> project.
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Please focus on the packaging during the next
> weeks.
> >> > Since
> >> > > > the
> >> > > > > > > work
> >> > > > > > > > > will
> >> > > > > > > > > > >> be
> >> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> should
> >> > also
> >> > > be
> >> > > > > > > there.
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Would be good to have the packaging in place by end
> >> Feb.
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Thanks,
> >> > > > > > > > > > >> Thomas
> >> > > > > > > > > > >>
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar
> <
> >> > > > > > > > > > >> chinmay@datatorrent.com>
> >> > > > > > > > > > >> wrote:
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> > Hi All,
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > We're planning a work on adding Apache Apex as a
> >> > > component
> >> > > > > to
> >> > > > > > > > Apache
> >> > > > > > > > > > >> > Bigtop.
> >> > > > > > > > > > >> > Bigtop is the packaging system for the Apache big
> >> data
> >> > > > > > > ecosystem.
> >> > > > > > > > > > >> Several
> >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for
> the
> >> > same:
> >> > > > > > > > > > >> >
> https://issues.apache.org/jira/browse/APEXCORE-331
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Proposed plan of execution is as follows:
> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> >> > > > > > > > > > >> > Apache bigtop has some hard and soft expectation
> >> for
> >> > new
> >> > > > > > > > components
> >> > > > > > > > > to
> >> > > > > > > > > > >> get
> >> > > > > > > > > > >> > integrated into Bigtop.
> >> > > > > > > > > > >> > Here is the list of it:
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Most of them seems to be standard ASF based
> >> > > requirements,
> >> > > > > but
> >> > > > > > > few
> >> > > > > > > > > need
> >> > > > > > > > > > >> to
> >> > > > > > > > > > >> > be checked for:
> >> > > > > > > > > > >> > 1. Software projects are expected to be Licensed
> >> under
> >> > > > > Apache
> >> > > > > > > > > License,
> >> > > > > > > > > > >> > Version 2.0 (and their dependencies are expected
> >> to be
> >> > > > > > > compatible
> >> > > > > > > > > with
> >> > > > > > > > > > >> this
> >> > > > > > > > > > >> > license)
> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> >> > > > > dependencies
> >> > > > > > of
> >> > > > > > > > > Apex
> >> > > > > > > > > > >> are
> >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> >> > > > > verification
> >> > > > > > > > check.
> >> > > > > > > > > > >> > 2. Software projects are expected to be
> compatible
> >> > with
> >> > > > all
> >> > > > > of
> >> > > > > > > the
> >> > > > > > > > > > >> > supported platforms that Bigtop distribution is
> >> > > targeting
> >> > > > > > > > > > >> >     - This needs verifying whether our software
> >> runs
> >> > > fine
> >> > > > in
> >> > > > > > > > > centos-6
> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> >> > opensuse-13.2.
> >> > > > > > > > > > >> > 3. What smoke tests that should be added for
> >> > deployment.
> >> > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> beyond
> >> > > smoke
> >> > > > > test
> >> > > > > > > > > > >> >     - These are basically the integration tests
> for
> >> > > > > > verification
> >> > > > > > > > > after
> >> > > > > > > > > > >> the
> >> > > > > > > > > > >> > deployment. This is a soft requirement, but aim
> is
> >> to
> >> > > > > achieve
> >> > > > > > > this
> >> > > > > > > > > as
> >> > > > > > > > > > >> well
> >> > > > > > > > > > >> > or at least have explanation why not to include.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > If there are any from the link which explicitly
> >> needs
> >> > to
> >> > > > be
> >> > > > > > > > checked
> >> > > > > > > > > > >> other
> >> > > > > > > > > > >> > than above 4, please let us know.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
> >> learnt
> >> > > that
> >> > > > > the
> >> > > > > > > > > bigtop
> >> > > > > > > > > > >> > community want to see the addition of new
> >> components
> >> > in
> >> > > > > > phases.
> >> > > > > > > > Here
> >> > > > > > > > > > are
> >> > > > > > > > > > >> > the phases:
> >> > > > > > > > > > >> > 1. Packaging
> >> > > > > > > > > > >> >     - This needs creating of package i.e. rpm &
> deb
> >> > > files.
> >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER,
> >> > NOTES
> >> > > > etc
> >> > > > > if
> >> > > > > > > any
> >> > > > > > > > > > >> needed.
> >> > > > > > > > > > >> >     - Any documentation that need to be added to
> >> > > > > distribution
> >> > > > > > of
> >> > > > > > > > our
> >> > > > > > > > > > >> > software.
> >> > > > > > > > > > >> >     - Any license information of dependencies
> >> required
> >> > > to
> >> > > > be
> >> > > > > > > added
> >> > > > > > > > > to
> >> > > > > > > > > > >> > package
> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> >> > > > configuration
> >> > > > > > > > > > >> >     - Add puppet recipes for automatic deployment
> >> > > > > > > > > > >> > 4. Integration tests
> >> > > > > > > > > > >> >     - For verification of deployments.
> >> > > > > > > > > > >> > 5. license clearance:
> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff
> is
> >> > > > compliant
> >> > > > > > > with
> >> > > > > > > > > ASF
> >> > > > > > > > > > >> > license requirements. If you add code licenses
> >> under
> >> > > > > different
> >> > > > > > > > > > licenses,
> >> > > > > > > > > > >> > those would need to be listed in the NOTICE.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Please share your thoughts on the approach.
> >> > > > > > > > > > >> > We'll start corresponding communication on bigtop
> >> > > mailing
> >> > > > > list
> >> > > > > > > as
> >> > > > > > > > > > well.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > We have some specific questions/suggestions
> >> related to
> >> > > > what
> >> > > > > > > should
> >> > > > > > > > > be
> >> > > > > > > > > > >> the
> >> > > > > > > > > > >> > content of the package and what should be the
> smoke
> >> > > tests,
> >> > > > > but
> >> > > > > > > in
> >> > > > > > > > > the
> >> > > > > > > > > > >> > interest of not having too much content here,
> we'll
> >> > put
> >> > > > the
> >> > > > > > > > > questions
> >> > > > > > > > > > >> as a
> >> > > > > > > > > > >> > separate mail in this mailthread.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Thanks,
> >> > > > > > > > > > >> > Chinmay.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@datatorrent.com>]

Yes. Made that change and also those suggested by pradeep.

Thanks,
Chinmay.
On 18 Feb 2016 7:15 p.m., "Munagala Ramanath" <ra...@datatorrent.com> wrote:

> Tushar is right: apex (formerly dtcli) talks directly to Yarn and does not
> depend on dtGateway or the REST API.
>
> Ram
>
> On Thu, Feb 18, 2016 at 12:07 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > In following statement, can you replace dtGateway with ApplicationMaster
> > and Yarn. or may be remove this statement entirely.
> > apex is a wrapper around the REST API provided by dtGatway,
> >
> > - Tushar.
> >
> >
> >
> >
> > On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > I've attached man page for apex in Jira. Here is the link to it:
> > > https://issues.apache.org/jira/secure/attachment/12788403/apex.1
> > >
> > > Please view it by: man -l apex.1
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <
> tushar@datatorrent.com>
> > > wrote:
> > >
> > > > Hi Chinmay,
> > > >
> > > > I am not able to find the attachment, may be attachments are not
> > allowed
> > > in
> > > > the mailing list. Can you attach the man page to the Jira?
> > > >
> > > > - Tushar.
> > > >
> > > >
> > > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <
> chinmay@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I've attached man page file for the purpose of bigtop integration.
> > > > > Please have a look and let me know if there any any feedback about
> > it.
> > > > >
> > > > > You can see the man page content of this file with:
> > > > > man -l apex.1
> > > > >
> > > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> > > chinmay@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Thanks. Will follow bigtop recommendation. i.e. remove
> > "-incubating".
> > > > >>
> > > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
> > thomas@datatorrent.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >>> Please follow the recommendation and continue discussion on the
> > > Bigtop
> > > > >>> list.
> > > > >>>
> > > > >>> --
> > > > >>> sent from mobile
> > > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <chinmay@apache.org
> >
> > > > wrote:
> > > > >>>
> > > > >>> > I got a recommendation from bigtop community to drop
> > "-incubating"
> > > > from
> > > > >>> > version.
> > > > >>> >
> > > > >>> > Here is the mail thread:
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> > > > >>> >
> > > > >>> > Please let me know if removing "-incubating" is fine.
> > > > >>> >
> > > > >>> > Thanks,
> > > > >>> > Chinmay.
> > > > >>> >
> > > > >>> >
> > > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> > > > chinmay@apache.org
> > > > >>> >
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> > > I've already started communication there.
> > > > >>> > >
> > > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> > > > >>> thomas@datatorrent.com>
> > > > >>> > > wrote:
> > > > >>> > >
> > > > >>> > >> This is a question you should ask on the Bigtop mailing
> list.
> > > > >>> > >>
> > > > >>> > >> --
> > > > >>> > >> sent from mobile
> > > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <
> prad@apache.org>
> > > > >>> wrote:
> > > > >>> > >>
> > > > >>> > >> > Yes. In case of RPM, the only restriction on version is
> that
> > > it
> > > > >>> can
> > > > >>> > not
> > > > >>> > >> > contain '-'.
> > > > >>> > >> > However, as a workaround, one can use 'incubating' as a
> part
> > > of
> > > > >>> > >> %{release}.
> > > > >>> > >> >
> > > > >>> > >> > Version: 3.3.0
> > > > >>> > >> > Release: incubating
> > > > >>> > >> >
> > > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> > > > >>> > >> >
> > > > >>> > >> > It is the same practice followed for pre-release packages
> > e.g.
> > > > >>> with
> > > > >>> > rcX,
> > > > >>> > >> > alphaX, betaX being used in %{release}.
> > > > >>> > >> >
> > > > >>> > >> > Regards,
> > > > >>> > >> > Pradeep A. Dalvi
> > > > >>> > >> >
> > > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> > > > >>> > chinmay@apache.org>
> > > > >>> > >> > wrote:
> > > > >>> > >> >
> > > > >>> > >> > > Hi Everyone,
> > > > >>> > >> > >
> > > > >>> > >> > > In one of the previous mails in this thread, we decided
> > that
> > > > >>> version
> > > > >>> > >> of
> > > > >>> > >> > the
> > > > >>> > >> > > package will be "3.3.0-incubating".
> > > > >>> > >> > > The problem that I'm facing is in RPM base version "-"
> is
> > > not
> > > > >>> > allowed.
> > > > >>> > >> > >
> > > > >>> > >> > > Can we have package version to "3.3.0"?
> > > > >>> > >> > > i.e. the packages created will be as follows:
> > > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> > > > >>> > >> > >
> > > > >>> > >> > > There is another integration for Zepplin in Bigtop which
> > is
> > > > >>> > incubating
> > > > >>> > >> > and
> > > > >>> > >> > > is having base version as "0.5.6".
> > > > >>> > >> > >
> > > > >>> > >> > > Please let me know your thoughts.
> > > > >>> > >> > >
> > > > >>> > >> > > Thanks,
> > > > >>> > >> > > Chinmay.
> > > > >>> > >> > >
> > > > >>> > >> > >
> > > > >>> > >> > >
> > > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> > > > >>> > >> chinmay@apache.org>
> > > > >>> > >> > > wrote:
> > > > >>> > >> > >
> > > > >>> > >> > > > Thanks everyone for the feedback.
> > > > >>> > >> > > >
> > > > >>> > >> > > > Also as Ram and David mentioned, I'll start creating
> the
> > > man
> > > > >>> page
> > > > >>> > as
> > > > >>> > >> > > well.
> > > > >>> > >> > > >
> > > > >>> > >> > > > Thanks,
> > > > >>> > >> > > > Chinmay.
> > > > >>> > >> > > >
> > > > >>> > >> > > >
> > > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> > > > >>> > >> thomas@datatorrent.com>
> > > > >>> > >> > > > wrote:
> > > > >>> > >> > > >
> > > > >>> > >> > > >> I think we already agreed that we will retain the
> > current
> > > > >>> script
> > > > >>> > >> name
> > > > >>> > >> > in
> > > > >>> > >> > > >> this bigtop iteration. There is a ticket for
> renaming,
> > > add
> > > > >>> you
> > > > >>> > >> > > suggestions
> > > > >>> > >> > > >> there.
> > > > >>> > >> > > >>
> > > > >>> > >> > > >> --
> > > > >>> > >> > > >> sent from mobile
> > > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > > > >>> > sandesh@datatorrent.com>
> > > > >>> > >> > > wrote:
> > > > >>> > >> > > >>
> > > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start the
> > > transition
> > > > >>> > >> process by
> > > > >>> > >> > > >> making
> > > > >>> > >> > > >> > dtcli as the symlink to apex cli.
> > > > >>> > >> > > >> >
> > > > >>> > >> > > >> > Renaming is a breaking change should be done during
> > > major
> > > > >>> > version
> > > > >>> > >> > > number
> > > > >>> > >> > > >> > change.
> > > > >>> > >> > > >> >
> > > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> > > > >>> > david@datatorrent.com
> > > > >>> > >> >
> > > > >>> > >> > > >> wrote:
> > > > >>> > >> > > >> >
> > > > >>> > >> > > >> > > Apologies for hitting the "Send" button too
> early.
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > What is the scope of the name change?  If we
> simply
> > > > have
> > > > >>> a
> > > > >>> > >> symlink
> > > > >>> > >> > > >> from
> > > > >>> > >> > > >> > > apex to dtcli, then it should have no backward
> > > > >>> compatibility
> > > > >>> > >> > problem
> > > > >>> > >> > > >> at
> > > > >>> > >> > > >> > > all.
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > But the proper name change would involve:
> > > > >>> > >> > > >> > > - Change the name of DTCli class to ApexCli
> > > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> > > > >>> > >> > > >> > > - Change the initial greeting when running the
> CLI
> > > from
> > > > >>> > >> something
> > > > >>> > >> > > like
> > > > >>> > >> > > >> > "DT
> > > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > The above changes may break existing scripts that
> > > rely
> > > > >>> on the
> > > > >>> > >> the
> > > > >>> > >> > > >> string
> > > > >>> > >> > > >> > > "DT".
> > > > >>> > >> > > >> > > But if folks are okay with this potential
> problem,
> > I
> > > am
> > > > >>> okay
> > > > >>> > >> with
> > > > >>> > >> > > the
> > > > >>> > >> > > >> > name
> > > > >>> > >> > > >> > > change.
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > For the man page, we can start with the content
> of
> > > this
> > > > >>> page:
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > Plus the command line options when doing dtcli
> -h:
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > usage: DTCli
> > > > >>> > >> > > >> > >  -e <arg>    Commands are read from the argument
> > > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at all time
> > > > >>> > >> > > >> > >  -h          Print this help
> > > > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos principal
> > > > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> > > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> > > > >>> > >> > > >> > >  -r          JSON Raw mode
> > > > >>> > >> > > >> > >  -v          Verbose mode level 1
> > > > >>> > >> > > >> > >  -vv         Verbose mode level 2
> > > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> > > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > David
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> > > > >>> > >> david@datatorrent.com
> > > > >>> > >> > >
> > > > >>> > >> > > >> > wrote:
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > > What is the scope of the name change?  If we
> > simply
> > > > >>> have a
> > > > >>> > >> > symlink
> > > > >>> > >> > > >> from
> > > > >>> > >> > > >> > > > apex to dtcli, then it should have no backward
> > > > >>> > compatibility
> > > > >>> > >> > > >> problem at
> > > > >>> > >> > > >> > > all.
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > > But the proper name change would involve:
> > > > >>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
> > > > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> > > > >>> > >> > > >> > > > - Change the initial greeting when running the
> > CLI
> > > > from
> > > > >>> > >> > something
> > > > >>> > >> > > >> like
> > > > >>> > >> > > >> > > "DT
> > > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > > The above changes may break existing scripts
> that
> > > > rely
> > > > >>> on
> > > > >>> > the
> > > > >>> > >> > the
> > > > >>> > >> > > >> > string
> > > > >>> > >> > > >> > > > "DT".
> > > > >>> > >> > > >> > > > But if folks are okay with this potential
> > problem,
> > > I
> > > > am
> > > > >>> > okay
> > > > >>> > >> > with
> > > > >>> > >> > > >> the
> > > > >>> > >> > > >> > > name
> > > > >>> > >> > > >> > > > change.
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > > For the man page, we can start with the content
> > of
> > > > this
> > > > >>> > page:
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay
> > Kolhatkar
> > > <
> > > > >>> > >> > > >> chinmay@apache.org
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > > wrote:
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >> Here is a suggestion I have related to name
> > > change:
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> 1. While packaging we change the name of
> "dtcli"
> > > > >>> script to
> > > > >>> > >> > > "apex".
> > > > >>> > >> > > >> > This
> > > > >>> > >> > > >> > > is
> > > > >>> > >> > > >> > > >> done so that in future when we change the name
> > to
> > > > >>> "apex",
> > > > >>> > >> users
> > > > >>> > >> > > of
> > > > >>> > >> > > >> > > bigtop
> > > > >>> > >> > > >> > > >> apex don't have to transition much.
> > > > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli which
> > points
> > > > to
> > > > >>> apex
> > > > >>> > >> > > script.
> > > > >>> > >> > > >> > This
> > > > >>> > >> > > >> > > is
> > > > >>> > >> > > >> > > >> for backward compatibility.
> > > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first
> > > > integration
> > > > >>> in
> > > > >>> > >> > bigtop.
> > > > >>> > >> > > >> This
> > > > >>> > >> > > >> > is
> > > > >>> > >> > > >> > > >> in
> > > > >>> > >> > > >> > > >> the interest of not changing the content of
> > dtcli
> > > or
> > > > >>> apex
> > > > >>> > >> > script.
> > > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these
> > naming
> > > > >>> related
> > > > >>> > >> > > changes
> > > > >>> > >> > > >> and
> > > > >>> > >> > > >> > > >> update bigtop repository later to remove
> > > references
> > > > to
> > > > >>> > dtcli
> > > > >>> > >> > all
> > > > >>> > >> > > >> > > together.
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> Please share your thoughts on above approach.
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> Also, please share what could be the man page
> > > > content
> > > > >>> for
> > > > >>> > >> > "apex".
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> Thanks,
> > > > >>> > >> > > >> > > >> Chinmay.
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay
> > > Kolhatkar <
> > > > >>> > >> > > >> > chinmay@apache.org>
> > > > >>> > >> > > >> > > >> wrote:
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> > Yes.. It'll be build from source tar
> > downloaded
> > > > >>> from one
> > > > >>> > >> of
> > > > >>> > >> > the
> > > > >>> > >> > > >> > apache
> > > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced
> from
> > > > dtcli.
> > > > >>> > This
> > > > >>> > >> > > leaves
> > > > >>> > >> > > >> us
> > > > >>> > >> > > >> > 3
> > > > >>> > >> > > >> > > >> > options:
> > > > >>> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli
> > would
> > > > >>> need a
> > > > >>> > >> > change
> > > > >>> > >> > > >> while
> > > > >>> > >> > > >> > > >> > packaging.
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> > But if we want to use dtcli as it is, then
> we
> > > > would
> > > > >>> need
> > > > >>> > >> > > >> dt-env.sh
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas
> Weise
> > <
> > > > >>> > >> > > >> > thomas@datatorrent.com
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >> > wrote:
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> >> Looks good overall, though there shouldn't
> be
> > > any
> > > > >>> > >> dt-env.sh
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >> What will the bigtop package be built from,
> > the
> > > > >>> source
> > > > >>> > >> tar?
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay
> > > > Kolhatkar
> > > > >>> <
> > > > >>> > >> > > >> > > chinmay@apache.org
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> >> wrote:
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >> > Hi Thomas,
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > First, we're not changing the name
> > anywhere.
> > > > >>> We'll
> > > > >>> > >> follow
> > > > >>> > >> > > what
> > > > >>> > >> > > >> > > >> >> currently is
> > > > >>> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating
> > > version.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory
> > > structure
> > > > >>> below
> > > > >>> > >> which
> > > > >>> > >> > > is
> > > > >>> > >> > > >> > > inline
> > > > >>> > >> > > >> > > >> >> with
> > > > >>> > >> > > >> > > >> >> > other existing integrations with bigtop.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > The need for each file is as follows:
> > > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> > > > >>> > >> > > >> engine/src/main/scripts/
> > > > >>> > >> > > >> > > of
> > > > >>> > >> > > >> > > >> >> source
> > > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from
> > > dtcli.
> > > > >>> This
> > > > >>> > >> > > searches
> > > > >>> > >> > > >> for
> > > > >>> > >> > > >> > > >> hadoop
> > > > >>> > >> > > >> > > >> >> > binary path and exports an env variable
> for
> > > > >>> dtcli to
> > > > >>> > >> use
> > > > >>> > >> > it.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list
> is
> > > > >>> extracted
> > > > >>> > >> in a
> > > > >>> > >> > > >> similar
> > > > >>> > >> > > >> > > >> way to
> > > > >>> > >> > > >> > > >> >> > how DT Community Edition finds dependency
> > > jars.
> > > > >>> > >> > > >> > > >> >> > Please note that this is the first
> > iteration
> > > > >>> list of
> > > > >>> > >> jars.
> > > > >>> > >> > > I'm
> > > > >>> > >> > > >> > > >> trying to
> > > > >>> > >> > > >> > > >> >> > narrow this down to only those which are
> > > really
> > > > >>> > >> required.
> > > > >>> > >> > > >> > > >> >> > The test that I'm running to check if
> dtcli
> > > > runs
> > > > >>> fine
> > > > >>> > >> with
> > > > >>> > >> > > >> given
> > > > >>> > >> > > >> > > >> >> dependency
> > > > >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi
> > > demo.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > Above is the directory structure which is
> > > > >>> required
> > > > >>> > for
> > > > >>> > >> CLI
> > > > >>> > >> > > to
> > > > >>> > >> > > >> > work.
> > > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned
> > are
> > > > some
> > > > >>> > >> > additional
> > > > >>> > >> > > >> > files
> > > > >>> > >> > > >> > > >> which
> > > > >>> > >> > > >> > > >> >> > makes the like of administrator easier.
> For
> > > eg.
> > > > >>> > >> /etc/skel
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > Please let me know if this is inline with
> > > what
> > > > >>> you're
> > > > >>> > >> > > >> thinking.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > Thanks,
> > > > >>> > >> > > >> > > >> >> > Chinmay.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > .
> > > > >>> > >> > > >> > > >> >> > |-- bin
> > > > >>> > >> > > >> > > >> >> > |   `-- dtcli
> > > > >>> > >> > > >> > > >> >> > |-- conf
> > > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > > > >>> > >> > > >> > > >> >> > `-- lib
> > > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > > > >>> > >> > > >> > > >> >> >     |--
> > > apex-bufferserver-3.3.0-incubating.jar
> > > > >>> > >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > > > >>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > > > >>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > > > >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > > > >>> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > > >>> > >> > > >> > > >> >> >     |--
> > jetty-websocket-8.1.10.v20130312.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > > > >>> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > > > >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas
> > > Weise <
> > > > >>> > >> > > >> > > >> thomas@datatorrent.com>
> > > > >>> > >> > > >> > > >> >> > wrote:
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > > Chinmay,
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> > > Before discussing where to put the
> files,
> > > > let's
> > > > >>> > make
> > > > >>> > >> > sure
> > > > >>> > >> > > >> they
> > > > >>> > >> > > >> > > are
> > > > >>> > >> > > >> > > >> >> really
> > > > >>> > >> > > >> > > >> >> > > needed for the operation of the CLI. As
> > for
> > > > >>> names,
> > > > >>> > >> > > anything
> > > > >>> > >> > > >> new
> > > > >>> > >> > > >> > > >> needs
> > > > >>> > >> > > >> > > >> >> to
> > > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and should
> > follow
> > > > >>> common
> > > > >>> > >> > > >> conventions,
> > > > >>> > >> > > >> > > >> >> especially
> > > > >>> > >> > > >> > > >> >> > > in Bigtop where there are many existing
> > > > >>> > integrations
> > > > >>> > >> to
> > > > >>> > >> > > look
> > > > >>> > >> > > >> > at.
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> > > Thanks,
> > > > >>> > >> > > >> > > >> >> > > Thomas
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM,
> Chinmay
> > > > >>> Kolhatkar
> > > > >>> > <
> > > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > > > >>> > >> > > >> > > >> >> > > wrote:
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep and
> > Aniruddha.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to change
> > the
> > > > >>> dtcli
> > > > >>> > >> > > >> considering it
> > > > >>> > >> > > >> > > >> works
> > > > >>> > >> > > >> > > >> >> > with
> > > > >>> > >> > > >> > > >> >> > > DT
> > > > >>> > >> > > >> > > >> >> > > > Community edition. We can keep the
> > > > directory
> > > > >>> > >> structure
> > > > >>> > >> > > >> > similar
> > > > >>> > >> > > >> > > to
> > > > >>> > >> > > >> > > >> >> that.
> > > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which
> contains
> > > > >>> > information
> > > > >>> > >> > > >> required
> > > > >>> > >> > > >> > for
> > > > >>> > >> > > >> > > >> >> dtcli
> > > > >>> > >> > > >> > > >> >> > to
> > > > >>> > >> > > >> > > >> >> > > > launch.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop community
> > > that
> > > > >>> > whether
> > > > >>> > >> > they
> > > > >>> > >> > > >> > > >> facilitate
> > > > >>> > >> > > >> > > >> >> the
> > > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs before user
> > is
> > > > >>> created.
> > > > >>> > >> In
> > > > >>> > >> > > >> either
> > > > >>> > >> > > >> > > case,
> > > > >>> > >> > > >> > > >> >> > current
> > > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home
> > > > directory.
> > > > >>> > Also
> > > > >>> > >> > > before
> > > > >>> > >> > > >> > > putting
> > > > >>> > >> > > >> > > >> >> > > anything
> > > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define what
> are
> > > the
> > > > >>> > default
> > > > >>> > >> > > >> contents
> > > > >>> > >> > > >> > > that
> > > > >>> > >> > > >> > > >> >> > should
> > > > >>> > >> > > >> > > >> >> > > go
> > > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no
> > > defaults,
> > > > >>> > >> probably we
> > > > >>> > >> > > >> should
> > > > >>> > >> > > >> > > not
> > > > >>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks
> nice.
> > > > This
> > > > >>> way
> > > > >>> > >> > > contents
> > > > >>> > >> > > >> of
> > > > >>> > >> > > >> > > >> >> dt-env.sh
> > > > >>> > >> > > >> > > >> >> > > are
> > > > >>> > >> > > >> > > >> >> > > > present as env variables. But I see a
> > > catch
> > > > >>> > there.
> > > > >>> > >> > > Adding
> > > > >>> > >> > > >> > > >> dt-env.sh
> > > > >>> > >> > > >> > > >> >> to
> > > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the
> > > variables
> > > > >>> > >> available
> > > > >>> > >> > at
> > > > >>> > >> > > >> > > runtime
> > > > >>> > >> > > >> > > >> all
> > > > >>> > >> > > >> > > >> >> > the
> > > > >>> > >> > > >> > > >> >> > > > time. I feel a little skeptical about
> > > that.
> > > > >>> > Maybe a
> > > > >>> > >> > > >> possible
> > > > >>> > >> > > >> > > >> >> collision
> > > > >>> > >> > > >> > > >> >> > > can
> > > > >>> > >> > > >> > > >> >> > > > occur with other application vars.
> > > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
> > > > >>> > >> > "../conf/dt-env.sh"
> > > > >>> > >> > > >> and
> > > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables
> > are
> > > > >>> anyway
> > > > >>> > >> > > >> available.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is
> > > > >>> happening
> > > > >>> > at
> > > > >>> > >> > > Bigtop
> > > > >>> > >> > > >> > > >> >> > > Jira/repository.
> > > > >>> > >> > > >> > > >> >> > > > Packaging related code usually goes
> > > there.
> > > > >>> > (That's
> > > > >>> > >> > what
> > > > >>> > >> > > >> all
> > > > >>> > >> > > >> > the
> > > > >>> > >> > > >> > > >> >> > > components
> > > > >>> > >> > > >> > > >> >> > > > in bigtop does).
> > > > >>> > >> > > >> > > >> >> > > > Having said that, once a PR is
> created,
> > > > I'll
> > > > >>> be
> > > > >>> > >> > sharing
> > > > >>> > >> > > >> the
> > > > >>> > >> > > >> > > link
> > > > >>> > >> > > >> > > >> of
> > > > >>> > >> > > >> > > >> >> PR
> > > > >>> > >> > > >> > > >> >> > > > here, so that apex community as well
> > can
> > > > >>> review
> > > > >>> > it.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > Thanks,
> > > > >>> > >> > > >> > > >> >> > > > Chinmay.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM,
> > > Aniruddha
> > > > >>> > >> Thombare <
> > > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> > > > >>> > >> > > >> > > >> >> > > > > We may need to iron out details
> about
> > > > exact
> > > > >>> > paths
> > > > >>> > >> > etc.
> > > > >>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR
> > > comments.
> > > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep
> > A.
> > > > >>> Dalvi <
> > > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > > > >>> > >> > > >> > > >> >> > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM,
> > > > Chinmay
> > > > >>> > >> > Kolhatkar <
> > > > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > > > >>> > >> > > >> > > >> >> > > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar
> present
> > in
> > > > DT
> > > > >>> > >> community
> > > > >>> > >> > > >> > edition
> > > > >>> > >> > > >> > > >> will
> > > > >>> > >> > > >> > > >> >> be
> > > > >>> > >> > > >> > > >> >> > > > > included
> > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I
> found
> > > from
> > > > >>> > common
> > > > >>> > >> > > >> > dependencies
> > > > >>> > >> > > >> > > >> >> between
> > > > >>> > >> > > >> > > >> >> > > of
> > > > >>> > >> > > >> > > >> >> > > > > apex
> > > > >>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> > > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > jetty-websocket-8.1.10.v20130312.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some
> > > tests
> > > > do
> > > > >>> > check
> > > > >>> > >> > that
> > > > >>> > >> > > >> > dtcli
> > > > >>> > >> > > >> > > >> works
> > > > >>> > >> > > >> > > >> >> > > > properly
> > > > >>> > >> > > >> > > >> >> > > > > > for
> > > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps
> with
> > > > only
> > > > >>> > these
> > > > >>> > >> > > >> libraries
> > > > >>> > >> > > >> > > >> >> present in
> > > > >>> > >> > > >> > > >> >> > > > > > isolation
> > > > >>> > >> > > >> > > >> >> > > > > > > without dependency on local m2.
> > > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there are
> > unwanted
> > > > >>> jars
> > > > >>> > >> which
> > > > >>> > >> > are
> > > > >>> > >> > > >> used
> > > > >>> > >> > > >> > > for
> > > > >>> > >> > > >> > > >> >> > compile
> > > > >>> > >> > > >> > > >> >> > > > > time
> > > > >>> > >> > > >> > > >> >> > > > > > > dependency and not runtime in
> > above
> > > > >>> list
> > > > >>> > >> which I
> > > > >>> > >> > > can
> > > > >>> > >> > > >> > drop
> > > > >>> > >> > > >> > > >> to
> > > > >>> > >> > > >> > > >> >> keep
> > > > >>> > >> > > >> > > >> >> > > > > package
> > > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if
> > all
> > > > of
> > > > >>> the
> > > > >>> > >> > > >> dt-env.sh is
> > > > >>> > >> > > >> > > >> >> required.
> > > > >>> > >> > > >> > > >> >> > > > Only
> > > > >>> > >> > > >> > > >> >> > > > > > > required values I see in it are
> > > > >>> classpath.
> > > > >>> > >> and
> > > > >>> > >> > > >> dtcli is
> > > > >>> > >> > > >> > > >> >> capable
> > > > >>> > >> > > >> > > >> >> > of
> > > > >>> > >> > > >> > > >> >> > > > > > building
> > > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll
> have
> > > to
> > > > >>> see
> > > > >>> > how
> > > > >>> > >> > dtcli
> > > > >>> > >> > > >> can
> > > > >>> > >> > > >> > > find
> > > > >>> > >> > > >> > > >> >> the
> > > > >>> > >> > > >> > > >> >> > jar
> > > > >>> > >> > > >> > > >> >> > > > > path,
> > > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf
> > files.
> > > If
> > > > >>> we
> > > > >>> > >> need a
> > > > >>> > >> > > >> change
> > > > >>> > >> > > >> > in
> > > > >>> > >> > > >> > > >> >> dtcli,
> > > > >>> > >> > > >> > > >> >> > > then
> > > > >>> > >> > > >> > > >> >> > > > > > > community's opinion is required
> > for
> > > > >>> whether
> > > > >>> > >> > dtcli
> > > > >>> > >> > > >> > should
> > > > >>> > >> > > >> > > >> >> change
> > > > >>> > >> > > >> > > >> >> > in
> > > > >>> > >> > > >> > > >> >> > > > our
> > > > >>> > >> > > >> > > >> >> > > > > > repo
> > > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with required
> > > > changes
> > > > >>> > >> exist in
> > > > >>> > >> > > >> bigtop
> > > > >>> > >> > > >> > > >> repo
> > > > >>> > >> > > >> > > >> >> > until
> > > > >>> > >> > > >> > > >> >> > > we
> > > > >>> > >> > > >> > > >> >> > > > > > make
> > > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > Can we have these file paths set
> > > using
> > > > >>> > >> environment
> > > > >>> > >> > > >> > > variables
> > > > >>> > >> > > >> > > >> >> with
> > > > >>> > >> > > >> > > >> >> > > some
> > > > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> > > > >>> > >> > > >> > > >> >> > > > > > Then we can set such params in
> > > > dt-env.sh.
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this
> > file
> > > > >>> > different
> > > > >>> > >> > from
> > > > >>> > >> > > >> > > >> >> dt-site.xml?
> > > > >>> > >> > > >> > > >> >> > Not
> > > > >>> > >> > > >> > > >> >> > > > > sure
> > > > >>> > >> > > >> > > >> >> > > > > > if
> > > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help.
> > > > Files &
> > > > >>> > Dirs
> > > > >>> > >> > from
> > > > >>> > >> > > >> > > /etc/skel
> > > > >>> > >> > > >> > > >> >> are
> > > > >>> > >> > > >> > > >> >> > > > copied
> > > > >>> > >> > > >> > > >> >> > > > > to
> > > > >>> > >> > > >> > > >> >> > > > > > > home of new user when useradd
> > > program
> > > > >>> is
> > > > >>> > >> called.
> > > > >>> > >> > > But
> > > > >>> > >> > > >> > for
> > > > >>> > >> > > >> > > >> >> existing
> > > > >>> > >> > > >> > > >> >> > > > users
> > > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any use.
> > Moreover,
> > > > >>> dtcli
> > > > >>> > >> > creates
> > > > >>> > >> > > >> > ~/.dt/
> > > > >>> > >> > > >> > > on
> > > > >>> > >> > > >> > > >> >> the
> > > > >>> > >> > > >> > > >> >> > fly
> > > > >>> > >> > > >> > > >> >> > > > if
> > > > >>> > >> > > >> > > >> >> > > > > > not
> > > > >>> > >> > > >> > > >> >> > > > > > > encountered for the first time.
> > > Again
> > > > >>> > >> correct me
> > > > >>> > >> > > if
> > > > >>> > >> > > >> I'm
> > > > >>> > >> > > >> > > >> wrong.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right. However this
> > > would
> > > > >>> > >> probably be
> > > > >>> > >> > > >> > > necessary
> > > > >>> > >> > > >> > > >> >> step
> > > > >>> > >> > > >> > > >> >> > for
> > > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get
> > > installed
> > > > >>> > during
> > > > >>> > >> OS
> > > > >>> > >> > > >> install
> > > > >>> > >> > > >> > > and
> > > > >>> > >> > > >> > > >> >> > before
> > > > >>> > >> > > >> > > >> >> > > > user
> > > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc &
> bash_profile
> > > ->
> > > > >>> I'm
> > > > >>> > not
> > > > >>> > >> > sure
> > > > >>> > >> > > >> its
> > > > >>> > >> > > >> > the
> > > > >>> > >> > > >> > > >> best
> > > > >>> > >> > > >> > > >> >> > > idea.
> > > > >>> > >> > > >> > > >> >> > > > > > We'll
> > > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location
> > which
> > > is
> > > > >>> by
> > > > >>> > >> default
> > > > >>> > >> > > >> > present
> > > > >>> > >> > > >> > > in
> > > > >>> > >> > > >> > > >> >> path
> > > > >>> > >> > > >> > > >> >> > > i.e.
> > > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > For other env variables specific
> to
> > > > apex,
> > > > >>> > I'll
> > > > >>> > >> > > prefer
> > > > >>> > >> > > >> to
> > > > >>> > >> > > >> > > use
> > > > >>> > >> > > >> > > >> >> > > dt-env.sh
> > > > >>> > >> > > >> > > >> >> > > > > and
> > > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather than
> > > > changing
> > > > >>> > >> bashrc
> > > > >>> > >> > > >> etc...
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
> > > > >>> /etc/profile.d/
> > > > >>> > >> shall
> > > > >>> > >> > > be
> > > > >>> > >> > > >> the
> > > > >>> > >> > > >> > > >> >> cleaner
> > > > >>> > >> > > >> > > >> >> > way
> > > > >>> > >> > > >> > > >> >> > > > to
> > > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > Thanks,
> > > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07
> PM,
> > > > >>> Aniruddha
> > > > >>> > >> > > Thombare
> > > > >>> > >> > > >> <
> > > > >>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com>
> > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > We need to consider
> following:
> > > > >>> > >> > > >> > > >> >> > > > > > > > System wide default config
> > files
> > > > can
> > > > >>> be
> > > > >>> > >> > located
> > > > >>> > >> > > at
> > > > >>> > >> > > >> > > >> following
> > > > >>> > >> > > >> > > >> >> > > > > locations:
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may
> > have
> > > > to
> > > > >>> > change
> > > > >>> > >> > > dtcli
> > > > >>> > >> > > >> > > >> behaviour
> > > > >>> > >> > > >> > > >> >> on
> > > > >>> > >> > > >> > > >> >> > > how
> > > > >>> > >> > > >> > > >> >> > > > it
> > > > >>> > >> > > >> > > >> >> > > > > > > finds
> > > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may
> > > have
> > > > to
> > > > >>> > >> change
> > > > >>> > >> > > dtcli
> > > > >>> > >> > > >> > > >> >> behaviour on
> > > > >>> > >> > > >> > > >> >> > > how
> > > > >>> > >> > > >> > > >> >> > > > > it
> > > > >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
> > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml
> and
> > > > other
> > > > >>> > files
> > > > >>> > >> > (for
> > > > >>> > >> > > >> new
> > > > >>> > >> > > >> > > users
> > > > >>> > >> > > >> > > >> >> that
> > > > >>> > >> > > >> > > >> >> > > will
> > > > >>> > >> > > >> > > >> >> > > > > be
> > > > >>> > >> > > >> > > >> >> > > > > > > > created in system in future.)
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > We may also have to modify
> > > bashrc /
> > > > >>> > >> > bashprofile
> > > > >>> > >> > > >> for
> > > > >>> > >> > > >> > > >> >> population
> > > > >>> > >> > > >> > > >> >> > > any
> > > > >>> > >> > > >> > > >> >> > > > > > > > variables that are required.
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your
> > > comments /
> > > > >>> > >> > suggestion.
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16
> > AM,
> > > > >>> Thomas
> > > > >>> > >> Weise
> > > > >>> > >> > <
> > > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > -->
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05
> > AM,
> > > > >>> Chinmay
> > > > >>> > >> > > >> Kolhatkar <
> > > > >>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging
> > > > effort
> > > > >>> > >> (rpm+deb)
> > > > >>> > >> > > of
> > > > >>> > >> > > >> > apex,
> > > > >>> > >> > > >> > > >> here
> > > > >>> > >> > > >> > > >> >> > are
> > > > >>> > >> > > >> > > >> >> > > > some
> > > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > about package structure
> > etc..
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on
> > bbigtop
> > > > >>> mailing
> > > > >>> > >> > list, I
> > > > >>> > >> > > >> have
> > > > >>> > >> > > >> > > >> some
> > > > >>> > >> > > >> > > >> >> > > question
> > > > >>> > >> > > >> > > >> >> > > > > for
> > > > >>> > >> > > >> > > >> >> > > > > > > > apex
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory
> > structure
> > > of
> > > > >>> apex
> > > > >>> > >> > package
> > > > >>> > >> > > >> for
> > > > >>> > >> > > >> > > both
> > > > >>> > >> > > >> > > >> >> deb &
> > > > >>> > >> > > >> > > >> >> > > > rpm:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other
> > > > >>> dependent
> > > > >>> > >> jars>
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> > > > >>> > /usr/lib/apex/bin/dtcli
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> /usr/share/doc/man/man1/dtcli.1.gz
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > /usr/share/doc/apex/license/LICENSE.txt
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > /usr/share/doc/apex/CHANGELOG
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> /usr/share/doc/apex/NOTICE
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
> > 3.3.0-incubating.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> =
> > All
> > > > the
> > > > >>> 3rd
> > > > >>> > >> party
> > > > >>> > >> > > >> jars
> > > > >>> > >> > > >> > > which
> > > > >>> > >> > > >> > > >> >> are
> > > > >>> > >> > > >> > > >> >> > > > > required
> > > > >>> > >> > > >> > > >> >> > > > > > > for
> > > > >>> > >> > > >> > > >> >> > > > > > > > > apex
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the
> > > > dependencies
> > > > >>> are
> > > > >>> > >> > > packaged
> > > > >>> > >> > > >> as
> > > > >>> > >> > > >> > > >> part of
> > > > >>> > >> > > >> > > >> >> > > > rpm/deb
> > > > >>> > >> > > >> > > >> >> > > > > by
> > > > >>> > >> > > >> > > >> >> > > > > > > any
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what those
> > jars
> > > > >>> are in
> > > > >>> > >> > Bigtop
> > > > >>> > >> > > >> > > context?
> > > > >>> > >> > > >> > > >> >> Same
> > > > >>> > >> > > >> > > >> >> > as
> > > > >>> > >> > > >> > > >> >> > > > > > shipped
> > > > >>> > >> > > >> > > >> >> > > > > > > > with
> > > > >>> > >> > > >> > > >> >> > > > > > > > > DT community addition under
> > > lib/
> > > > ?
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> =
> > Licenses
> > > of
> > > > >>> > >> > > corresponding
> > > > >>> > >> > > >> of
> > > > >>> > >> > > >> > > 3rd
> > > > >>> > >> > > >> > > >> >> party
> > > > >>> > >> > > >> > > >> >> > > jars
> > > > >>> > >> > > >> > > >> >> > > > > > which
> > > > >>> > >> > > >> > > >> >> > > > > > > > > needs
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > to included while
> > packaging.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions related to
> this:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli
> > of
> > > > >>> apex as
> > > > >>> > >> > "apex"
> > > > >>> > >> > > >> > instead
> > > > >>> > >> > > >> > > >> of
> > > > >>> > >> > > >> > > >> >> > > "dtcli"
> > > > >>> > >> > > >> > > >> >> > > > in
> > > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > I think we should keep the
> > name
> > > > >>> until
> > > > >>> > we
> > > > >>> > >> are
> > > > >>> > >> > > >> able
> > > > >>> > >> > > >> > to
> > > > >>> > >> > > >> > > >> >> change
> > > > >>> > >> > > >> > > >> >> > it
> > > > >>> > >> > > >> > > >> >> > > in
> > > > >>> > >> > > >> > > >> >> > > > > > Apex.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get
> > > confusing.
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all
> softwares
> > > in
> > > > >>> bigtop
> > > > >>> > >> have
> > > > >>> > >> > > man
> > > > >>> > >> > > >> > page
> > > > >>> > >> > > >> > > >> for
> > > > >>> > >> > > >> > > >> >> > their
> > > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we should have it
> > too
> > > > for
> > > > >>> > >> dtcli.
> > > > >>> > >> > Is
> > > > >>> > >> > > >> there
> > > > >>> > >> > > >> > > any
> > > > >>> > >> > > >> > > >> >> > > > > documentation
> > > > >>> > >> > > >> > > >> >> > > > > > > > > which I
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man page?
> or
> > > can
> > > > >>> I use
> > > > >>> > >> > output
> > > > >>> > >> > > >> of
> > > > >>> > >> > > >> > > >> "dtcli
> > > > >>> > >> > > >> > > >> >> > > --help"?
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call
> > version
> > > > of
> > > > >>> apex
> > > > >>> > >> in
> > > > >>> > >> > > >> Bigtop
> > > > >>> > >> > > >> > as
> > > > >>> > >> > > >> > > >> >> 3.3.0 OR
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex
> > package
> > > of
> > > > >>> > bigtop
> > > > >>> > >> to
> > > > >>> > >> > > >> depend
> > > > >>> > >> > > >> > on
> > > > >>> > >> > > >> > > >> >> 2.7.1
> > > > >>> > >> > > >> > > >> >> > > > > version
> > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any
> problems
> > > > that
> > > > >>> we
> > > > >>> > see
> > > > >>> > >> > with
> > > > >>> > >> > > >> this
> > > > >>> > >> > > >> > > >> >> > dependency?
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > Should work without
> changed.
> > I
> > > > >>> thought
> > > > >>> > we
> > > > >>> > >> > > >> certified
> > > > >>> > >> > > >> > > >> >> against
> > > > >>> > >> > > >> > > >> >> > > > 2.7.0?
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the
> apache
> > > > mirror
> > > > >>> > from
> > > > >>> > >> > which
> > > > >>> > >> > > >> > bigtop
> > > > >>> > >> > > >> > > >> will
> > > > >>> > >> > > >> > > >> >> > pick
> > > > >>> > >> > > >> > > >> >> > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > apex
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > source for compilation
> and
> > > > >>> packaging.
> > > > >>> > >> > Please
> > > > >>> > >> > > >> > > correct
> > > > >>> > >> > > >> > > >> if
> > > > >>> > >> > > >> > > >> >> > > > > incorrect:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > >
> > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
> > apache.osuosl.org
> > > is
> > > > >>> the
> > > > >>> > >> mirror
> > > > >>> > >> > > >> used
> > > > >>> > >> > > >> > by
> > > > >>> > >> > > >> > > >> all
> > > > >>> > >> > > >> > > >> >> the
> > > > >>> > >> > > >> > > >> >> > > > > > softwares
> > > > >>> > >> > > >> > > >> >> > > > > > > in
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share your
> opinion.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at
> > 11:19
> > > > AM,
> > > > >>> > >> Chinmay
> > > > >>> > >> > > >> > Kolhatkar
> > > > >>> > >> > > >> > > <
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work
> on
> > > the
> > > > >>> > >> packaging.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already started
> > > > >>> discussion on
> > > > >>> > >> > bigtop
> > > > >>> > >> > > >> dev
> > > > >>> > >> > > >> > > >> mailing
> > > > >>> > >> > > >> > > >> >> > list
> > > > >>> > >> > > >> > > >> >> > > > for
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also
> > created a
> > > > >>> Jira
> > > > >>> > for
> > > > >>> > >> the
> > > > >>> > >> > > >> same.
> > > > >>> > >> > > >> > > For
> > > > >>> > >> > > >> > > >> >> this
> > > > >>> > >> > > >> > > >> >> > > > > > communities
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is the
> > > bigtop
> > > > >>> Jira:
> > > > >>> > >> > > >> > BIGTOP-2313.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at
> > 1:13
> > > > AM,
> > > > >>> > >> Thomas
> > > > >>> > >> > > >> Weise <
> > > > >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything
> > under
> > > > >>> > >> > prerequisites
> > > > >>> > >> > > >> that
> > > > >>> > >> > > >> > > >> would
> > > > >>> > >> > > >> > > >> >> be a
> > > > >>> > >> > > >> > > >> >> > > > > > problem.
> > > > >>> > >> > > >> > > >> >> > > > > > > We
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
> > > > >>> compatibility
> > > > >>> > >> as
> > > > >>> > >> > > part
> > > > >>> > >> > > >> of
> > > > >>> > >> > > >> > > >> >> becoming
> > > > >>> > >> > > >> > > >> >> > an
> > > > >>> > >> > > >> > > >> >> > > > > > > incubator
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the
> > > > packaging
> > > > >>> > during
> > > > >>> > >> > the
> > > > >>> > >> > > >> next
> > > > >>> > >> > > >> > > >> weeks.
> > > > >>> > >> > > >> > > >> >> > Since
> > > > >>> > >> > > >> > > >> >> > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > work
> > > > >>> > >> > > >> > > >> >> > > > > > > > > will
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop,
> related
> > > > >>> > discussions
> > > > >>> > >> and
> > > > >>> > >> > > >> JIRAs
> > > > >>> > >> > > >> > > >> should
> > > > >>> > >> > > >> > > >> >> > also
> > > > >>> > >> > > >> > > >> >> > > be
> > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have
> > the
> > > > >>> > packaging
> > > > >>> > >> in
> > > > >>> > >> > > >> place
> > > > >>> > >> > > >> > by
> > > > >>> > >> > > >> > > >> end
> > > > >>> > >> > > >> > > >> >> Feb.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at
> > > 10:14
> > > > >>> AM,
> > > > >>> > >> > Chinmay
> > > > >>> > >> > > >> > > >> Kolhatkar <
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> chinmay@datatorrent.com
> > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a
> work
> > on
> > > > >>> adding
> > > > >>> > >> > Apache
> > > > >>> > >> > > >> Apex
> > > > >>> > >> > > >> > > as a
> > > > >>> > >> > > >> > > >> >> > > component
> > > > >>> > >> > > >> > > >> >> > > > > to
> > > > >>> > >> > > >> > > >> >> > > > > > > > Apache
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the
> > packaging
> > > > >>> system
> > > > >>> > for
> > > > >>> > >> > the
> > > > >>> > >> > > >> > Apache
> > > > >>> > >> > > >> > > >> big
> > > > >>> > >> > > >> > > >> >> data
> > > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use
> it,
> > > > most
> > > > >>> > >> recently
> > > > >>> > >> > > EMR.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking
> > > Jira
> > > > >>> task
> > > > >>> > in
> > > > >>> > >> > > >> APEXCORE
> > > > >>> > >> > > >> > for
> > > > >>> > >> > > >> > > >> the
> > > > >>> > >> > > >> > > >> >> > same:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >>
> > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of
> > > execution
> > > > >>> is as
> > > > >>> > >> > > follows:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
> > > > >>> prerequisites*
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has
> some
> > > > hard
> > > > >>> and
> > > > >>> > >> soft
> > > > >>> > >> > > >> > > expectation
> > > > >>> > >> > > >> > > >> >> for
> > > > >>> > >> > > >> > > >> >> > new
> > > > >>> > >> > > >> > > >> >> > > > > > > > components
> > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into
> > Bigtop.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of
> > it:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> >
> > > > >>> > >> > > >>
> > > > >>> > >> > >
> > > > >>> > >> >
> > > > >>> > >>
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems
> to
> > be
> > > > >>> > standard
> > > > >>> > >> ASF
> > > > >>> > >> > > >> based
> > > > >>> > >> > > >> > > >> >> > > requirements,
> > > > >>> > >> > > >> > > >> >> > > > > but
> > > > >>> > >> > > >> > > >> >> > > > > > > few
> > > > >>> > >> > > >> > > >> >> > > > > > > > > need
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects
> > are
> > > > >>> > expected
> > > > >>> > >> to
> > > > >>> > >> > be
> > > > >>> > >> > > >> > > Licensed
> > > > >>> > >> > > >> > > >> >> under
> > > > >>> > >> > > >> > > >> >> > > > > Apache
> > > > >>> > >> > > >> > > >> >> > > > > > > > > License,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and
> their
> > > > >>> > >> dependencies
> > > > >>> > >> > are
> > > > >>> > >> > > >> > > expected
> > > > >>> > >> > > >> > > >> >> to be
> > > > >>> > >> > > >> > > >> >> > > > > > > compatible
> > > > >>> > >> > > >> > > >> >> > > > > > > > > with
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under
> > ASL
> > > > 2.0
> > > > >>> but
> > > > >>> > >> need
> > > > >>> > >> > to
> > > > >>> > >> > > >> > check
> > > > >>> > >> > > >> > > if
> > > > >>> > >> > > >> > > >> >> > > > > dependencies
> > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL
> > 2.0.
> > > > >>> This I
> > > > >>> > >> guess
> > > > >>> > >> > > >> would
> > > > >>> > >> > > >> > > be a
> > > > >>> > >> > > >> > > >> >> > > > > verification
> > > > >>> > >> > > >> > > >> >> > > > > > > > check.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects
> > are
> > > > >>> > expected
> > > > >>> > >> to
> > > > >>> > >> > be
> > > > >>> > >> > > >> > > >> compatible
> > > > >>> > >> > > >> > > >> >> > with
> > > > >>> > >> > > >> > > >> >> > > > all
> > > > >>> > >> > > >> > > >> >> > > > > of
> > > > >>> > >> > > >> > > >> >> > > > > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms
> > that
> > > > >>> Bigtop
> > > > >>> > >> > > >> distribution
> > > > >>> > >> > > >> > > is
> > > > >>> > >> > > >> > > >> >> > > targeting
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > > verifying
> > > > >>> > whether
> > > > >>> > >> > our
> > > > >>> > >> > > >> > > software
> > > > >>> > >> > > >> > > >> >> runs
> > > > >>> > >> > > >> > > >> >> > > fine
> > > > >>> > >> > > >> > > >> >> > > > in
> > > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
> > > > >>> ubuntu-14.04
> > > > >>> > >> > > debian-8
> > > > >>> > >> > > >> > > >> >> > opensuse-13.2.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests
> > that
> > > > >>> should
> > > > >>> > be
> > > > >>> > >> > > added
> > > > >>> > >> > > >> for
> > > > >>> > >> > > >> > > >> >> > deployment.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the
> > test
> > > > >>> > artifacts
> > > > >>> > >> > which
> > > > >>> > >> > > >> goes
> > > > >>> > >> > > >> > > >> beyond
> > > > >>> > >> > > >> > > >> >> > > smoke
> > > > >>> > >> > > >> > > >> >> > > > > test
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are
> > > basically
> > > > >>> the
> > > > >>> > >> > > integration
> > > > >>> > >> > > >> > > tests
> > > > >>> > >> > > >> > > >> for
> > > > >>> > >> > > >> > > >> >> > > > > > verification
> > > > >>> > >> > > >> > > >> >> > > > > > > > > after
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is
> a
> > > soft
> > > > >>> > >> > requirement,
> > > > >>> > >> > > >> but
> > > > >>> > >> > > >> > > aim
> > > > >>> > >> > > >> > > >> is
> > > > >>> > >> > > >> > > >> >> to
> > > > >>> > >> > > >> > > >> >> > > > > achieve
> > > > >>> > >> > > >> > > >> >> > > > > > > this
> > > > >>> > >> > > >> > > >> >> > > > > > > > > as
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have
> > > > >>> explanation why
> > > > >>> > >> not
> > > > >>> > >> > to
> > > > >>> > >> > > >> > > include.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any
> from
> > > the
> > > > >>> link
> > > > >>> > >> which
> > > > >>> > >> > > >> > > explicitly
> > > > >>> > >> > > >> > > >> >> needs
> > > > >>> > >> > > >> > > >> >> > to
> > > > >>> > >> > > >> > > >> >> > > > be
> > > > >>> > >> > > >> > > >> >> > > > > > > > checked
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please
> > let
> > > > us
> > > > >>> > know.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex
> > as
> > > > >>> > component
> > > > >>> > >> to
> > > > >>> > >> > > >> Bigtop*
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail
> > > > >>> archive of
> > > > >>> > >> > Bigtop,
> > > > >>> > >> > > >> it
> > > > >>> > >> > > >> > was
> > > > >>> > >> > > >> > > >> >> learnt
> > > > >>> > >> > > >> > > >> >> > > that
> > > > >>> > >> > > >> > > >> >> > > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community want to
> see
> > > the
> > > > >>> > >> addition of
> > > > >>> > >> > > new
> > > > >>> > >> > > >> > > >> >> components
> > > > >>> > >> > > >> > > >> >> > in
> > > > >>> > >> > > >> > > >> >> > > > > > phases.
> > > > >>> > >> > > >> > > >> >> > > > > > > > Here
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > are
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > > creating
> > > > of
> > > > >>> > >> package
> > > > >>> > >> > > i.e.
> > > > >>> > >> > > >> > rpm
> > > > >>> > >> > > >> > > &
> > > > >>> > >> > > >> > > >> deb
> > > > >>> > >> > > >> > > >> >> > > files.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > > > >>> documentations/READMEs,
> > > > >>> > >> > LICENSE,
> > > > >>> > >> > > >> > > >> DISCLAMER,
> > > > >>> > >> > > >> > > >> >> > NOTES
> > > > >>> > >> > > >> > > >> >> > > > etc
> > > > >>> > >> > > >> > > >> >> > > > > if
> > > > >>> > >> > > >> > > >> >> > > > > > > any
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> > documentation
> > > > that
> > > > >>> > need
> > > > >>> > >> to
> > > > >>> > >> > be
> > > > >>> > >> > > >> > added
> > > > >>> > >> > > >> > > to
> > > > >>> > >> > > >> > > >> >> > > > > distribution
> > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > >>> > >> > > >> > > >> >> > > > > > > > our
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license
> > > > >>> information of
> > > > >>> > >> > > >> > dependencies
> > > > >>> > >> > > >> > > >> >> required
> > > > >>> > >> > > >> > > >> >> > > to
> > > > >>> > >> > > >> > > >> >> > > > be
> > > > >>> > >> > > >> > > >> >> > > > > > > added
> > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at
> > very
> > > > >>> least)
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke
> > test
> > > > for
> > > > >>> > >> > packaging.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes
> for
> > > > >>> automatic
> > > > >>> > >> > > >> deployment
> > > > >>> > >> > > >> > and
> > > > >>> > >> > > >> > > >> >> > > > configuration
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet
> > recipes
> > > > for
> > > > >>> > >> > automatic
> > > > >>> > >> > > >> > > >> deployment
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For
> verification
> > > of
> > > > >>> > >> > deployments.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license
> clearance:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat'
> > to
> > > > make
> > > > >>> > sure
> > > > >>> > >> all
> > > > >>> > >> > > new
> > > > >>> > >> > > >> > > stuff
> > > > >>> > >> > > >> > > >> is
> > > > >>> > >> > > >> > > >> >> > > > compliant
> > > > >>> > >> > > >> > > >> >> > > > > > > with
> > > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license
> requirements.
> > If
> > > > >>> you add
> > > > >>> > >> code
> > > > >>> > >> > > >> > licenses
> > > > >>> > >> > > >> > > >> >> under
> > > > >>> > >> > > >> > > >> >> > > > > different
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need to
> be
> > > > >>> listed in
> > > > >>> > >> the
> > > > >>> > >> > > >> NOTICE.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your
> > > thoughts
> > > > >>> on
> > > > >>> > the
> > > > >>> > >> > > >> approach.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
> > > corresponding
> > > > >>> > >> > communication
> > > > >>> > >> > > >> on
> > > > >>> > >> > > >> > > >> bigtop
> > > > >>> > >> > > >> > > >> >> > > mailing
> > > > >>> > >> > > >> > > >> >> > > > > list
> > > > >>> > >> > > >> > > >> >> > > > > > > as
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some
> specific
> > > > >>> > >> > > >> questions/suggestions
> > > > >>> > >> > > >> > > >> >> related to
> > > > >>> > >> > > >> > > >> >> > > > what
> > > > >>> > >> > > >> > > >> >> > > > > > > should
> > > > >>> > >> > > >> > > >> >> > > > > > > > > be
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the
> package
> > > and
> > > > >>> what
> > > > >>> > >> > should
> > > > >>> > >> > > be
> > > > >>> > >> > > >> > the
> > > > >>> > >> > > >> > > >> smoke
> > > > >>> > >> > > >> > > >> >> > > tests,
> > > > >>> > >> > > >> > > >> >> > > > > but
> > > > >>> > >> > > >> > > >> >> > > > > > > in
> > > > >>> > >> > > >> > > >> >> > > > > > > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not
> having
> > > too
> > > > >>> much
> > > > >>> > >> > content
> > > > >>> > >> > > >> > here,
> > > > >>> > >> > > >> > > >> we'll
> > > > >>> > >> > > >> > > >> >> > put
> > > > >>> > >> > > >> > > >> >> > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > questions
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in
> this
> > > > >>> > mailthread.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> >
> > > > >>> > >> > > >>
> > > > >>> > >> > > >
> > > > >>> > >> > > >
> > > > >>> > >> > >
> > > > >>> > >> >
> > > > >>> > >>
> > > > >>> > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Munagala Ramanath <ra...@datatorrent.com>]

Tushar is right: apex (formerly dtcli) talks directly to Yarn and does not
depend on dtGateway or the REST API.

Ram

On Thu, Feb 18, 2016 at 12:07 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:

> In following statement, can you replace dtGateway with ApplicationMaster
> and Yarn. or may be remove this statement entirely.
> apex is a wrapper around the REST API provided by dtGatway,
>
> - Tushar.
>
>
>
>
> On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Hi,
> >
> > I've attached man page for apex in Jira. Here is the link to it:
> > https://issues.apache.org/jira/secure/attachment/12788403/apex.1
> >
> > Please view it by: man -l apex.1
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > Hi Chinmay,
> > >
> > > I am not able to find the attachment, may be attachments are not
> allowed
> > in
> > > the mailing list. Can you attach the man page to the Jira?
> > >
> > > - Tushar.
> > >
> > >
> > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <chinmay@apache.org
> >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I've attached man page file for the purpose of bigtop integration.
> > > > Please have a look and let me know if there any any feedback about
> it.
> > > >
> > > > You can see the man page content of this file with:
> > > > man -l apex.1
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > >
> > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> > chinmay@apache.org>
> > > > wrote:
> > > >
> > > >> Thanks. Will follow bigtop recommendation. i.e. remove
> "-incubating".
> > > >>
> > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
> thomas@datatorrent.com
> > >
> > > >> wrote:
> > > >>
> > > >>> Please follow the recommendation and continue discussion on the
> > Bigtop
> > > >>> list.
> > > >>>
> > > >>> --
> > > >>> sent from mobile
> > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <ch...@apache.org>
> > > wrote:
> > > >>>
> > > >>> > I got a recommendation from bigtop community to drop
> "-incubating"
> > > from
> > > >>> > version.
> > > >>> >
> > > >>> > Here is the mail thread:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> > > >>> >
> > > >>> > Please let me know if removing "-incubating" is fine.
> > > >>> >
> > > >>> > Thanks,
> > > >>> > Chinmay.
> > > >>> >
> > > >>> >
> > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> > > chinmay@apache.org
> > > >>> >
> > > >>> > wrote:
> > > >>> >
> > > >>> > > I've already started communication there.
> > > >>> > >
> > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> > > >>> thomas@datatorrent.com>
> > > >>> > > wrote:
> > > >>> > >
> > > >>> > >> This is a question you should ask on the Bigtop mailing list.
> > > >>> > >>
> > > >>> > >> --
> > > >>> > >> sent from mobile
> > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org>
> > > >>> wrote:
> > > >>> > >>
> > > >>> > >> > Yes. In case of RPM, the only restriction on version is that
> > it
> > > >>> can
> > > >>> > not
> > > >>> > >> > contain '-'.
> > > >>> > >> > However, as a workaround, one can use 'incubating' as a part
> > of
> > > >>> > >> %{release}.
> > > >>> > >> >
> > > >>> > >> > Version: 3.3.0
> > > >>> > >> > Release: incubating
> > > >>> > >> >
> > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> > > >>> > >> >
> > > >>> > >> > It is the same practice followed for pre-release packages
> e.g.
> > > >>> with
> > > >>> > rcX,
> > > >>> > >> > alphaX, betaX being used in %{release}.
> > > >>> > >> >
> > > >>> > >> > Regards,
> > > >>> > >> > Pradeep A. Dalvi
> > > >>> > >> >
> > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> > > >>> > chinmay@apache.org>
> > > >>> > >> > wrote:
> > > >>> > >> >
> > > >>> > >> > > Hi Everyone,
> > > >>> > >> > >
> > > >>> > >> > > In one of the previous mails in this thread, we decided
> that
> > > >>> version
> > > >>> > >> of
> > > >>> > >> > the
> > > >>> > >> > > package will be "3.3.0-incubating".
> > > >>> > >> > > The problem that I'm facing is in RPM base version "-" is
> > not
> > > >>> > allowed.
> > > >>> > >> > >
> > > >>> > >> > > Can we have package version to "3.3.0"?
> > > >>> > >> > > i.e. the packages created will be as follows:
> > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> > > >>> > >> > >
> > > >>> > >> > > There is another integration for Zepplin in Bigtop which
> is
> > > >>> > incubating
> > > >>> > >> > and
> > > >>> > >> > > is having base version as "0.5.6".
> > > >>> > >> > >
> > > >>> > >> > > Please let me know your thoughts.
> > > >>> > >> > >
> > > >>> > >> > > Thanks,
> > > >>> > >> > > Chinmay.
> > > >>> > >> > >
> > > >>> > >> > >
> > > >>> > >> > >
> > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> > > >>> > >> chinmay@apache.org>
> > > >>> > >> > > wrote:
> > > >>> > >> > >
> > > >>> > >> > > > Thanks everyone for the feedback.
> > > >>> > >> > > >
> > > >>> > >> > > > Also as Ram and David mentioned, I'll start creating the
> > man
> > > >>> page
> > > >>> > as
> > > >>> > >> > > well.
> > > >>> > >> > > >
> > > >>> > >> > > > Thanks,
> > > >>> > >> > > > Chinmay.
> > > >>> > >> > > >
> > > >>> > >> > > >
> > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> > > >>> > >> thomas@datatorrent.com>
> > > >>> > >> > > > wrote:
> > > >>> > >> > > >
> > > >>> > >> > > >> I think we already agreed that we will retain the
> current
> > > >>> script
> > > >>> > >> name
> > > >>> > >> > in
> > > >>> > >> > > >> this bigtop iteration. There is a ticket for renaming,
> > add
> > > >>> you
> > > >>> > >> > > suggestions
> > > >>> > >> > > >> there.
> > > >>> > >> > > >>
> > > >>> > >> > > >> --
> > > >>> > >> > > >> sent from mobile
> > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > > >>> > sandesh@datatorrent.com>
> > > >>> > >> > > wrote:
> > > >>> > >> > > >>
> > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start the
> > transition
> > > >>> > >> process by
> > > >>> > >> > > >> making
> > > >>> > >> > > >> > dtcli as the symlink to apex cli.
> > > >>> > >> > > >> >
> > > >>> > >> > > >> > Renaming is a breaking change should be done during
> > major
> > > >>> > version
> > > >>> > >> > > number
> > > >>> > >> > > >> > change.
> > > >>> > >> > > >> >
> > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> > > >>> > david@datatorrent.com
> > > >>> > >> >
> > > >>> > >> > > >> wrote:
> > > >>> > >> > > >> >
> > > >>> > >> > > >> > > Apologies for hitting the "Send" button too early.
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > What is the scope of the name change?  If we simply
> > > have
> > > >>> a
> > > >>> > >> symlink
> > > >>> > >> > > >> from
> > > >>> > >> > > >> > > apex to dtcli, then it should have no backward
> > > >>> compatibility
> > > >>> > >> > problem
> > > >>> > >> > > >> at
> > > >>> > >> > > >> > > all.
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > But the proper name change would involve:
> > > >>> > >> > > >> > > - Change the name of DTCli class to ApexCli
> > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> > > >>> > >> > > >> > > - Change the initial greeting when running the CLI
> > from
> > > >>> > >> something
> > > >>> > >> > > like
> > > >>> > >> > > >> > "DT
> > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > The above changes may break existing scripts that
> > rely
> > > >>> on the
> > > >>> > >> the
> > > >>> > >> > > >> string
> > > >>> > >> > > >> > > "DT".
> > > >>> > >> > > >> > > But if folks are okay with this potential problem,
> I
> > am
> > > >>> okay
> > > >>> > >> with
> > > >>> > >> > > the
> > > >>> > >> > > >> > name
> > > >>> > >> > > >> > > change.
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > For the man page, we can start with the content of
> > this
> > > >>> page:
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > Plus the command line options when doing dtcli -h:
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > usage: DTCli
> > > >>> > >> > > >> > >  -e <arg>    Commands are read from the argument
> > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at all time
> > > >>> > >> > > >> > >  -h          Print this help
> > > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos principal
> > > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> > > >>> > >> > > >> > >  -r          JSON Raw mode
> > > >>> > >> > > >> > >  -v          Verbose mode level 1
> > > >>> > >> > > >> > >  -vv         Verbose mode level 2
> > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > David
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> > > >>> > >> david@datatorrent.com
> > > >>> > >> > >
> > > >>> > >> > > >> > wrote:
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > > What is the scope of the name change?  If we
> simply
> > > >>> have a
> > > >>> > >> > symlink
> > > >>> > >> > > >> from
> > > >>> > >> > > >> > > > apex to dtcli, then it should have no backward
> > > >>> > compatibility
> > > >>> > >> > > >> problem at
> > > >>> > >> > > >> > > all.
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > But the proper name change would involve:
> > > >>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
> > > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> > > >>> > >> > > >> > > > - Change the initial greeting when running the
> CLI
> > > from
> > > >>> > >> > something
> > > >>> > >> > > >> like
> > > >>> > >> > > >> > > "DT
> > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > The above changes may break existing scripts that
> > > rely
> > > >>> on
> > > >>> > the
> > > >>> > >> > the
> > > >>> > >> > > >> > string
> > > >>> > >> > > >> > > > "DT".
> > > >>> > >> > > >> > > > But if folks are okay with this potential
> problem,
> > I
> > > am
> > > >>> > okay
> > > >>> > >> > with
> > > >>> > >> > > >> the
> > > >>> > >> > > >> > > name
> > > >>> > >> > > >> > > > change.
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > For the man page, we can start with the content
> of
> > > this
> > > >>> > page:
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay
> Kolhatkar
> > <
> > > >>> > >> > > >> chinmay@apache.org
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > > wrote:
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >> Here is a suggestion I have related to name
> > change:
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> 1. While packaging we change the name of "dtcli"
> > > >>> script to
> > > >>> > >> > > "apex".
> > > >>> > >> > > >> > This
> > > >>> > >> > > >> > > is
> > > >>> > >> > > >> > > >> done so that in future when we change the name
> to
> > > >>> "apex",
> > > >>> > >> users
> > > >>> > >> > > of
> > > >>> > >> > > >> > > bigtop
> > > >>> > >> > > >> > > >> apex don't have to transition much.
> > > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli which
> points
> > > to
> > > >>> apex
> > > >>> > >> > > script.
> > > >>> > >> > > >> > This
> > > >>> > >> > > >> > > is
> > > >>> > >> > > >> > > >> for backward compatibility.
> > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first
> > > integration
> > > >>> in
> > > >>> > >> > bigtop.
> > > >>> > >> > > >> This
> > > >>> > >> > > >> > is
> > > >>> > >> > > >> > > >> in
> > > >>> > >> > > >> > > >> the interest of not changing the content of
> dtcli
> > or
> > > >>> apex
> > > >>> > >> > script.
> > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these
> naming
> > > >>> related
> > > >>> > >> > > changes
> > > >>> > >> > > >> and
> > > >>> > >> > > >> > > >> update bigtop repository later to remove
> > references
> > > to
> > > >>> > dtcli
> > > >>> > >> > all
> > > >>> > >> > > >> > > together.
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> Please share your thoughts on above approach.
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> Also, please share what could be the man page
> > > content
> > > >>> for
> > > >>> > >> > "apex".
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> Thanks,
> > > >>> > >> > > >> > > >> Chinmay.
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay
> > Kolhatkar <
> > > >>> > >> > > >> > chinmay@apache.org>
> > > >>> > >> > > >> > > >> wrote:
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> > Yes.. It'll be build from source tar
> downloaded
> > > >>> from one
> > > >>> > >> of
> > > >>> > >> > the
> > > >>> > >> > > >> > apache
> > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced from
> > > dtcli.
> > > >>> > This
> > > >>> > >> > > leaves
> > > >>> > >> > > >> us
> > > >>> > >> > > >> > 3
> > > >>> > >> > > >> > > >> > options:
> > > >>> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli
> would
> > > >>> need a
> > > >>> > >> > change
> > > >>> > >> > > >> while
> > > >>> > >> > > >> > > >> > packaging.
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> > But if we want to use dtcli as it is, then we
> > > would
> > > >>> need
> > > >>> > >> > > >> dt-env.sh
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise
> <
> > > >>> > >> > > >> > thomas@datatorrent.com
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >> > wrote:
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >> Looks good overall, though there shouldn't be
> > any
> > > >>> > >> dt-env.sh
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >> What will the bigtop package be built from,
> the
> > > >>> source
> > > >>> > >> tar?
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay
> > > Kolhatkar
> > > >>> <
> > > >>> > >> > > >> > > chinmay@apache.org
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >> wrote:
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >> > Hi Thomas,
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > First, we're not changing the name
> anywhere.
> > > >>> We'll
> > > >>> > >> follow
> > > >>> > >> > > what
> > > >>> > >> > > >> > > >> >> currently is
> > > >>> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating
> > version.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory
> > structure
> > > >>> below
> > > >>> > >> which
> > > >>> > >> > > is
> > > >>> > >> > > >> > > inline
> > > >>> > >> > > >> > > >> >> with
> > > >>> > >> > > >> > > >> >> > other existing integrations with bigtop.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > The need for each file is as follows:
> > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> > > >>> > >> > > >> engine/src/main/scripts/
> > > >>> > >> > > >> > > of
> > > >>> > >> > > >> > > >> >> source
> > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from
> > dtcli.
> > > >>> This
> > > >>> > >> > > searches
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > >> hadoop
> > > >>> > >> > > >> > > >> >> > binary path and exports an env variable for
> > > >>> dtcli to
> > > >>> > >> use
> > > >>> > >> > it.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is
> > > >>> extracted
> > > >>> > >> in a
> > > >>> > >> > > >> similar
> > > >>> > >> > > >> > > >> way to
> > > >>> > >> > > >> > > >> >> > how DT Community Edition finds dependency
> > jars.
> > > >>> > >> > > >> > > >> >> > Please note that this is the first
> iteration
> > > >>> list of
> > > >>> > >> jars.
> > > >>> > >> > > I'm
> > > >>> > >> > > >> > > >> trying to
> > > >>> > >> > > >> > > >> >> > narrow this down to only those which are
> > really
> > > >>> > >> required.
> > > >>> > >> > > >> > > >> >> > The test that I'm running to check if dtcli
> > > runs
> > > >>> fine
> > > >>> > >> with
> > > >>> > >> > > >> given
> > > >>> > >> > > >> > > >> >> dependency
> > > >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi
> > demo.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Above is the directory structure which is
> > > >>> required
> > > >>> > for
> > > >>> > >> CLI
> > > >>> > >> > > to
> > > >>> > >> > > >> > work.
> > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned
> are
> > > some
> > > >>> > >> > additional
> > > >>> > >> > > >> > files
> > > >>> > >> > > >> > > >> which
> > > >>> > >> > > >> > > >> >> > makes the like of administrator easier. For
> > eg.
> > > >>> > >> /etc/skel
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Please let me know if this is inline with
> > what
> > > >>> you're
> > > >>> > >> > > >> thinking.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Thanks,
> > > >>> > >> > > >> > > >> >> > Chinmay.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > .
> > > >>> > >> > > >> > > >> >> > |-- bin
> > > >>> > >> > > >> > > >> >> > |   `-- dtcli
> > > >>> > >> > > >> > > >> >> > |-- conf
> > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > > >>> > >> > > >> > > >> >> > `-- lib
> > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > > >>> > >> > > >> > > >> >> >     |--
> > apex-bufferserver-3.3.0-incubating.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > > >>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > > >>> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > > >>> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |--
> jetty-websocket-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > > >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas
> > Weise <
> > > >>> > >> > > >> > > >> thomas@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > wrote:
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > > Chinmay,
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > Before discussing where to put the files,
> > > let's
> > > >>> > make
> > > >>> > >> > sure
> > > >>> > >> > > >> they
> > > >>> > >> > > >> > > are
> > > >>> > >> > > >> > > >> >> really
> > > >>> > >> > > >> > > >> >> > > needed for the operation of the CLI. As
> for
> > > >>> names,
> > > >>> > >> > > anything
> > > >>> > >> > > >> new
> > > >>> > >> > > >> > > >> needs
> > > >>> > >> > > >> > > >> >> to
> > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and should
> follow
> > > >>> common
> > > >>> > >> > > >> conventions,
> > > >>> > >> > > >> > > >> >> especially
> > > >>> > >> > > >> > > >> >> > > in Bigtop where there are many existing
> > > >>> > integrations
> > > >>> > >> to
> > > >>> > >> > > look
> > > >>> > >> > > >> > at.
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > Thanks,
> > > >>> > >> > > >> > > >> >> > > Thomas
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay
> > > >>> Kolhatkar
> > > >>> > <
> > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > wrote:
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep and
> Aniruddha.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to change
> the
> > > >>> dtcli
> > > >>> > >> > > >> considering it
> > > >>> > >> > > >> > > >> works
> > > >>> > >> > > >> > > >> >> > with
> > > >>> > >> > > >> > > >> >> > > DT
> > > >>> > >> > > >> > > >> >> > > > Community edition. We can keep the
> > > directory
> > > >>> > >> structure
> > > >>> > >> > > >> > similar
> > > >>> > >> > > >> > > to
> > > >>> > >> > > >> > > >> >> that.
> > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which contains
> > > >>> > information
> > > >>> > >> > > >> required
> > > >>> > >> > > >> > for
> > > >>> > >> > > >> > > >> >> dtcli
> > > >>> > >> > > >> > > >> >> > to
> > > >>> > >> > > >> > > >> >> > > > launch.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop community
> > that
> > > >>> > whether
> > > >>> > >> > they
> > > >>> > >> > > >> > > >> facilitate
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs before user
> is
> > > >>> created.
> > > >>> > >> In
> > > >>> > >> > > >> either
> > > >>> > >> > > >> > > case,
> > > >>> > >> > > >> > > >> >> > current
> > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home
> > > directory.
> > > >>> > Also
> > > >>> > >> > > before
> > > >>> > >> > > >> > > putting
> > > >>> > >> > > >> > > >> >> > > anything
> > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define what are
> > the
> > > >>> > default
> > > >>> > >> > > >> contents
> > > >>> > >> > > >> > > that
> > > >>> > >> > > >> > > >> >> > should
> > > >>> > >> > > >> > > >> >> > > go
> > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no
> > defaults,
> > > >>> > >> probably we
> > > >>> > >> > > >> should
> > > >>> > >> > > >> > > not
> > > >>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice.
> > > This
> > > >>> way
> > > >>> > >> > > contents
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > >> >> dt-env.sh
> > > >>> > >> > > >> > > >> >> > > are
> > > >>> > >> > > >> > > >> >> > > > present as env variables. But I see a
> > catch
> > > >>> > there.
> > > >>> > >> > > Adding
> > > >>> > >> > > >> > > >> dt-env.sh
> > > >>> > >> > > >> > > >> >> to
> > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the
> > variables
> > > >>> > >> available
> > > >>> > >> > at
> > > >>> > >> > > >> > > runtime
> > > >>> > >> > > >> > > >> all
> > > >>> > >> > > >> > > >> >> > the
> > > >>> > >> > > >> > > >> >> > > > time. I feel a little skeptical about
> > that.
> > > >>> > Maybe a
> > > >>> > >> > > >> possible
> > > >>> > >> > > >> > > >> >> collision
> > > >>> > >> > > >> > > >> >> > > can
> > > >>> > >> > > >> > > >> >> > > > occur with other application vars.
> > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
> > > >>> > >> > "../conf/dt-env.sh"
> > > >>> > >> > > >> and
> > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables
> are
> > > >>> anyway
> > > >>> > >> > > >> available.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is
> > > >>> happening
> > > >>> > at
> > > >>> > >> > > Bigtop
> > > >>> > >> > > >> > > >> >> > > Jira/repository.
> > > >>> > >> > > >> > > >> >> > > > Packaging related code usually goes
> > there.
> > > >>> > (That's
> > > >>> > >> > what
> > > >>> > >> > > >> all
> > > >>> > >> > > >> > the
> > > >>> > >> > > >> > > >> >> > > components
> > > >>> > >> > > >> > > >> >> > > > in bigtop does).
> > > >>> > >> > > >> > > >> >> > > > Having said that, once a PR is created,
> > > I'll
> > > >>> be
> > > >>> > >> > sharing
> > > >>> > >> > > >> the
> > > >>> > >> > > >> > > link
> > > >>> > >> > > >> > > >> of
> > > >>> > >> > > >> > > >> >> PR
> > > >>> > >> > > >> > > >> >> > > > here, so that apex community as well
> can
> > > >>> review
> > > >>> > it.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > Chinmay.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM,
> > Aniruddha
> > > >>> > >> Thombare <
> > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> > > >>> > >> > > >> > > >> >> > > > > We may need to iron out details about
> > > exact
> > > >>> > paths
> > > >>> > >> > etc.
> > > >>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR
> > comments.
> > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep
> A.
> > > >>> Dalvi <
> > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > > >>> > >> > > >> > > >> >> > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM,
> > > Chinmay
> > > >>> > >> > Kolhatkar <
> > > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present
> in
> > > DT
> > > >>> > >> community
> > > >>> > >> > > >> > edition
> > > >>> > >> > > >> > > >> will
> > > >>> > >> > > >> > > >> >> be
> > > >>> > >> > > >> > > >> >> > > > > included
> > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I found
> > from
> > > >>> > common
> > > >>> > >> > > >> > dependencies
> > > >>> > >> > > >> > > >> >> between
> > > >>> > >> > > >> > > >> >> > > of
> > > >>> > >> > > >> > > >> >> > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > > >>> > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > > >>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > >
> > jetty-websocket-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some
> > tests
> > > do
> > > >>> > check
> > > >>> > >> > that
> > > >>> > >> > > >> > dtcli
> > > >>> > >> > > >> > > >> works
> > > >>> > >> > > >> > > >> >> > > > properly
> > > >>> > >> > > >> > > >> >> > > > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with
> > > only
> > > >>> > these
> > > >>> > >> > > >> libraries
> > > >>> > >> > > >> > > >> >> present in
> > > >>> > >> > > >> > > >> >> > > > > > isolation
> > > >>> > >> > > >> > > >> >> > > > > > > without dependency on local m2.
> > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there are
> unwanted
> > > >>> jars
> > > >>> > >> which
> > > >>> > >> > are
> > > >>> > >> > > >> used
> > > >>> > >> > > >> > > for
> > > >>> > >> > > >> > > >> >> > compile
> > > >>> > >> > > >> > > >> >> > > > > time
> > > >>> > >> > > >> > > >> >> > > > > > > dependency and not runtime in
> above
> > > >>> list
> > > >>> > >> which I
> > > >>> > >> > > can
> > > >>> > >> > > >> > drop
> > > >>> > >> > > >> > > >> to
> > > >>> > >> > > >> > > >> >> keep
> > > >>> > >> > > >> > > >> >> > > > > package
> > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if
> all
> > > of
> > > >>> the
> > > >>> > >> > > >> dt-env.sh is
> > > >>> > >> > > >> > > >> >> required.
> > > >>> > >> > > >> > > >> >> > > > Only
> > > >>> > >> > > >> > > >> >> > > > > > > required values I see in it are
> > > >>> classpath.
> > > >>> > >> and
> > > >>> > >> > > >> dtcli is
> > > >>> > >> > > >> > > >> >> capable
> > > >>> > >> > > >> > > >> >> > of
> > > >>> > >> > > >> > > >> >> > > > > > building
> > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have
> > to
> > > >>> see
> > > >>> > how
> > > >>> > >> > dtcli
> > > >>> > >> > > >> can
> > > >>> > >> > > >> > > find
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > jar
> > > >>> > >> > > >> > > >> >> > > > > path,
> > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf
> files.
> > If
> > > >>> we
> > > >>> > >> need a
> > > >>> > >> > > >> change
> > > >>> > >> > > >> > in
> > > >>> > >> > > >> > > >> >> dtcli,
> > > >>> > >> > > >> > > >> >> > > then
> > > >>> > >> > > >> > > >> >> > > > > > > community's opinion is required
> for
> > > >>> whether
> > > >>> > >> > dtcli
> > > >>> > >> > > >> > should
> > > >>> > >> > > >> > > >> >> change
> > > >>> > >> > > >> > > >> >> > in
> > > >>> > >> > > >> > > >> >> > > > our
> > > >>> > >> > > >> > > >> >> > > > > > repo
> > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with required
> > > changes
> > > >>> > >> exist in
> > > >>> > >> > > >> bigtop
> > > >>> > >> > > >> > > >> repo
> > > >>> > >> > > >> > > >> >> > until
> > > >>> > >> > > >> > > >> >> > > we
> > > >>> > >> > > >> > > >> >> > > > > > make
> > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Can we have these file paths set
> > using
> > > >>> > >> environment
> > > >>> > >> > > >> > > variables
> > > >>> > >> > > >> > > >> >> with
> > > >>> > >> > > >> > > >> >> > > some
> > > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> > > >>> > >> > > >> > > >> >> > > > > > Then we can set such params in
> > > dt-env.sh.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this
> file
> > > >>> > different
> > > >>> > >> > from
> > > >>> > >> > > >> > > >> >> dt-site.xml?
> > > >>> > >> > > >> > > >> >> > Not
> > > >>> > >> > > >> > > >> >> > > > > sure
> > > >>> > >> > > >> > > >> >> > > > > > if
> > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help.
> > > Files &
> > > >>> > Dirs
> > > >>> > >> > from
> > > >>> > >> > > >> > > /etc/skel
> > > >>> > >> > > >> > > >> >> are
> > > >>> > >> > > >> > > >> >> > > > copied
> > > >>> > >> > > >> > > >> >> > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > home of new user when useradd
> > program
> > > >>> is
> > > >>> > >> called.
> > > >>> > >> > > But
> > > >>> > >> > > >> > for
> > > >>> > >> > > >> > > >> >> existing
> > > >>> > >> > > >> > > >> >> > > > users
> > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any use.
> Moreover,
> > > >>> dtcli
> > > >>> > >> > creates
> > > >>> > >> > > >> > ~/.dt/
> > > >>> > >> > > >> > > on
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > fly
> > > >>> > >> > > >> > > >> >> > > > if
> > > >>> > >> > > >> > > >> >> > > > > > not
> > > >>> > >> > > >> > > >> >> > > > > > > encountered for the first time.
> > Again
> > > >>> > >> correct me
> > > >>> > >> > > if
> > > >>> > >> > > >> I'm
> > > >>> > >> > > >> > > >> wrong.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right. However this
> > would
> > > >>> > >> probably be
> > > >>> > >> > > >> > > necessary
> > > >>> > >> > > >> > > >> >> step
> > > >>> > >> > > >> > > >> >> > for
> > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get
> > installed
> > > >>> > during
> > > >>> > >> OS
> > > >>> > >> > > >> install
> > > >>> > >> > > >> > > and
> > > >>> > >> > > >> > > >> >> > before
> > > >>> > >> > > >> > > >> >> > > > user
> > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile
> > ->
> > > >>> I'm
> > > >>> > not
> > > >>> > >> > sure
> > > >>> > >> > > >> its
> > > >>> > >> > > >> > the
> > > >>> > >> > > >> > > >> best
> > > >>> > >> > > >> > > >> >> > > idea.
> > > >>> > >> > > >> > > >> >> > > > > > We'll
> > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location
> which
> > is
> > > >>> by
> > > >>> > >> default
> > > >>> > >> > > >> > present
> > > >>> > >> > > >> > > in
> > > >>> > >> > > >> > > >> >> path
> > > >>> > >> > > >> > > >> >> > > i.e.
> > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > For other env variables specific to
> > > apex,
> > > >>> > I'll
> > > >>> > >> > > prefer
> > > >>> > >> > > >> to
> > > >>> > >> > > >> > > use
> > > >>> > >> > > >> > > >> >> > > dt-env.sh
> > > >>> > >> > > >> > > >> >> > > > > and
> > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather than
> > > changing
> > > >>> > >> bashrc
> > > >>> > >> > > >> etc...
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
> > > >>> /etc/profile.d/
> > > >>> > >> shall
> > > >>> > >> > > be
> > > >>> > >> > > >> the
> > > >>> > >> > > >> > > >> >> cleaner
> > > >>> > >> > > >> > > >> >> > way
> > > >>> > >> > > >> > > >> >> > > > to
> > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM,
> > > >>> Aniruddha
> > > >>> > >> > > Thombare
> > > >>> > >> > > >> <
> > > >>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com>
> wrote:
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > We need to consider following:
> > > >>> > >> > > >> > > >> >> > > > > > > > System wide default config
> files
> > > can
> > > >>> be
> > > >>> > >> > located
> > > >>> > >> > > at
> > > >>> > >> > > >> > > >> following
> > > >>> > >> > > >> > > >> >> > > > > locations:
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may
> have
> > > to
> > > >>> > change
> > > >>> > >> > > dtcli
> > > >>> > >> > > >> > > >> behaviour
> > > >>> > >> > > >> > > >> >> on
> > > >>> > >> > > >> > > >> >> > > how
> > > >>> > >> > > >> > > >> >> > > > it
> > > >>> > >> > > >> > > >> >> > > > > > > finds
> > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may
> > have
> > > to
> > > >>> > >> change
> > > >>> > >> > > dtcli
> > > >>> > >> > > >> > > >> >> behaviour on
> > > >>> > >> > > >> > > >> >> > > how
> > > >>> > >> > > >> > > >> >> > > > > it
> > > >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
> > > >>> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and
> > > other
> > > >>> > files
> > > >>> > >> > (for
> > > >>> > >> > > >> new
> > > >>> > >> > > >> > > users
> > > >>> > >> > > >> > > >> >> that
> > > >>> > >> > > >> > > >> >> > > will
> > > >>> > >> > > >> > > >> >> > > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > > created in system in future.)
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > We may also have to modify
> > bashrc /
> > > >>> > >> > bashprofile
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > >> >> population
> > > >>> > >> > > >> > > >> >> > > any
> > > >>> > >> > > >> > > >> >> > > > > > > > variables that are required.
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your
> > comments /
> > > >>> > >> > suggestion.
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16
> AM,
> > > >>> Thomas
> > > >>> > >> Weise
> > > >>> > >> > <
> > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > -->
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05
> AM,
> > > >>> Chinmay
> > > >>> > >> > > >> Kolhatkar <
> > > >>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging
> > > effort
> > > >>> > >> (rpm+deb)
> > > >>> > >> > > of
> > > >>> > >> > > >> > apex,
> > > >>> > >> > > >> > > >> here
> > > >>> > >> > > >> > > >> >> > are
> > > >>> > >> > > >> > > >> >> > > > some
> > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> > > >>> > >> > > >> > > >> >> > > > > > > > > > about package structure
> etc..
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on
> bbigtop
> > > >>> mailing
> > > >>> > >> > list, I
> > > >>> > >> > > >> have
> > > >>> > >> > > >> > > >> some
> > > >>> > >> > > >> > > >> >> > > question
> > > >>> > >> > > >> > > >> >> > > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory
> structure
> > of
> > > >>> apex
> > > >>> > >> > package
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > both
> > > >>> > >> > > >> > > >> >> deb &
> > > >>> > >> > > >> > > >> >> > > > rpm:
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other
> > > >>> dependent
> > > >>> > >> jars>
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> > > >>> > /usr/lib/apex/bin/dtcli
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> /usr/share/doc/man/man1/dtcli.1.gz
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > /usr/share/doc/apex/license/LICENSE.txt
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> /usr/share/doc/apex/CHANGELOG
> > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
> 3.3.0-incubating.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> =
> All
> > > the
> > > >>> 3rd
> > > >>> > >> party
> > > >>> > >> > > >> jars
> > > >>> > >> > > >> > > which
> > > >>> > >> > > >> > > >> >> are
> > > >>> > >> > > >> > > >> >> > > > > required
> > > >>> > >> > > >> > > >> >> > > > > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the
> > > dependencies
> > > >>> are
> > > >>> > >> > > packaged
> > > >>> > >> > > >> as
> > > >>> > >> > > >> > > >> part of
> > > >>> > >> > > >> > > >> >> > > > rpm/deb
> > > >>> > >> > > >> > > >> >> > > > > by
> > > >>> > >> > > >> > > >> >> > > > > > > any
> > > >>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what those
> jars
> > > >>> are in
> > > >>> > >> > Bigtop
> > > >>> > >> > > >> > > context?
> > > >>> > >> > > >> > > >> >> Same
> > > >>> > >> > > >> > > >> >> > as
> > > >>> > >> > > >> > > >> >> > > > > > shipped
> > > >>> > >> > > >> > > >> >> > > > > > > > with
> > > >>> > >> > > >> > > >> >> > > > > > > > > DT community addition under
> > lib/
> > > ?
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> =
> Licenses
> > of
> > > >>> > >> > > corresponding
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > 3rd
> > > >>> > >> > > >> > > >> >> party
> > > >>> > >> > > >> > > >> >> > > jars
> > > >>> > >> > > >> > > >> >> > > > > > which
> > > >>> > >> > > >> > > >> >> > > > > > > > > needs
> > > >>> > >> > > >> > > >> >> > > > > > > > > > to included while
> packaging.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions related to this:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli
> of
> > > >>> apex as
> > > >>> > >> > "apex"
> > > >>> > >> > > >> > instead
> > > >>> > >> > > >> > > >> of
> > > >>> > >> > > >> > > >> >> > > "dtcli"
> > > >>> > >> > > >> > > >> >> > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > I think we should keep the
> name
> > > >>> until
> > > >>> > we
> > > >>> > >> are
> > > >>> > >> > > >> able
> > > >>> > >> > > >> > to
> > > >>> > >> > > >> > > >> >> change
> > > >>> > >> > > >> > > >> >> > it
> > > >>> > >> > > >> > > >> >> > > in
> > > >>> > >> > > >> > > >> >> > > > > > Apex.
> > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get
> > confusing.
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares
> > in
> > > >>> bigtop
> > > >>> > >> have
> > > >>> > >> > > man
> > > >>> > >> > > >> > page
> > > >>> > >> > > >> > > >> for
> > > >>> > >> > > >> > > >> >> > their
> > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we should have it
> too
> > > for
> > > >>> > >> dtcli.
> > > >>> > >> > Is
> > > >>> > >> > > >> there
> > > >>> > >> > > >> > > any
> > > >>> > >> > > >> > > >> >> > > > > documentation
> > > >>> > >> > > >> > > >> >> > > > > > > > > which I
> > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man page? or
> > can
> > > >>> I use
> > > >>> > >> > output
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > >> "dtcli
> > > >>> > >> > > >> > > >> >> > > --help"?
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call
> version
> > > of
> > > >>> apex
> > > >>> > >> in
> > > >>> > >> > > >> Bigtop
> > > >>> > >> > > >> > as
> > > >>> > >> > > >> > > >> >> 3.3.0 OR
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex
> package
> > of
> > > >>> > bigtop
> > > >>> > >> to
> > > >>> > >> > > >> depend
> > > >>> > >> > > >> > on
> > > >>> > >> > > >> > > >> >> 2.7.1
> > > >>> > >> > > >> > > >> >> > > > > version
> > > >>> > >> > > >> > > >> >> > > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems
> > > that
> > > >>> we
> > > >>> > see
> > > >>> > >> > with
> > > >>> > >> > > >> this
> > > >>> > >> > > >> > > >> >> > dependency?
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > Should work without changed.
> I
> > > >>> thought
> > > >>> > we
> > > >>> > >> > > >> certified
> > > >>> > >> > > >> > > >> >> against
> > > >>> > >> > > >> > > >> >> > > > 2.7.0?
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache
> > > mirror
> > > >>> > from
> > > >>> > >> > which
> > > >>> > >> > > >> > bigtop
> > > >>> > >> > > >> > > >> will
> > > >>> > >> > > >> > > >> >> > pick
> > > >>> > >> > > >> > > >> >> > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > source for compilation and
> > > >>> packaging.
> > > >>> > >> > Please
> > > >>> > >> > > >> > > correct
> > > >>> > >> > > >> > > >> if
> > > >>> > >> > > >> > > >> >> > > > > incorrect:
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > >
> http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
> apache.osuosl.org
> > is
> > > >>> the
> > > >>> > >> mirror
> > > >>> > >> > > >> used
> > > >>> > >> > > >> > by
> > > >>> > >> > > >> > > >> all
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > > > > > softwares
> > > >>> > >> > > >> > > >> >> > > > > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share your opinion.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at
> 11:19
> > > AM,
> > > >>> > >> Chinmay
> > > >>> > >> > > >> > Kolhatkar
> > > >>> > >> > > >> > > <
> > > >>> > >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on
> > the
> > > >>> > >> packaging.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already started
> > > >>> discussion on
> > > >>> > >> > bigtop
> > > >>> > >> > > >> dev
> > > >>> > >> > > >> > > >> mailing
> > > >>> > >> > > >> > > >> >> > list
> > > >>> > >> > > >> > > >> >> > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also
> created a
> > > >>> Jira
> > > >>> > for
> > > >>> > >> the
> > > >>> > >> > > >> same.
> > > >>> > >> > > >> > > For
> > > >>> > >> > > >> > > >> >> this
> > > >>> > >> > > >> > > >> >> > > > > > communities
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is the
> > bigtop
> > > >>> Jira:
> > > >>> > >> > > >> > BIGTOP-2313.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at
> 1:13
> > > AM,
> > > >>> > >> Thomas
> > > >>> > >> > > >> Weise <
> > > >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything
> under
> > > >>> > >> > prerequisites
> > > >>> > >> > > >> that
> > > >>> > >> > > >> > > >> would
> > > >>> > >> > > >> > > >> >> be a
> > > >>> > >> > > >> > > >> >> > > > > > problem.
> > > >>> > >> > > >> > > >> >> > > > > > > We
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
> > > >>> compatibility
> > > >>> > >> as
> > > >>> > >> > > part
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > >> >> becoming
> > > >>> > >> > > >> > > >> >> > an
> > > >>> > >> > > >> > > >> >> > > > > > > incubator
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the
> > > packaging
> > > >>> > during
> > > >>> > >> > the
> > > >>> > >> > > >> next
> > > >>> > >> > > >> > > >> weeks.
> > > >>> > >> > > >> > > >> >> > Since
> > > >>> > >> > > >> > > >> >> > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > work
> > > >>> > >> > > >> > > >> >> > > > > > > > > will
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related
> > > >>> > discussions
> > > >>> > >> and
> > > >>> > >> > > >> JIRAs
> > > >>> > >> > > >> > > >> should
> > > >>> > >> > > >> > > >> >> > also
> > > >>> > >> > > >> > > >> >> > > be
> > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have
> the
> > > >>> > packaging
> > > >>> > >> in
> > > >>> > >> > > >> place
> > > >>> > >> > > >> > by
> > > >>> > >> > > >> > > >> end
> > > >>> > >> > > >> > > >> >> Feb.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at
> > 10:14
> > > >>> AM,
> > > >>> > >> > Chinmay
> > > >>> > >> > > >> > > >> Kolhatkar <
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com
> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work
> on
> > > >>> adding
> > > >>> > >> > Apache
> > > >>> > >> > > >> Apex
> > > >>> > >> > > >> > > as a
> > > >>> > >> > > >> > > >> >> > > component
> > > >>> > >> > > >> > > >> >> > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > > Apache
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the
> packaging
> > > >>> system
> > > >>> > for
> > > >>> > >> > the
> > > >>> > >> > > >> > Apache
> > > >>> > >> > > >> > > >> big
> > > >>> > >> > > >> > > >> >> data
> > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it,
> > > most
> > > >>> > >> recently
> > > >>> > >> > > EMR.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking
> > Jira
> > > >>> task
> > > >>> > in
> > > >>> > >> > > >> APEXCORE
> > > >>> > >> > > >> > for
> > > >>> > >> > > >> > > >> the
> > > >>> > >> > > >> > > >> >> > same:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >>
> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of
> > execution
> > > >>> is as
> > > >>> > >> > > follows:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
> > > >>> prerequisites*
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some
> > > hard
> > > >>> and
> > > >>> > >> soft
> > > >>> > >> > > >> > > expectation
> > > >>> > >> > > >> > > >> >> for
> > > >>> > >> > > >> > > >> >> > new
> > > >>> > >> > > >> > > >> >> > > > > > > > components
> > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into
> Bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of
> it:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> >
> > > >>> > >> > > >>
> > > >>> > >> > >
> > > >>> > >> >
> > > >>> > >>
> > > >>> >
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to
> be
> > > >>> > standard
> > > >>> > >> ASF
> > > >>> > >> > > >> based
> > > >>> > >> > > >> > > >> >> > > requirements,
> > > >>> > >> > > >> > > >> >> > > > > but
> > > >>> > >> > > >> > > >> >> > > > > > > few
> > > >>> > >> > > >> > > >> >> > > > > > > > > need
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects
> are
> > > >>> > expected
> > > >>> > >> to
> > > >>> > >> > be
> > > >>> > >> > > >> > > Licensed
> > > >>> > >> > > >> > > >> >> under
> > > >>> > >> > > >> > > >> >> > > > > Apache
> > > >>> > >> > > >> > > >> >> > > > > > > > > License,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
> > > >>> > >> dependencies
> > > >>> > >> > are
> > > >>> > >> > > >> > > expected
> > > >>> > >> > > >> > > >> >> to be
> > > >>> > >> > > >> > > >> >> > > > > > > compatible
> > > >>> > >> > > >> > > >> >> > > > > > > > > with
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under
> ASL
> > > 2.0
> > > >>> but
> > > >>> > >> need
> > > >>> > >> > to
> > > >>> > >> > > >> > check
> > > >>> > >> > > >> > > if
> > > >>> > >> > > >> > > >> >> > > > > dependencies
> > > >>> > >> > > >> > > >> >> > > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL
> 2.0.
> > > >>> This I
> > > >>> > >> guess
> > > >>> > >> > > >> would
> > > >>> > >> > > >> > > be a
> > > >>> > >> > > >> > > >> >> > > > > verification
> > > >>> > >> > > >> > > >> >> > > > > > > > check.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects
> are
> > > >>> > expected
> > > >>> > >> to
> > > >>> > >> > be
> > > >>> > >> > > >> > > >> compatible
> > > >>> > >> > > >> > > >> >> > with
> > > >>> > >> > > >> > > >> >> > > > all
> > > >>> > >> > > >> > > >> >> > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms
> that
> > > >>> Bigtop
> > > >>> > >> > > >> distribution
> > > >>> > >> > > >> > > is
> > > >>> > >> > > >> > > >> >> > > targeting
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > verifying
> > > >>> > whether
> > > >>> > >> > our
> > > >>> > >> > > >> > > software
> > > >>> > >> > > >> > > >> >> runs
> > > >>> > >> > > >> > > >> >> > > fine
> > > >>> > >> > > >> > > >> >> > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
> > > >>> ubuntu-14.04
> > > >>> > >> > > debian-8
> > > >>> > >> > > >> > > >> >> > opensuse-13.2.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests
> that
> > > >>> should
> > > >>> > be
> > > >>> > >> > > added
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > >> >> > deployment.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the
> test
> > > >>> > artifacts
> > > >>> > >> > which
> > > >>> > >> > > >> goes
> > > >>> > >> > > >> > > >> beyond
> > > >>> > >> > > >> > > >> >> > > smoke
> > > >>> > >> > > >> > > >> >> > > > > test
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are
> > basically
> > > >>> the
> > > >>> > >> > > integration
> > > >>> > >> > > >> > > tests
> > > >>> > >> > > >> > > >> for
> > > >>> > >> > > >> > > >> >> > > > > > verification
> > > >>> > >> > > >> > > >> >> > > > > > > > > after
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a
> > soft
> > > >>> > >> > requirement,
> > > >>> > >> > > >> but
> > > >>> > >> > > >> > > aim
> > > >>> > >> > > >> > > >> is
> > > >>> > >> > > >> > > >> >> to
> > > >>> > >> > > >> > > >> >> > > > > achieve
> > > >>> > >> > > >> > > >> >> > > > > > > this
> > > >>> > >> > > >> > > >> >> > > > > > > > > as
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have
> > > >>> explanation why
> > > >>> > >> not
> > > >>> > >> > to
> > > >>> > >> > > >> > > include.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any from
> > the
> > > >>> link
> > > >>> > >> which
> > > >>> > >> > > >> > > explicitly
> > > >>> > >> > > >> > > >> >> needs
> > > >>> > >> > > >> > > >> >> > to
> > > >>> > >> > > >> > > >> >> > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > > checked
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please
> let
> > > us
> > > >>> > know.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex
> as
> > > >>> > component
> > > >>> > >> to
> > > >>> > >> > > >> Bigtop*
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail
> > > >>> archive of
> > > >>> > >> > Bigtop,
> > > >>> > >> > > >> it
> > > >>> > >> > > >> > was
> > > >>> > >> > > >> > > >> >> learnt
> > > >>> > >> > > >> > > >> >> > > that
> > > >>> > >> > > >> > > >> >> > > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community want to see
> > the
> > > >>> > >> addition of
> > > >>> > >> > > new
> > > >>> > >> > > >> > > >> >> components
> > > >>> > >> > > >> > > >> >> > in
> > > >>> > >> > > >> > > >> >> > > > > > phases.
> > > >>> > >> > > >> > > >> >> > > > > > > > Here
> > > >>> > >> > > >> > > >> >> > > > > > > > > > are
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > creating
> > > of
> > > >>> > >> package
> > > >>> > >> > > i.e.
> > > >>> > >> > > >> > rpm
> > > >>> > >> > > >> > > &
> > > >>> > >> > > >> > > >> deb
> > > >>> > >> > > >> > > >> >> > > files.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > > >>> documentations/READMEs,
> > > >>> > >> > LICENSE,
> > > >>> > >> > > >> > > >> DISCLAMER,
> > > >>> > >> > > >> > > >> >> > NOTES
> > > >>> > >> > > >> > > >> >> > > > etc
> > > >>> > >> > > >> > > >> >> > > > > if
> > > >>> > >> > > >> > > >> >> > > > > > > any
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> documentation
> > > that
> > > >>> > need
> > > >>> > >> to
> > > >>> > >> > be
> > > >>> > >> > > >> > added
> > > >>> > >> > > >> > > to
> > > >>> > >> > > >> > > >> >> > > > > distribution
> > > >>> > >> > > >> > > >> >> > > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > > our
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license
> > > >>> information of
> > > >>> > >> > > >> > dependencies
> > > >>> > >> > > >> > > >> >> required
> > > >>> > >> > > >> > > >> >> > > to
> > > >>> > >> > > >> > > >> >> > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > added
> > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at
> very
> > > >>> least)
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke
> test
> > > for
> > > >>> > >> > packaging.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for
> > > >>> automatic
> > > >>> > >> > > >> deployment
> > > >>> > >> > > >> > and
> > > >>> > >> > > >> > > >> >> > > > configuration
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet
> recipes
> > > for
> > > >>> > >> > automatic
> > > >>> > >> > > >> > > >> deployment
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For verification
> > of
> > > >>> > >> > deployments.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat'
> to
> > > make
> > > >>> > sure
> > > >>> > >> all
> > > >>> > >> > > new
> > > >>> > >> > > >> > > stuff
> > > >>> > >> > > >> > > >> is
> > > >>> > >> > > >> > > >> >> > > > compliant
> > > >>> > >> > > >> > > >> >> > > > > > > with
> > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license requirements.
> If
> > > >>> you add
> > > >>> > >> code
> > > >>> > >> > > >> > licenses
> > > >>> > >> > > >> > > >> >> under
> > > >>> > >> > > >> > > >> >> > > > > different
> > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need to be
> > > >>> listed in
> > > >>> > >> the
> > > >>> > >> > > >> NOTICE.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your
> > thoughts
> > > >>> on
> > > >>> > the
> > > >>> > >> > > >> approach.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
> > corresponding
> > > >>> > >> > communication
> > > >>> > >> > > >> on
> > > >>> > >> > > >> > > >> bigtop
> > > >>> > >> > > >> > > >> >> > > mailing
> > > >>> > >> > > >> > > >> >> > > > > list
> > > >>> > >> > > >> > > >> >> > > > > > > as
> > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some specific
> > > >>> > >> > > >> questions/suggestions
> > > >>> > >> > > >> > > >> >> related to
> > > >>> > >> > > >> > > >> >> > > > what
> > > >>> > >> > > >> > > >> >> > > > > > > should
> > > >>> > >> > > >> > > >> >> > > > > > > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the package
> > and
> > > >>> what
> > > >>> > >> > should
> > > >>> > >> > > be
> > > >>> > >> > > >> > the
> > > >>> > >> > > >> > > >> smoke
> > > >>> > >> > > >> > > >> >> > > tests,
> > > >>> > >> > > >> > > >> >> > > > > but
> > > >>> > >> > > >> > > >> >> > > > > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not having
> > too
> > > >>> much
> > > >>> > >> > content
> > > >>> > >> > > >> > here,
> > > >>> > >> > > >> > > >> we'll
> > > >>> > >> > > >> > > >> >> > put
> > > >>> > >> > > >> > > >> >> > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > questions
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this
> > > >>> > mailthread.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> >
> > > >>> > >> > > >>
> > > >>> > >> > > >
> > > >>> > >> > > >
> > > >>> > >> > >
> > > >>> > >> >
> > > >>> > >>
> > > >>> > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Thanks Thomas.

I've created pull request to bigtop for addition of smoke tests:
https://github.com/apache/bigtop/pull/94


On Fri, Mar 4, 2016 at 12:28 AM, Thomas Weise <th...@datatorrent.com>
wrote:

> Not only is there no objection, here is an enthusiastic +1
>
> On Thu, Mar 3, 2016 at 10:47 AM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Hello Everyone,
> >
> > I believe there is no object on the smoke test to be added to bigtop for
> > apex package.
> > I've created smoke tests for bigtop as per above.
> >
> > If there no further comments, I'll create a pull request for the same.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Fri, Feb 26, 2016 at 2:36 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Bigtop has consumed first patch for building apex packages into it's
> > > master branch.
> > > Commit mail:
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-commits/201602.mbox/%3C06c0ca62ac1b483d8ded090f0f2c8d73%40git.apache.org%3E
> > >
> > > What this means is bigtop master will be able to generate apex packages
> > > (3.3.0-incubating), both deb & rpm, when it build.
> > >
> > > For the second phase, we need to add smoke tests for our package in
> > bigtop
> > > repo.
> > > This is an important steps for bigtop to host the generate package in
> > > bigtop repository during its 1.2.0 release.
> > >
> > > Here are my initial thoughts about smoke test:
> > > TEST::
> > > 1. Create mvn app with archetype
> > > 2. Build app
> > > 3. Run app with apex/dtcli
> > >
> > > VERIFICATION:
> > > 1. stdout file gets has random numbers per line.
> > >
> > > I see app created by archetype is randomNumberGenerator -> Console.
> > > I think this should be a good enough smoke test for apex package.
> > >
> > > Of course I need to check if we can do above with bigtop framework.
> But I
> > > have a hunch that we can, because there are standalone frameworks like
> us
> > > which has similar types of tests.
> > >
> > > Please share your opinion and thoughts on this.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > >
> > > On Fri, Feb 19, 2016 at 11:38 AM, Thomas Weise <thomas@datatorrent.com
> >
> > > wrote:
> > >
> > >> We should not package any Hadoop dependencies, they are provided by
> the
> > >> Bigtop stack.
> > >>
> > >>
> > >> On Thu, Feb 18, 2016 at 8:56 PM, Tushar Gosavi <
> tushar@datatorrent.com>
> > >> wrote:
> > >>
> > >> > I just did a quick install of bigtop 1.1.0 on ubuntu and observed
> that
> > >> >  jersey-client-1.9.jar is already packaged in hadoop
> > >> > /usr/lib/hadoop/client/jersey-client.jar, so I guess it will be ok
> to
> > >> > include those in apex lib too. or If we are including hadoop jars
> > >> > in classpath while launching the apex application then it may not be
> > >> > needed.
> > >> >
> > >> > - Tushar.
> > >> >
> > >> >
> > >> >
> > >> > On Fri, Feb 19, 2016 at 8:44 AM, Chinmay Kolhatkar <
> > chinmay@apache.org>
> > >> > wrote:
> > >> >
> > >> > > No worries David. I've updated the manpage.
> > >> > >
> > >> > > Just an update: apex bigtop packaging for rpm+deb is done. Here is
> > the
> > >> > > branch on which I'm working if someone wants to look at:
> > >> > > https://github.com/chinmaykolhatkar/bigtop/blob/BIGTOP-2314
> > >> > >
> > >> > > Only the 3rd questions below is the question that is left to be
> > >> answered.
> > >> > > Once this is resolved, I'll create the pull request in bigtop.
> > >> > >
> > >> > > Following questions are related to licenses and inclusion of
> certain
> > >> > files
> > >> > > in apex rpm+deb:
> > >> > > 1. I'll be included LICENSE, DISCLAIMER, NOTICE from top folder of
> > >> core
> > >> > > repo in our rpm and deb package. Those will be present in standard
> > >> > location
> > >> > > i.e. /usr/lib/apex/
> > >> > > Please let me know otherwise.
> > >> > >
> > >> > > 2. Most of our runtime dependencies has ASF license, but certain
> > >> > libraries
> > >> > > has MIT and BSD licenses, hence I've included those licenses too
> in
> > >> rpm &
> > >> > > deb package in same location as above.
> > >> > > Please let me know otherwise.
> > >> > >
> > >> > > BSD - jline-2.11.jar, kryo-2.24.0.jar, minlog-1.2.jar
> > >> > > MIT - mbassador-1.1.9.jar
> > >> > >
> > >> > > 3. There are 2 jars (jersey-apache-client4-1.9.jar,
> > >> > jersey-client-1.9.jar)
> > >> > > which has duel license of CDDL + GPLv2 with classpath exception:
> > >> > > https://jersey.java.net/license.html
> > >> > > I'm not an expert here. Hence can someone let me know whether
> these
> > >> > > libraries is OK to be packaged in rpm+deb of apex?
> > >> > >
> > >> > > If yes, should we also include that license in our package?
> > >> > >
> > >> > > Thanks,
> > >> > > Chinmay.
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Thu, Feb 18, 2016 at 11:35 PM, David Yan <
> david@datatorrent.com>
> > >> > wrote:
> > >> > >
> > >> > > > I should've mentioned that I had a pull request that was merged
> > >> > yesterday
> > >> > > > that corrected the statement:
> > >> > > >
> > >> > > > https://github.com/DataTorrent/docs/pull/45
> > >> > > >
> > >> > > > On Thu, Feb 18, 2016 at 12:07 AM, Tushar Gosavi <
> > >> > tushar@datatorrent.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > In following statement, can you replace dtGateway with
> > >> > > ApplicationMaster
> > >> > > > > and Yarn. or may be remove this statement entirely.
> > >> > > > > apex is a wrapper around the REST API provided by dtGatway,
> > >> > > > >
> > >> > > > > - Tushar.
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <
> > >> > > chinmay@apache.org>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi,
> > >> > > > > >
> > >> > > > > > I've attached man page for apex in Jira. Here is the link to
> > it:
> > >> > > > > >
> > >> https://issues.apache.org/jira/secure/attachment/12788403/apex.1
> > >> > > > > >
> > >> > > > > > Please view it by: man -l apex.1
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Chinmay.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <
> > >> > > > tushar@datatorrent.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Chinmay,
> > >> > > > > > >
> > >> > > > > > > I am not able to find the attachment, may be attachments
> are
> > >> not
> > >> > > > > allowed
> > >> > > > > > in
> > >> > > > > > > the mailing list. Can you attach the man page to the Jira?
> > >> > > > > > >
> > >> > > > > > > - Tushar.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <
> > >> > > > chinmay@apache.org
> > >> > > > > >
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi All,
> > >> > > > > > > >
> > >> > > > > > > > I've attached man page file for the purpose of bigtop
> > >> > > integration.
> > >> > > > > > > > Please have a look and let me know if there any any
> > feedback
> > >> > > about
> > >> > > > > it.
> > >> > > > > > > >
> > >> > > > > > > > You can see the man page content of this file with:
> > >> > > > > > > > man -l apex.1
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > > Chinmay.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> > >> > > > > > chinmay@apache.org>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > >> Thanks. Will follow bigtop recommendation. i.e. remove
> > >> > > > > "-incubating".
> > >> > > > > > > >>
> > >> > > > > > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
> > >> > > > > thomas@datatorrent.com
> > >> > > > > > >
> > >> > > > > > > >> wrote:
> > >> > > > > > > >>
> > >> > > > > > > >>> Please follow the recommendation and continue
> discussion
> > >> on
> > >> > the
> > >> > > > > > Bigtop
> > >> > > > > > > >>> list.
> > >> > > > > > > >>>
> > >> > > > > > > >>> --
> > >> > > > > > > >>> sent from mobile
> > >> > > > > > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <
> > >> > > chinmay@apache.org
> > >> > > > >
> > >> > > > > > > wrote:
> > >> > > > > > > >>>
> > >> > > > > > > >>> > I got a recommendation from bigtop community to drop
> > >> > > > > "-incubating"
> > >> > > > > > > from
> > >> > > > > > > >>> > version.
> > >> > > > > > > >>> >
> > >> > > > > > > >>> > Here is the mail thread:
> > >> > > > > > > >>> >
> > >> > > > > > > >>> >
> > >> > > > > > > >>>
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> > >> > > > > > > >>> >
> > >> > > > > > > >>> > Please let me know if removing "-incubating" is
> fine.
> > >> > > > > > > >>> >
> > >> > > > > > > >>> > Thanks,
> > >> > > > > > > >>> > Chinmay.
> > >> > > > > > > >>> >
> > >> > > > > > > >>> >
> > >> > > > > > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> > >> > > > > > > chinmay@apache.org
> > >> > > > > > > >>> >
> > >> > > > > > > >>> > wrote:
> > >> > > > > > > >>> >
> > >> > > > > > > >>> > > I've already started communication there.
> > >> > > > > > > >>> > >
> > >> > > > > > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> > >> > > > > > > >>> thomas@datatorrent.com>
> > >> > > > > > > >>> > > wrote:
> > >> > > > > > > >>> > >
> > >> > > > > > > >>> > >> This is a question you should ask on the Bigtop
> > >> mailing
> > >> > > > list.
> > >> > > > > > > >>> > >>
> > >> > > > > > > >>> > >> --
> > >> > > > > > > >>> > >> sent from mobile
> > >> > > > > > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <
> > >> > > > prad@apache.org>
> > >> > > > > > > >>> wrote:
> > >> > > > > > > >>> > >>
> > >> > > > > > > >>> > >> > Yes. In case of RPM, the only restriction on
> > >> version
> > >> > is
> > >> > > > that
> > >> > > > > > it
> > >> > > > > > > >>> can
> > >> > > > > > > >>> > not
> > >> > > > > > > >>> > >> > contain '-'.
> > >> > > > > > > >>> > >> > However, as a workaround, one can use
> > 'incubating'
> > >> as
> > >> > a
> > >> > > > part
> > >> > > > > > of
> > >> > > > > > > >>> > >> %{release}.
> > >> > > > > > > >>> > >> >
> > >> > > > > > > >>> > >> > Version: 3.3.0
> > >> > > > > > > >>> > >> > Release: incubating
> > >> > > > > > > >>> > >> >
> > >> > > > > > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> > >> > > > > > > >>> > >> >
> > >> > > > > > > >>> > >> > It is the same practice followed for
> pre-release
> > >> > > packages
> > >> > > > > e.g.
> > >> > > > > > > >>> with
> > >> > > > > > > >>> > rcX,
> > >> > > > > > > >>> > >> > alphaX, betaX being used in %{release}.
> > >> > > > > > > >>> > >> >
> > >> > > > > > > >>> > >> > Regards,
> > >> > > > > > > >>> > >> > Pradeep A. Dalvi
> > >> > > > > > > >>> > >> >
> > >> > > > > > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay
> > Kolhatkar
> > >> <
> > >> > > > > > > >>> > chinmay@apache.org>
> > >> > > > > > > >>> > >> > wrote:
> > >> > > > > > > >>> > >> >
> > >> > > > > > > >>> > >> > > Hi Everyone,
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> > > In one of the previous mails in this thread,
> we
> > >> > > decided
> > >> > > > > that
> > >> > > > > > > >>> version
> > >> > > > > > > >>> > >> of
> > >> > > > > > > >>> > >> > the
> > >> > > > > > > >>> > >> > > package will be "3.3.0-incubating".
> > >> > > > > > > >>> > >> > > The problem that I'm facing is in RPM base
> > >> version
> > >> > "-"
> > >> > > > is
> > >> > > > > > not
> > >> > > > > > > >>> > allowed.
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> > > Can we have package version to "3.3.0"?
> > >> > > > > > > >>> > >> > > i.e. the packages created will be as follows:
> > >> > > > > > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > >> > > > > > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> > > There is another integration for Zepplin in
> > >> Bigtop
> > >> > > which
> > >> > > > > is
> > >> > > > > > > >>> > incubating
> > >> > > > > > > >>> > >> > and
> > >> > > > > > > >>> > >> > > is having base version as "0.5.6".
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> > > Please let me know your thoughts.
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> > > Thanks,
> > >> > > > > > > >>> > >> > > Chinmay.
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay
> > >> Kolhatkar
> > >> > <
> > >> > > > > > > >>> > >> chinmay@apache.org>
> > >> > > > > > > >>> > >> > > wrote:
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> > > > Thanks everyone for the feedback.
> > >> > > > > > > >>> > >> > > >
> > >> > > > > > > >>> > >> > > > Also as Ram and David mentioned, I'll start
> > >> > creating
> > >> > > > the
> > >> > > > > > man
> > >> > > > > > > >>> page
> > >> > > > > > > >>> > as
> > >> > > > > > > >>> > >> > > well.
> > >> > > > > > > >>> > >> > > >
> > >> > > > > > > >>> > >> > > > Thanks,
> > >> > > > > > > >>> > >> > > > Chinmay.
> > >> > > > > > > >>> > >> > > >
> > >> > > > > > > >>> > >> > > >
> > >> > > > > > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas
> > Weise <
> > >> > > > > > > >>> > >> thomas@datatorrent.com>
> > >> > > > > > > >>> > >> > > > wrote:
> > >> > > > > > > >>> > >> > > >
> > >> > > > > > > >>> > >> > > >> I think we already agreed that we will
> > retain
> > >> the
> > >> > > > > current
> > >> > > > > > > >>> script
> > >> > > > > > > >>> > >> name
> > >> > > > > > > >>> > >> > in
> > >> > > > > > > >>> > >> > > >> this bigtop iteration. There is a ticket
> for
> > >> > > > renaming,
> > >> > > > > > add
> > >> > > > > > > >>> you
> > >> > > > > > > >>> > >> > > suggestions
> > >> > > > > > > >>> > >> > > >> there.
> > >> > > > > > > >>> > >> > > >>
> > >> > > > > > > >>> > >> > > >> --
> > >> > > > > > > >>> > >> > > >> sent from mobile
> > >> > > > > > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > >> > > > > > > >>> > sandesh@datatorrent.com>
> > >> > > > > > > >>> > >> > > wrote:
> > >> > > > > > > >>> > >> > > >>
> > >> > > > > > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or
> start
> > >> the
> > >> > > > > > transition
> > >> > > > > > > >>> > >> process by
> > >> > > > > > > >>> > >> > > >> making
> > >> > > > > > > >>> > >> > > >> > dtcli as the symlink to apex cli.
> > >> > > > > > > >>> > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > Renaming is a breaking change should be
> > done
> > >> > > during
> > >> > > > > > major
> > >> > > > > > > >>> > version
> > >> > > > > > > >>> > >> > > number
> > >> > > > > > > >>> > >> > > >> > change.
> > >> > > > > > > >>> > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David
> Yan
> > <
> > >> > > > > > > >>> > david@datatorrent.com
> > >> > > > > > > >>> > >> >
> > >> > > > > > > >>> > >> > > >> wrote:
> > >> > > > > > > >>> > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > > Apologies for hitting the "Send"
> button
> > >> too
> > >> > > > early.
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > What is the scope of the name change?
> > If
> > >> we
> > >> > > > simply
> > >> > > > > > > have
> > >> > > > > > > >>> a
> > >> > > > > > > >>> > >> symlink
> > >> > > > > > > >>> > >> > > >> from
> > >> > > > > > > >>> > >> > > >> > > apex to dtcli, then it should have no
> > >> > backward
> > >> > > > > > > >>> compatibility
> > >> > > > > > > >>> > >> > problem
> > >> > > > > > > >>> > >> > > >> at
> > >> > > > > > > >>> > >> > > >> > > all.
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > But the proper name change would
> > involve:
> > >> > > > > > > >>> > >> > > >> > > - Change the name of DTCli class to
> > >> ApexCli
> > >> > > > > > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> > >> > > > > > > >>> > >> > > >> > > - Change the initial greeting when
> > running
> > >> > the
> > >> > > > CLI
> > >> > > > > > from
> > >> > > > > > > >>> > >> something
> > >> > > > > > > >>> > >> > > like
> > >> > > > > > > >>> > >> > > >> > "DT
> > >> > > > > > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI
> > >> ...".
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > The above changes may break existing
> > >> scripts
> > >> > > that
> > >> > > > > > rely
> > >> > > > > > > >>> on the
> > >> > > > > > > >>> > >> the
> > >> > > > > > > >>> > >> > > >> string
> > >> > > > > > > >>> > >> > > >> > > "DT".
> > >> > > > > > > >>> > >> > > >> > > But if folks are okay with this
> > potential
> > >> > > > problem,
> > >> > > > > I
> > >> > > > > > am
> > >> > > > > > > >>> okay
> > >> > > > > > > >>> > >> with
> > >> > > > > > > >>> > >> > > the
> > >> > > > > > > >>> > >> > > >> > name
> > >> > > > > > > >>> > >> > > >> > > change.
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > For the man page, we can start with
> the
> > >> > content
> > >> > > > of
> > >> > > > > > this
> > >> > > > > > > >>> page:
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > Plus the command line options when
> doing
> > >> > dtcli
> > >> > > > -h:
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > usage: DTCli
> > >> > > > > > > >>> > >> > > >> > >  -e <arg>    Commands are read from
> the
> > >> > > argument
> > >> > > > > > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt
> at
> > >> all
> > >> > > time
> > >> > > > > > > >>> > >> > > >> > >  -h          Print this help
> > >> > > > > > > >>> > >> > > >> > >  -kp <arg>   Use the specified
> kerberos
> > >> > > principal
> > >> > > > > > > >>> > >> > > >> > >  -kt <arg>   Use the specified
> kerberos
> > >> > keytab
> > >> > > > > > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> > >> > > > > > > >>> > >> > > >> > >  -r          JSON Raw mode
> > >> > > > > > > >>> > >> > > >> > >  -v          Verbose mode level 1
> > >> > > > > > > >>> > >> > > >> > >  -vv         Verbose mode level 2
> > >> > > > > > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> > >> > > > > > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > David
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David
> > >> Yan <
> > >> > > > > > > >>> > >> david@datatorrent.com
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> > > >> > wrote:
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > > What is the scope of the name
> change?
> > >> If
> > >> > we
> > >> > > > > simply
> > >> > > > > > > >>> have a
> > >> > > > > > > >>> > >> > symlink
> > >> > > > > > > >>> > >> > > >> from
> > >> > > > > > > >>> > >> > > >> > > > apex to dtcli, then it should have
> no
> > >> > > backward
> > >> > > > > > > >>> > compatibility
> > >> > > > > > > >>> > >> > > >> problem at
> > >> > > > > > > >>> > >> > > >> > > all.
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > > > But the proper name change would
> > >> involve:
> > >> > > > > > > >>> > >> > > >> > > > - Change the name of DTCli class to
> > >> ApexCli
> > >> > > > > > > >>> > >> > > >> > > > - Change the prompt from dt> to
> apex>
> > >> > > > > > > >>> > >> > > >> > > > - Change the initial greeting when
> > >> running
> > >> > > the
> > >> > > > > CLI
> > >> > > > > > > from
> > >> > > > > > > >>> > >> > something
> > >> > > > > > > >>> > >> > > >> like
> > >> > > > > > > >>> > >> > > >> > > "DT
> > >> > > > > > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex
> CLI
> > >> > ...".
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > > > The above changes may break existing
> > >> > scripts
> > >> > > > that
> > >> > > > > > > rely
> > >> > > > > > > >>> on
> > >> > > > > > > >>> > the
> > >> > > > > > > >>> > >> > the
> > >> > > > > > > >>> > >> > > >> > string
> > >> > > > > > > >>> > >> > > >> > > > "DT".
> > >> > > > > > > >>> > >> > > >> > > > But if folks are okay with this
> > >> potential
> > >> > > > > problem,
> > >> > > > > > I
> > >> > > > > > > am
> > >> > > > > > > >>> > okay
> > >> > > > > > > >>> > >> > with
> > >> > > > > > > >>> > >> > > >> the
> > >> > > > > > > >>> > >> > > >> > > name
> > >> > > > > > > >>> > >> > > >> > > > change.
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > > > For the man page, we can start with
> > the
> > >> > > content
> > >> > > > > of
> > >> > > > > > > this
> > >> > > > > > > >>> > page:
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM,
> > Chinmay
> > >> > > > > Kolhatkar
> > >> > > > > > <
> > >> > > > > > > >>> > >> > > >> chinmay@apache.org
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> > > > wrote:
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> Here is a suggestion I have related
> > to
> > >> > name
> > >> > > > > > change:
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > > >>> > >> > > >> > > >> 1. While packaging we change the
> name
> > >> of
> > >> > > > "dtcli"
> > >> > > > > > > >>> script to
> > >> > > > > > > >>> > >> > > "apex".
> > >> > > > > > > >>> > >> > > >> > This
> > >> > > > > > > >>> > >> > > >> > > is
> > >> > > > > > > >>> > >> > > >> > > >> done so that in future when we
> change
> > >> the
> > >> > > name
> > >> > > > > to
> > >> > > > > > > >>> "apex",
> > >> > > > > > > >>> > >> users
> > >> > > > > > > >>> > >> > > of
> > >> > > > > > > >>> > >> > > >> > > bigtop
> > >> > > > > > > >>> > >> > > >> > > >> apex don't have to transition much.
> > >> > > > > > > >>> > >> > > >> > > >> 2. We also keep a symlink named
> dtcli
> > >> > which
> > >> > > > > points
> > >> > > > > > > to
> > >> > > > > > > >>> apex
> > >> > > > > > > >>> > >> > > script.
> > >> > > > > > > >>> > >> > > >> > This
> > >> > > > > > > >>> > >> > > >> > > is
> > >> > > > > > > >>> > >> > > >> > > >> for backward compatibility.
> > >> > > > > > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for
> > >> first
> > >> > > > > > > integration
> > >> > > > > > > >>> in
> > >> > > > > > > >>> > >> > bigtop.
> > >> > > > > > > >>> > >> > > >> This
> > >> > > > > > > >>> > >> > > >> > is
> > >> > > > > > > >>> > >> > > >> > > >> in
> > >> > > > > > > >>> > >> > > >> > > >> the interest of not changing the
> > >> content
> > >> > of
> > >> > > > > dtcli
> > >> > > > > > or
> > >> > > > > > > >>> apex
> > >> > > > > > > >>> > >> > script.
> > >> > > > > > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of
> all
> > >> > these
> > >> > > > > naming
> > >> > > > > > > >>> related
> > >> > > > > > > >>> > >> > > changes
> > >> > > > > > > >>> > >> > > >> and
> > >> > > > > > > >>> > >> > > >> > > >> update bigtop repository later to
> > >> remove
> > >> > > > > > references
> > >> > > > > > > to
> > >> > > > > > > >>> > dtcli
> > >> > > > > > > >>> > >> > all
> > >> > > > > > > >>> > >> > > >> > > together.
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > > >>> > >> > > >> > > >> Please share your thoughts on above
> > >> > > approach.
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > > >>> > >> > > >> > > >> Also, please share what could be
> the
> > >> man
> > >> > > page
> > >> > > > > > > content
> > >> > > > > > > >>> for
> > >> > > > > > > >>> > >> > "apex".
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > > >>> > >> > > >> > > >> Thanks,
> > >> > > > > > > >>> > >> > > >> > > >> Chinmay.
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM,
> > >> Chinmay
> > >> > > > > > Kolhatkar <
> > >> > > > > > > >>> > >> > > >> > chinmay@apache.org>
> > >> > > > > > > >>> > >> > > >> > > >> wrote:
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > > >>> > >> > > >> > > >> > Yes.. It'll be build from source
> > tar
> > >> > > > > downloaded
> > >> > > > > > > >>> from one
> > >> > > > > > > >>> > >> of
> > >> > > > > > > >>> > >> > the
> > >> > > > > > > >>> > >> > > >> > apache
> > >> > > > > > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> > >> > > > > > > >>> > >> > > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's
> > >> sourced
> > >> > > > from
> > >> > > > > > > dtcli.
> > >> > > > > > > >>> > This
> > >> > > > > > > >>> > >> > > leaves
> > >> > > > > > > >>> > >> > > >> us
> > >> > > > > > > >>> > >> > > >> > 3
> > >> > > > > > > >>> > >> > > >> > > >> > options:
> > >> > > > > > > >>> > >> > > >> > > >> > If we should not have any
> > dt-env.sh,
> > >> > dtcli
> > >> > > > > would
> > >> > > > > > > >>> need a
> > >> > > > > > > >>> > >> > change
> > >> > > > > > > >>> > >> > > >> while
> > >> > > > > > > >>> > >> > > >> > > >> > packaging.
> > >> > > > > > > >>> > >> > > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> > But if we want to use dtcli as it
> > is,
> > >> > then
> > >> > > > we
> > >> > > > > > > would
> > >> > > > > > > >>> need
> > >> > > > > > > >>> > >> > > >> dt-env.sh
> > >> > > > > > > >>> > >> > > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM,
> > >> Thomas
> > >> > > > Weise
> > >> > > > > <
> > >> > > > > > > >>> > >> > > >> > thomas@datatorrent.com
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> Looks good overall, though there
> > >> > > shouldn't
> > >> > > > be
> > >> > > > > > any
> > >> > > > > > > >>> > >> dt-env.sh
> > >> > > > > > > >>> > >> > > >> > > >> >>
> > >> > > > > > > >>> > >> > > >> > > >> >> What will the bigtop package be
> > >> built
> > >> > > from,
> > >> > > > > the
> > >> > > > > > > >>> source
> > >> > > > > > > >>> > >> tar?
> > >> > > > > > > >>> > >> > > >> > > >> >>
> > >> > > > > > > >>> > >> > > >> > > >> >>
> > >> > > > > > > >>> > >> > > >> > > >> >>
> > >> > > > > > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM,
> > >> > Chinmay
> > >> > > > > > > Kolhatkar
> > >> > > > > > > >>> <
> > >> > > > > > > >>> > >> > > >> > > chinmay@apache.org
> > >> > > > > > > >>> > >> > > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >>
> > >> > > > > > > >>> > >> > > >> > > >> >> > Hi Thomas,
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > First, we're not changing the
> > name
> > >> > > > > anywhere.
> > >> > > > > > > >>> We'll
> > >> > > > > > > >>> > >> follow
> > >> > > > > > > >>> > >> > > what
> > >> > > > > > > >>> > >> > > >> > > >> >> currently is
> > >> > > > > > > >>> > >> > > >> > > >> >> > in source tarball for
> > >> > 3.3.0-incubating
> > >> > > > > > version.
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a
> > >> directory
> > >> > > > > > structure
> > >> > > > > > > >>> below
> > >> > > > > > > >>> > >> which
> > >> > > > > > > >>> > >> > > is
> > >> > > > > > > >>> > >> > > >> > > inline
> > >> > > > > > > >>> > >> > > >> > > >> >> with
> > >> > > > > > > >>> > >> > > >> > > >> >> > other existing integrations
> with
> > >> > > bigtop.
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > The need for each file is as
> > >> follows:
> > >> > > > > > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex
> > >> picked
> > >> > > from
> > >> > > > > > > >>> > >> > > >> engine/src/main/scripts/
> > >> > > > > > > >>> > >> > > >> > > of
> > >> > > > > > > >>> > >> > > >> > > >> >> source
> > >> > > > > > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is
> > >> source
> > >> > > from
> > >> > > > > > dtcli.
> > >> > > > > > > >>> This
> > >> > > > > > > >>> > >> > > searches
> > >> > > > > > > >>> > >> > > >> for
> > >> > > > > > > >>> > >> > > >> > > >> hadoop
> > >> > > > > > > >>> > >> > > >> > > >> >> > binary path and exports an env
> > >> > variable
> > >> > > > for
> > >> > > > > > > >>> dtcli to
> > >> > > > > > > >>> > >> use
> > >> > > > > > > >>> > >> > it.
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI.
> > This
> > >> > list
> > >> > > > is
> > >> > > > > > > >>> extracted
> > >> > > > > > > >>> > >> in a
> > >> > > > > > > >>> > >> > > >> similar
> > >> > > > > > > >>> > >> > > >> > > >> way to
> > >> > > > > > > >>> > >> > > >> > > >> >> > how DT Community Edition finds
> > >> > > dependency
> > >> > > > > > jars.
> > >> > > > > > > >>> > >> > > >> > > >> >> > Please note that this is the
> > first
> > >> > > > > iteration
> > >> > > > > > > >>> list of
> > >> > > > > > > >>> > >> jars.
> > >> > > > > > > >>> > >> > > I'm
> > >> > > > > > > >>> > >> > > >> > > >> trying to
> > >> > > > > > > >>> > >> > > >> > > >> >> > narrow this down to only those
> > >> which
> > >> > > are
> > >> > > > > > really
> > >> > > > > > > >>> > >> required.
> > >> > > > > > > >>> > >> > > >> > > >> >> > The test that I'm running to
> > >> check if
> > >> > > > dtcli
> > >> > > > > > > runs
> > >> > > > > > > >>> fine
> > >> > > > > > > >>> > >> with
> > >> > > > > > > >>> > >> > > >> given
> > >> > > > > > > >>> > >> > > >> > > >> >> dependency
> > >> > > > > > > >>> > >> > > >> > > >> >> > is to
> > launch+shutdown+kill+status
> > >> for
> > >> > > pi
> > >> > > > > > demo.
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > Above is the directory
> structure
> > >> > which
> > >> > > is
> > >> > > > > > > >>> required
> > >> > > > > > > >>> > for
> > >> > > > > > > >>> > >> CLI
> > >> > > > > > > >>> > >> > > to
> > >> > > > > > > >>> > >> > > >> > work.
> > >> > > > > > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has
> > >> > > mentioned
> > >> > > > > are
> > >> > > > > > > some
> > >> > > > > > > >>> > >> > additional
> > >> > > > > > > >>> > >> > > >> > files
> > >> > > > > > > >>> > >> > > >> > > >> which
> > >> > > > > > > >>> > >> > > >> > > >> >> > makes the like of
> administrator
> > >> > easier.
> > >> > > > For
> > >> > > > > > eg.
> > >> > > > > > > >>> > >> /etc/skel
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > Please let me know if this is
> > >> inline
> > >> > > with
> > >> > > > > > what
> > >> > > > > > > >>> you're
> > >> > > > > > > >>> > >> > > >> thinking.
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > Thanks,
> > >> > > > > > > >>> > >> > > >> > > >> >> > Chinmay.
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > .
> > >> > > > > > > >>> > >> > > >> > > >> >> > |-- bin
> > >> > > > > > > >>> > >> > > >> > > >> >> > |   `-- dtcli
> > >> > > > > > > >>> > >> > > >> > > >> >> > |-- conf
> > >> > > > > > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > >> > > > > > > >>> > >> > > >> > > >> >> > `-- lib
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> apex-api-3.3.0-incubating.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> > > > > > apex-bufferserver-3.3.0-incubating.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> > > apex-common-3.3.0-incubating.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> async-http-client-1.7.20.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> commons-beanutils-1.8.3.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > commons-logging-1.1.3.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> > grizzly-http-servlet-2.1.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> hadoop-common-2.2.0-tests.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > jackson-core-asl-1.9.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> jackson-mapper-asl-1.9.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> javax.servlet-api-3.0.1.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> jersey-apache-client4-1.9.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> > jetty-http-8.1.10.v20130312.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> jetty-io-8.1.10.v20130312.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> > jetty-util-8.1.10.v20130312.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> > > > > jetty-websocket-8.1.10.v20130312.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > >> > validation-api-1.1.0.Final.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > xbean-asm5-shaded-4.3.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24
> AM,
> > >> > Thomas
> > >> > > > > > Weise <
> > >> > > > > > > >>> > >> > > >> > > >> thomas@datatorrent.com>
> > >> > > > > > > >>> > >> > > >> > > >> >> > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > Chinmay,
> > >> > > > > > > >>> > >> > > >> > > >> >> > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > Before discussing where to
> put
> > >> the
> > >> > > > files,
> > >> > > > > > > let's
> > >> > > > > > > >>> > make
> > >> > > > > > > >>> > >> > sure
> > >> > > > > > > >>> > >> > > >> they
> > >> > > > > > > >>> > >> > > >> > > are
> > >> > > > > > > >>> > >> > > >> > > >> >> really
> > >> > > > > > > >>> > >> > > >> > > >> >> > > needed for the operation of
> > the
> > >> > CLI.
> > >> > > As
> > >> > > > > for
> > >> > > > > > > >>> names,
> > >> > > > > > > >>> > >> > > anything
> > >> > > > > > > >>> > >> > > >> new
> > >> > > > > > > >>> > >> > > >> > > >> needs
> > >> > > > > > > >>> > >> > > >> > > >> >> to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and
> > >> should
> > >> > > > > follow
> > >> > > > > > > >>> common
> > >> > > > > > > >>> > >> > > >> conventions,
> > >> > > > > > > >>> > >> > > >> > > >> >> especially
> > >> > > > > > > >>> > >> > > >> > > >> >> > > in Bigtop where there are
> many
> > >> > > existing
> > >> > > > > > > >>> > integrations
> > >> > > > > > > >>> > >> to
> > >> > > > > > > >>> > >> > > look
> > >> > > > > > > >>> > >> > > >> > at.
> > >> > > > > > > >>> > >> > > >> > > >> >> > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > Thanks,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > Thomas
> > >> > > > > > > >>> > >> > > >> > > >> >> > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07
> > AM,
> > >> > > > Chinmay
> > >> > > > > > > >>> Kolhatkar
> > >> > > > > > > >>> > <
> > >> > > > > > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep
> > and
> > >> > > > > Aniruddha.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need
> > to
> > >> > > change
> > >> > > > > the
> > >> > > > > > > >>> dtcli
> > >> > > > > > > >>> > >> > > >> considering it
> > >> > > > > > > >>> > >> > > >> > > >> works
> > >> > > > > > > >>> > >> > > >> > > >> >> > with
> > >> > > > > > > >>> > >> > > >> > > >> >> > > DT
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > Community edition. We can
> > keep
> > >> > the
> > >> > > > > > > directory
> > >> > > > > > > >>> > >> structure
> > >> > > > > > > >>> > >> > > >> > similar
> > >> > > > > > > >>> > >> > > >> > > to
> > >> > > > > > > >>> > >> > > >> > > >> >> that.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables
> > which
> > >> > > > contains
> > >> > > > > > > >>> > information
> > >> > > > > > > >>> > >> > > >> required
> > >> > > > > > > >>> > >> > > >> > for
> > >> > > > > > > >>> > >> > > >> > > >> >> dtcli
> > >> > > > > > > >>> > >> > > >> > > >> >> > to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > launch.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > 2. Let me check with
> bigtop
> > >> > > community
> > >> > > > > > that
> > >> > > > > > > >>> > whether
> > >> > > > > > > >>> > >> > they
> > >> > > > > > > >>> > >> > > >> > > >> facilitate
> > >> > > > > > > >>> > >> > > >> > > >> >> the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs
> > >> before
> > >> > > user
> > >> > > > > is
> > >> > > > > > > >>> created.
> > >> > > > > > > >>> > >> In
> > >> > > > > > > >>> > >> > > >> either
> > >> > > > > > > >>> > >> > > >> > > case,
> > >> > > > > > > >>> > >> > > >> > > >> >> > current
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder
> > in
> > >> > home
> > >> > > > > > > directory.
> > >> > > > > > > >>> > Also
> > >> > > > > > > >>> > >> > > before
> > >> > > > > > > >>> > >> > > >> > > putting
> > >> > > > > > > >>> > >> > > >> > > >> >> > > anything
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to
> > define
> > >> > what
> > >> > > > are
> > >> > > > > > the
> > >> > > > > > > >>> > default
> > >> > > > > > > >>> > >> > > >> contents
> > >> > > > > > > >>> > >> > > >> > > that
> > >> > > > > > > >>> > >> > > >> > > >> >> > should
> > >> > > > > > > >>> > >> > > >> > > >> >> > > go
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there
> > is
> > >> no
> > >> > > > > > defaults,
> > >> > > > > > > >>> > >> probably we
> > >> > > > > > > >>> > >> > > >> should
> > >> > > > > > > >>> > >> > > >> > > not
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > explicitly add it in
> > >> /etc/skel.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d
> approach
> > >> looks
> > >> > > > nice.
> > >> > > > > > > This
> > >> > > > > > > >>> way
> > >> > > > > > > >>> > >> > > contents
> > >> > > > > > > >>> > >> > > >> of
> > >> > > > > > > >>> > >> > > >> > > >> >> dt-env.sh
> > >> > > > > > > >>> > >> > > >> > > >> >> > > are
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > present as env variables.
> > But
> > >> I
> > >> > > see a
> > >> > > > > > catch
> > >> > > > > > > >>> > there.
> > >> > > > > > > >>> > >> > > Adding
> > >> > > > > > > >>> > >> > > >> > > >> dt-env.sh
> > >> > > > > > > >>> > >> > > >> > > >> >> to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make
> > all
> > >> the
> > >> > > > > > variables
> > >> > > > > > > >>> > >> available
> > >> > > > > > > >>> > >> > at
> > >> > > > > > > >>> > >> > > >> > > runtime
> > >> > > > > > > >>> > >> > > >> > > >> all
> > >> > > > > > > >>> > >> > > >> > > >> >> > the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > time. I feel a little
> > >> skeptical
> > >> > > about
> > >> > > > > > that.
> > >> > > > > > > >>> > Maybe a
> > >> > > > > > > >>> > >> > > >> possible
> > >> > > > > > > >>> > >> > > >> > > >> >> collision
> > >> > > > > > > >>> > >> > > >> > > >> >> > > can
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > occur with other
> application
> > >> > vars.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli
> does
> > >> > source
> > >> > > > > > > >>> > >> > "../conf/dt-env.sh"
> > >> > > > > > > >>> > >> > > >> and
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So
> those
> > >> > > variables
> > >> > > > > are
> > >> > > > > > > >>> anyway
> > >> > > > > > > >>> > >> > > >> available.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and
> PR
> > >> > effort
> > >> > > is
> > >> > > > > > > >>> happening
> > >> > > > > > > >>> > at
> > >> > > > > > > >>> > >> > > Bigtop
> > >> > > > > > > >>> > >> > > >> > > >> >> > > Jira/repository.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > Packaging related code
> > usually
> > >> > goes
> > >> > > > > > there.
> > >> > > > > > > >>> > (That's
> > >> > > > > > > >>> > >> > what
> > >> > > > > > > >>> > >> > > >> all
> > >> > > > > > > >>> > >> > > >> > the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > components
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > in bigtop does).
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > Having said that, once a
> PR
> > is
> > >> > > > created,
> > >> > > > > > > I'll
> > >> > > > > > > >>> be
> > >> > > > > > > >>> > >> > sharing
> > >> > > > > > > >>> > >> > > >> the
> > >> > > > > > > >>> > >> > > >> > > link
> > >> > > > > > > >>> > >> > > >> > > >> of
> > >> > > > > > > >>> > >> > > >> > > >> >> PR
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > here, so that apex
> community
> > >> as
> > >> > > well
> > >> > > > > can
> > >> > > > > > > >>> review
> > >> > > > > > > >>> > it.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > Thanks,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > Chinmay.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at
> 7:57
> > >> PM,
> > >> > > > > > Aniruddha
> > >> > > > > > > >>> > >> Thombare <
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com
> >
> > >> > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and
> > >> approach.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > We may need to iron out
> > >> details
> > >> > > > about
> > >> > > > > > > exact
> > >> > > > > > > >>> > paths
> > >> > > > > > > >>> > >> > etc.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > Which can be done on
> jira
> > /
> > >> PR
> > >> > > > > > comments.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53
> > pm
> > >> > > Pradeep
> > >> > > > > A.
> > >> > > > > > > >>> Dalvi <
> > >> > > > > > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016
> at
> > >> 2:51
> > >> > > PM,
> > >> > > > > > > Chinmay
> > >> > > > > > > >>> > >> > Kolhatkar <
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> chinmay@datatorrent.com
> > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the
> > jar
> > >> > > > present
> > >> > > > > in
> > >> > > > > > > DT
> > >> > > > > > > >>> > >> community
> > >> > > > > > > >>> > >> > > >> > edition
> > >> > > > > > > >>> > >> > > >> > > >> will
> > >> > > > > > > >>> > >> > > >> > > >> >> be
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > included
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of
> > >> jars I
> > >> > > > found
> > >> > > > > > from
> > >> > > > > > > >>> > common
> > >> > > > > > > >>> > >> > > >> > dependencies
> > >> > > > > > > >>> > >> > > >> > > >> >> between
> > >> > > > > > > >>> > >> > > >> > > >> >> > > of
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > apex
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > and DT Community
> > >> edition:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> jackson-core-asl-1.9.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > jackson-mapper-asl-1.9.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > async-http-client-1.7.20.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> netty-3.6.6.Final.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > validation-api-1.1.0.Final.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> commons-lang3-3.1.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> commons-beanutils-1.8.3.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> commons-codec-1.10.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > jetty-websocket-8.1.10.v20130312.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> xbean-asm5-shaded-4.3.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > jersey-apache-client4-1.9.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> ant-launcher-1.9.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > jackson-jaxrs-1.9.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > hadoop-common-2.2.0-tests.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be
> > >> running
> > >> > > some
> > >> > > > > > tests
> > >> > > > > > > do
> > >> > > > > > > >>> > check
> > >> > > > > > > >>> > >> > that
> > >> > > > > > > >>> > >> > > >> > dtcli
> > >> > > > > > > >>> > >> > > >> > > >> works
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > properly
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > for
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill
> > of
> > >> > apps
> > >> > > > with
> > >> > > > > > > only
> > >> > > > > > > >>> > these
> > >> > > > > > > >>> > >> > > >> libraries
> > >> > > > > > > >>> > >> > > >> > > >> >> present in
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > isolation
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > without dependency
> on
> > >> local
> > >> > > m2.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that
> there
> > >> are
> > >> > > > > unwanted
> > >> > > > > > > >>> jars
> > >> > > > > > > >>> > >> which
> > >> > > > > > > >>> > >> > are
> > >> > > > > > > >>> > >> > > >> used
> > >> > > > > > > >>> > >> > > >> > > for
> > >> > > > > > > >>> > >> > > >> > > >> >> > compile
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > time
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > dependency and not
> > >> runtime
> > >> > in
> > >> > > > > above
> > >> > > > > > > >>> list
> > >> > > > > > > >>> > >> which I
> > >> > > > > > > >>> > >> > > can
> > >> > > > > > > >>> > >> > > >> > drop
> > >> > > > > > > >>> > >> > > >> > > >> to
> > >> > > > > > > >>> > >> > > >> > > >> >> keep
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > package
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm
> > not
> > >> > sure
> > >> > > if
> > >> > > > > all
> > >> > > > > > > of
> > >> > > > > > > >>> the
> > >> > > > > > > >>> > >> > > >> dt-env.sh is
> > >> > > > > > > >>> > >> > > >> > > >> >> required.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > Only
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > required values I
> see
> > >> in it
> > >> > > are
> > >> > > > > > > >>> classpath.
> > >> > > > > > > >>> > >> and
> > >> > > > > > > >>> > >> > > >> dtcli is
> > >> > > > > > > >>> > >> > > >> > > >> >> capable
> > >> > > > > > > >>> > >> > > >> > > >> >> > of
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > building
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml ->
> Yes,
> > >> > we'll
> > >> > > > have
> > >> > > > > > to
> > >> > > > > > > >>> see
> > >> > > > > > > >>> > how
> > >> > > > > > > >>> > >> > dtcli
> > >> > > > > > > >>> > >> > > >> can
> > >> > > > > > > >>> > >> > > >> > > find
> > >> > > > > > > >>> > >> > > >> > > >> >> the
> > >> > > > > > > >>> > >> > > >> > > >> >> > jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > path,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all
> such
> > >> conf
> > >> > > > > files.
> > >> > > > > > If
> > >> > > > > > > >>> we
> > >> > > > > > > >>> > >> need a
> > >> > > > > > > >>> > >> > > >> change
> > >> > > > > > > >>> > >> > > >> > in
> > >> > > > > > > >>> > >> > > >> > > >> >> dtcli,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > then
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > community's opinion
> is
> > >> > > required
> > >> > > > > for
> > >> > > > > > > >>> whether
> > >> > > > > > > >>> > >> > dtcli
> > >> > > > > > > >>> > >> > > >> > should
> > >> > > > > > > >>> > >> > > >> > > >> >> change
> > >> > > > > > > >>> > >> > > >> > > >> >> > in
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > our
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > repo
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that
> with
> > >> > > required
> > >> > > > > > > changes
> > >> > > > > > > >>> > >> exist in
> > >> > > > > > > >>> > >> > > >> bigtop
> > >> > > > > > > >>> > >> > > >> > > >> repo
> > >> > > > > > > >>> > >> > > >> > > >> >> > until
> > >> > > > > > > >>> > >> > > >> > > >> >> > > we
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > make
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic
> > >> enough.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Can we have these file
> > >> paths
> > >> > > set
> > >> > > > > > using
> > >> > > > > > > >>> > >> environment
> > >> > > > > > > >>> > >> > > >> > > variables
> > >> > > > > > > >>> > >> > > >> > > >> >> with
> > >> > > > > > > >>> > >> > > >> > > >> >> > > some
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > default values in
> dtcli?
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Then we can set such
> > >> params
> > >> > in
> > >> > > > > > > dt-env.sh.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml ->
> How
> > >> is
> > >> > > this
> > >> > > > > file
> > >> > > > > > > >>> > different
> > >> > > > > > > >>> > >> > from
> > >> > > > > > > >>> > >> > > >> > > >> >> dt-site.xml?
> > >> > > > > > > >>> > >> > > >> > > >> >> > Not
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > sure
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > if
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel
> > >> would
> > >> > > help.
> > >> > > > > > > Files &
> > >> > > > > > > >>> > Dirs
> > >> > > > > > > >>> > >> > from
> > >> > > > > > > >>> > >> > > >> > > /etc/skel
> > >> > > > > > > >>> > >> > > >> > > >> >> are
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > copied
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > home of new user
> when
> > >> > useradd
> > >> > > > > > program
> > >> > > > > > > >>> is
> > >> > > > > > > >>> > >> called.
> > >> > > > > > > >>> > >> > > But
> > >> > > > > > > >>> > >> > > >> > for
> > >> > > > > > > >>> > >> > > >> > > >> >> existing
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > users
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any
> > >> use.
> > >> > > > > Moreover,
> > >> > > > > > > >>> dtcli
> > >> > > > > > > >>> > >> > creates
> > >> > > > > > > >>> > >> > > >> > ~/.dt/
> > >> > > > > > > >>> > >> > > >> > > on
> > >> > > > > > > >>> > >> > > >> > > >> >> the
> > >> > > > > > > >>> > >> > > >> > > >> >> > fly
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > if
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > not
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > encountered for the
> > >> first
> > >> > > time.
> > >> > > > > > Again
> > >> > > > > > > >>> > >> correct me
> > >> > > > > > > >>> > >> > > if
> > >> > > > > > > >>> > >> > > >> I'm
> > >> > > > > > > >>> > >> > > >> > > >> wrong.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right.
> > >> However
> > >> > > this
> > >> > > > > > would
> > >> > > > > > > >>> > >> probably be
> > >> > > > > > > >>> > >> > > >> > > necessary
> > >> > > > > > > >>> > >> > > >> > > >> >> step
> > >> > > > > > > >>> > >> > > >> > > >> >> > for
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may
> > also
> > >> get
> > >> > > > > > installed
> > >> > > > > > > >>> > during
> > >> > > > > > > >>> > >> OS
> > >> > > > > > > >>> > >> > > >> install
> > >> > > > > > > >>> > >> > > >> > > and
> > >> > > > > > > >>> > >> > > >> > > >> >> > before
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > user
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc &
> > >> > > > bash_profile
> > >> > > > > > ->
> > >> > > > > > > >>> I'm
> > >> > > > > > > >>> > not
> > >> > > > > > > >>> > >> > sure
> > >> > > > > > > >>> > >> > > >> its
> > >> > > > > > > >>> > >> > > >> > the
> > >> > > > > > > >>> > >> > > >> > > >> best
> > >> > > > > > > >>> > >> > > >> > > >> >> > > idea.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > We'll
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in
> > >> > location
> > >> > > > > which
> > >> > > > > > is
> > >> > > > > > > >>> by
> > >> > > > > > > >>> > >> default
> > >> > > > > > > >>> > >> > > >> > present
> > >> > > > > > > >>> > >> > > >> > > in
> > >> > > > > > > >>> > >> > > >> > > >> >> path
> > >> > > > > > > >>> > >> > > >> > > >> >> > > i.e.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > For other env
> variables
> > >> > > specific
> > >> > > > to
> > >> > > > > > > apex,
> > >> > > > > > > >>> > I'll
> > >> > > > > > > >>> > >> > > prefer
> > >> > > > > > > >>> > >> > > >> to
> > >> > > > > > > >>> > >> > > >> > > use
> > >> > > > > > > >>> > >> > > >> > > >> >> > > dt-env.sh
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > and
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli
> > >> rather
> > >> > > than
> > >> > > > > > > changing
> > >> > > > > > > >>> > >> bashrc
> > >> > > > > > > >>> > >> > > >> etc...
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Instead, coping
> > dt-env.sh
> > >> to
> > >> > > > > > > >>> /etc/profile.d/
> > >> > > > > > > >>> > >> shall
> > >> > > > > > > >>> > >> > > be
> > >> > > > > > > >>> > >> > > >> the
> > >> > > > > > > >>> > >> > > >> > > >> >> cleaner
> > >> > > > > > > >>> > >> > > >> > > >> >> > way
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Thanks,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016
> > at
> > >> > 1:07
> > >> > > > PM,
> > >> > > > > > > >>> Aniruddha
> > >> > > > > > > >>> > >> > > Thombare
> > >> > > > > > > >>> > >> > > >> <
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> aniruddha@datatorrent.com>
> > >> > > > > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > We need to
> consider
> > >> > > > following:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > System wide
> default
> > >> > config
> > >> > > > > files
> > >> > > > > > > can
> > >> > > > > > > >>> be
> > >> > > > > > > >>> > >> > located
> > >> > > > > > > >>> > >> > > at
> > >> > > > > > > >>> > >> > > >> > > >> following
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > locations:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> /etc/apex/dt-env.sh
> > >> (we
> > >> > > may
> > >> > > > > have
> > >> > > > > > > to
> > >> > > > > > > >>> > change
> > >> > > > > > > >>> > >> > > dtcli
> > >> > > > > > > >>> > >> > > >> > > >> behaviour
> > >> > > > > > > >>> > >> > > >> > > >> >> on
> > >> > > > > > > >>> > >> > > >> > > >> >> > > how
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > it
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > finds
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > /etc/apex/dt-site.xml
> > >> (we
> > >> > > may
> > >> > > > > > have
> > >> > > > > > > to
> > >> > > > > > > >>> > >> change
> > >> > > > > > > >>> > >> > > dtcli
> > >> > > > > > > >>> > >> > > >> > > >> >> behaviour on
> > >> > > > > > > >>> > >> > > >> > > >> >> > > how
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > it
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > finds those
> > locations)
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > /etc/skel/.dt/dt-sited.xml
> > >> > > > and
> > >> > > > > > > other
> > >> > > > > > > >>> > files
> > >> > > > > > > >>> > >> > (for
> > >> > > > > > > >>> > >> > > >> new
> > >> > > > > > > >>> > >> > > >> > > users
> > >> > > > > > > >>> > >> > > >> > > >> >> that
> > >> > > > > > > >>> > >> > > >> > > >> >> > > will
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > be
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > created in system
> in
> > >> > > future.)
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > We may also have
> to
> > >> > modify
> > >> > > > > > bashrc /
> > >> > > > > > > >>> > >> > bashprofile
> > >> > > > > > > >>> > >> > > >> for
> > >> > > > > > > >>> > >> > > >> > > >> >> population
> > >> > > > > > > >>> > >> > > >> > > >> >> > > any
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > variables that are
> > >> > > required.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put
> in
> > >> your
> > >> > > > > > comments /
> > >> > > > > > > >>> > >> > suggestion.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10,
> 2016
> > >> at
> > >> > > 1:16
> > >> > > > > AM,
> > >> > > > > > > >>> Thomas
> > >> > > > > > > >>> > >> Weise
> > >> > > > > > > >>> > >> > <
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > -->
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9,
> > 2016
> > >> at
> > >> > > 9:05
> > >> > > > > AM,
> > >> > > > > > > >>> Chinmay
> > >> > > > > > > >>> > >> > > >> Kolhatkar <
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> chinmay@datatorrent.com
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello
> Everyone!!
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing
> with
> > >> > > packaging
> > >> > > > > > > effort
> > >> > > > > > > >>> > >> (rpm+deb)
> > >> > > > > > > >>> > >> > > of
> > >> > > > > > > >>> > >> > > >> > apex,
> > >> > > > > > > >>> > >> > > >> > > >> here
> > >> > > > > > > >>> > >> > > >> > > >> >> > are
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > some
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > about package
> > >> > structure
> > >> > > > > etc..
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting
> > it
> > >> on
> > >> > > > > bbigtop
> > >> > > > > > > >>> mailing
> > >> > > > > > > >>> > >> > list, I
> > >> > > > > > > >>> > >> > > >> have
> > >> > > > > > > >>> > >> > > >> > > >> some
> > >> > > > > > > >>> > >> > > >> > > >> >> > > question
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > for
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > apex
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed
> > Directory
> > >> > > > > structure
> > >> > > > > > of
> > >> > > > > > > >>> apex
> > >> > > > > > > >>> > >> > package
> > >> > > > > > > >>> > >> > > >> for
> > >> > > > > > > >>> > >> > > >> > > both
> > >> > > > > > > >>> > >> > > >> > > >> >> deb &
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > rpm:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > /usr/lib/apex/bin/dtcli
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >>
> > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > /usr/lib/apex/lib/<other
> > >> > > > > > > >>> dependent
> > >> > > > > > > >>> > >> jars>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli
> ->
> > >> > > > > > > >>> > /usr/lib/apex/bin/dtcli
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> /usr/share/doc/man/man1/dtcli.1.gz
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > /usr/share/doc/apex/license/LICENSE.txt
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >>
> > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > /usr/share/doc/apex/CHANGELOG
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > /usr/share/doc/apex/NOTICE
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
> > >> > > > > 3.3.0-incubating.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <other
> dependent
> > >> > jars>
> > >> > > =
> > >> > > > > All
> > >> > > > > > > the
> > >> > > > > > > >>> 3rd
> > >> > > > > > > >>> > >> party
> > >> > > > > > > >>> > >> > > >> jars
> > >> > > > > > > >>> > >> > > >> > > which
> > >> > > > > > > >>> > >> > > >> > > >> >> are
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > required
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > for
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > apex
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to run.
> Usually
> > >> the
> > >> > > > > > > dependencies
> > >> > > > > > > >>> are
> > >> > > > > > > >>> > >> > > packaged
> > >> > > > > > > >>> > >> > > >> as
> > >> > > > > > > >>> > >> > > >> > > >> part of
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > rpm/deb
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > by
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > any
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > software in
> > >> bigtop.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify
> > what
> > >> > > those
> > >> > > > > jars
> > >> > > > > > > >>> are in
> > >> > > > > > > >>> > >> > Bigtop
> > >> > > > > > > >>> > >> > > >> > > context?
> > >> > > > > > > >>> > >> > > >> > > >> >> Same
> > >> > > > > > > >>> > >> > > >> > > >> >> > as
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > shipped
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > with
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > DT community
> > >> addition
> > >> > > under
> > >> > > > > > lib/
> > >> > > > > > > ?
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <package
> > LICENSE>
> > >> =
> > >> > > > > Licenses
> > >> > > > > > of
> > >> > > > > > > >>> > >> > > corresponding
> > >> > > > > > > >>> > >> > > >> of
> > >> > > > > > > >>> > >> > > >> > > 3rd
> > >> > > > > > > >>> > >> > > >> > > >> >> party
> > >> > > > > > > >>> > >> > > >> > > >> >> > > jars
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > which
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > needs
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to included
> > while
> > >> > > > > packaging.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions
> > related
> > >> to
> > >> > > > this:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we
> > call
> > >> the
> > >> > > cli
> > >> > > > > of
> > >> > > > > > > >>> apex as
> > >> > > > > > > >>> > >> > "apex"
> > >> > > > > > > >>> > >> > > >> > instead
> > >> > > > > > > >>> > >> > > >> > > >> of
> > >> > > > > > > >>> > >> > > >> > > >> >> > > "dtcli"
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > in
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > I think we
> should
> > >> keep
> > >> > > the
> > >> > > > > name
> > >> > > > > > > >>> until
> > >> > > > > > > >>> > we
> > >> > > > > > > >>> > >> are
> > >> > > > > > > >>> > >> > > >> able
> > >> > > > > > > >>> > >> > > >> > to
> > >> > > > > > > >>> > >> > > >> > > >> >> change
> > >> > > > > > > >>> > >> > > >> > > >> >> > it
> > >> > > > > > > >>> > >> > > >> > > >> >> > > in
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Apex.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this
> may
> > >> get
> > >> > > > > > confusing.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that
> > all
> > >> > > > softwares
> > >> > > > > > in
> > >> > > > > > > >>> bigtop
> > >> > > > > > > >>> > >> have
> > >> > > > > > > >>> > >> > > man
> > >> > > > > > > >>> > >> > > >> > page
> > >> > > > > > > >>> > >> > > >> > > >> for
> > >> > > > > > > >>> > >> > > >> > > >> >> > their
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we
> > should
> > >> > have
> > >> > > it
> > >> > > > > too
> > >> > > > > > > for
> > >> > > > > > > >>> > >> dtcli.
> > >> > > > > > > >>> > >> > Is
> > >> > > > > > > >>> > >> > > >> there
> > >> > > > > > > >>> > >> > > >> > > any
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > documentation
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > which I
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to
> > man
> > >> > > page?
> > >> > > > or
> > >> > > > > > can
> > >> > > > > > > >>> I use
> > >> > > > > > > >>> > >> > output
> > >> > > > > > > >>> > >> > > >> of
> > >> > > > > > > >>> > >> > > >> > > >> "dtcli
> > >> > > > > > > >>> > >> > > >> > > >> >> > > --help"?
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want
> to
> > >> call
> > >> > > > > version
> > >> > > > > > > of
> > >> > > > > > > >>> apex
> > >> > > > > > > >>> > >> in
> > >> > > > > > > >>> > >> > > >> Bigtop
> > >> > > > > > > >>> > >> > > >> > as
> > >> > > > > > > >>> > >> > > >> > > >> >> 3.3.0 OR
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > 3.3.0-incubating?
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be
> > >> -incubating
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok
> for
> > >> apex
> > >> > > > > package
> > >> > > > > > of
> > >> > > > > > > >>> > bigtop
> > >> > > > > > > >>> > >> to
> > >> > > > > > > >>> > >> > > >> depend
> > >> > > > > > > >>> > >> > > >> > on
> > >> > > > > > > >>> > >> > > >> > > >> >> 2.7.1
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > version
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop?
> > Any
> > >> > > > problems
> > >> > > > > > > that
> > >> > > > > > > >>> we
> > >> > > > > > > >>> > see
> > >> > > > > > > >>> > >> > with
> > >> > > > > > > >>> > >> > > >> this
> > >> > > > > > > >>> > >> > > >> > > >> >> > dependency?
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Should work
> > without
> > >> > > > changed.
> > >> > > > > I
> > >> > > > > > > >>> thought
> > >> > > > > > > >>> > we
> > >> > > > > > > >>> > >> > > >> certified
> > >> > > > > > > >>> > >> > > >> > > >> >> against
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > 2.7.0?
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following
> is
> > >> the
> > >> > > > apache
> > >> > > > > > > mirror
> > >> > > > > > > >>> > from
> > >> > > > > > > >>> > >> > which
> > >> > > > > > > >>> > >> > > >> > bigtop
> > >> > > > > > > >>> > >> > > >> > > >> will
> > >> > > > > > > >>> > >> > > >> > > >> >> > pick
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > apex
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > source for
> > >> > compilation
> > >> > > > and
> > >> > > > > > > >>> packaging.
> > >> > > > > > > >>> > >> > Please
> > >> > > > > > > >>> > >> > > >> > > correct
> > >> > > > > > > >>> > >> > > >> > > >> if
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > incorrect:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > >
> > >> > > > > > > >>> > >> > >
> > >> > > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
> > >> > > > > apache.osuosl.org
> > >> > > > > > is
> > >> > > > > > > >>> the
> > >> > > > > > > >>> > >> mirror
> > >> > > > > > > >>> > >> > > >> used
> > >> > > > > > > >>> > >> > > >> > by
> > >> > > > > > > >>> > >> > > >> > > >> all
> > >> > > > > > > >>> > >> > > >> > > >> >> the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > softwares
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > in
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share
> > your
> > >> > > > opinion.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8,
> > >> 2016
> > >> > at
> > >> > > > > 11:19
> > >> > > > > > > AM,
> > >> > > > > > > >>> > >> Chinmay
> > >> > > > > > > >>> > >> > > >> > Kolhatkar
> > >> > > > > > > >>> > >> > > >> > > <
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > chinmay@datatorrent.com>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes..
> Starting
> > >> to
> > >> > > work
> > >> > > > on
> > >> > > > > > the
> > >> > > > > > > >>> > >> packaging.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already
> > >> > started
> > >> > > > > > > >>> discussion on
> > >> > > > > > > >>> > >> > bigtop
> > >> > > > > > > >>> > >> > > >> dev
> > >> > > > > > > >>> > >> > > >> > > >> mailing
> > >> > > > > > > >>> > >> > > >> > > >> >> > list
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > for
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration.
> > >> Also
> > >> > > > > created a
> > >> > > > > > > >>> Jira
> > >> > > > > > > >>> > for
> > >> > > > > > > >>> > >> the
> > >> > > > > > > >>> > >> > > >> same.
> > >> > > > > > > >>> > >> > > >> > > For
> > >> > > > > > > >>> > >> > > >> > > >> >> this
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > communities
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference,
> > here
> > >> is
> > >> > > the
> > >> > > > > > bigtop
> > >> > > > > > > >>> Jira:
> > >> > > > > > > >>> > >> > > >> > BIGTOP-2313.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb
> 8,
> > >> 2016
> > >> > > at
> > >> > > > > 1:13
> > >> > > > > > > AM,
> > >> > > > > > > >>> > >> Thomas
> > >> > > > > > > >>> > >> > > >> Weise <
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > thomas@datatorrent.com
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see
> > >> > anything
> > >> > > > > under
> > >> > > > > > > >>> > >> > prerequisites
> > >> > > > > > > >>> > >> > > >> that
> > >> > > > > > > >>> > >> > > >> > > >> would
> > >> > > > > > > >>> > >> > > >> > > >> >> be a
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > problem.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > We
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF
> > >> > licencing
> > >> > > > > > > >>> compatibility
> > >> > > > > > > >>> > >> as
> > >> > > > > > > >>> > >> > > part
> > >> > > > > > > >>> > >> > > >> of
> > >> > > > > > > >>> > >> > > >> > > >> >> becoming
> > >> > > > > > > >>> > >> > > >> > > >> >> > an
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > incubator
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please
> focus
> > on
> > >> > the
> > >> > > > > > > packaging
> > >> > > > > > > >>> > during
> > >> > > > > > > >>> > >> > the
> > >> > > > > > > >>> > >> > > >> next
> > >> > > > > > > >>> > >> > > >> > > >> weeks.
> > >> > > > > > > >>> > >> > > >> > > >> >> > Since
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > work
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > will
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of
> > Bigtop,
> > >> > > > related
> > >> > > > > > > >>> > discussions
> > >> > > > > > > >>> > >> and
> > >> > > > > > > >>> > >> > > >> JIRAs
> > >> > > > > > > >>> > >> > > >> > > >> should
> > >> > > > > > > >>> > >> > > >> > > >> >> > also
> > >> > > > > > > >>> > >> > > >> > > >> >> > > be
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be
> good
> > >> to
> > >> > > have
> > >> > > > > the
> > >> > > > > > > >>> > packaging
> > >> > > > > > > >>> > >> in
> > >> > > > > > > >>> > >> > > >> place
> > >> > > > > > > >>> > >> > > >> > by
> > >> > > > > > > >>> > >> > > >> > > >> end
> > >> > > > > > > >>> > >> > > >> > > >> >> Feb.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb
> > 4,
> > >> > 2016
> > >> > > at
> > >> > > > > > 10:14
> > >> > > > > > > >>> AM,
> > >> > > > > > > >>> > >> > Chinmay
> > >> > > > > > > >>> > >> > > >> > > >> Kolhatkar <
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > > chinmay@datatorrent.com
> > >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're
> > >> planning a
> > >> > > > work
> > >> > > > > on
> > >> > > > > > > >>> adding
> > >> > > > > > > >>> > >> > Apache
> > >> > > > > > > >>> > >> > > >> Apex
> > >> > > > > > > >>> > >> > > >> > > as a
> > >> > > > > > > >>> > >> > > >> > > >> >> > > component
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Apache
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is
> > the
> > >> > > > > packaging
> > >> > > > > > > >>> system
> > >> > > > > > > >>> > for
> > >> > > > > > > >>> > >> > the
> > >> > > > > > > >>> > >> > > >> > Apache
> > >> > > > > > > >>> > >> > > >> > > >> big
> > >> > > > > > > >>> > >> > > >> > > >> >> data
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop
> > >> distros
> > >> > use
> > >> > > > it,
> > >> > > > > > > most
> > >> > > > > > > >>> > >> recently
> > >> > > > > > > >>> > >> > > EMR.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is
> the
> > >> > > tracking
> > >> > > > > > Jira
> > >> > > > > > > >>> task
> > >> > > > > > > >>> > in
> > >> > > > > > > >>> > >> > > >> APEXCORE
> > >> > > > > > > >>> > >> > > >> > for
> > >> > > > > > > >>> > >> > > >> > > >> the
> > >> > > > > > > >>> > >> > > >> > > >> >> > same:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > https://issues.apache.org/jira/browse/APEXCORE-331
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed
> > >> plan of
> > >> > > > > > execution
> > >> > > > > > > >>> is as
> > >> > > > > > > >>> > >> > > follows:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1)
> > >> Handle
> > >> > > > > > > >>> prerequisites*
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache
> > bigtop
> > >> > has
> > >> > > > some
> > >> > > > > > > hard
> > >> > > > > > > >>> and
> > >> > > > > > > >>> > >> soft
> > >> > > > > > > >>> > >> > > >> > > expectation
> > >> > > > > > > >>> > >> > > >> > > >> >> for
> > >> > > > > > > >>> > >> > > >> > > >> >> > new
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > components
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> integrated
> > >> into
> > >> > > > > Bigtop.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is
> the
> > >> list
> > >> > > of
> > >> > > > > it:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > >
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >>
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> >
> > >> > > > > > > >>> > >> > > >>
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> >
> > >> > > > > > > >>> > >>
> > >> > > > > > > >>> >
> > >> > > > > > > >>>
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of
> > them
> > >> > seems
> > >> > > > to
> > >> > > > > be
> > >> > > > > > > >>> > standard
> > >> > > > > > > >>> > >> ASF
> > >> > > > > > > >>> > >> > > >> based
> > >> > > > > > > >>> > >> > > >> > > >> >> > > requirements,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > but
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > few
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > need
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be
> checked
> > >> for:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1.
> Software
> > >> > > projects
> > >> > > > > are
> > >> > > > > > > >>> > expected
> > >> > > > > > > >>> > >> to
> > >> > > > > > > >>> > >> > be
> > >> > > > > > > >>> > >> > > >> > > Licensed
> > >> > > > > > > >>> > >> > > >> > > >> >> under
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > Apache
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > License,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version
> 2.0
> > >> (and
> > >> > > > their
> > >> > > > > > > >>> > >> dependencies
> > >> > > > > > > >>> > >> > are
> > >> > > > > > > >>> > >> > > >> > > expected
> > >> > > > > > > >>> > >> > > >> > > >> >> to be
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > compatible
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > with
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> Apex
> > is
> > >> > > under
> > >> > > > > ASL
> > >> > > > > > > 2.0
> > >> > > > > > > >>> but
> > >> > > > > > > >>> > >> need
> > >> > > > > > > >>> > >> > to
> > >> > > > > > > >>> > >> > > >> > check
> > >> > > > > > > >>> > >> > > >> > > if
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > dependencies
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> compatible
> > >> with
> > >> > > ASL
> > >> > > > > 2.0.
> > >> > > > > > > >>> This I
> > >> > > > > > > >>> > >> guess
> > >> > > > > > > >>> > >> > > >> would
> > >> > > > > > > >>> > >> > > >> > > be a
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > verification
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > check.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2.
> Software
> > >> > > projects
> > >> > > > > are
> > >> > > > > > > >>> > expected
> > >> > > > > > > >>> > >> to
> > >> > > > > > > >>> > >> > be
> > >> > > > > > > >>> > >> > > >> > > >> compatible
> > >> > > > > > > >>> > >> > > >> > > >> >> > with
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > all
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > of
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported
> > >> > > platforms
> > >> > > > > that
> > >> > > > > > > >>> Bigtop
> > >> > > > > > > >>> > >> > > >> distribution
> > >> > > > > > > >>> > >> > > >> > > is
> > >> > > > > > > >>> > >> > > >> > > >> >> > > targeting
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> This
> > >> needs
> > >> > > > > > verifying
> > >> > > > > > > >>> > whether
> > >> > > > > > > >>> > >> > our
> > >> > > > > > > >>> > >> > > >> > > software
> > >> > > > > > > >>> > >> > > >> > > >> >> runs
> > >> > > > > > > >>> > >> > > >> > > >> >> > > fine
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > in
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7
> > >> > fedora-20
> > >> > > > > > > >>> ubuntu-14.04
> > >> > > > > > > >>> > >> > > debian-8
> > >> > > > > > > >>> > >> > > >> > > >> >> > opensuse-13.2.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What
> > smoke
> > >> > > tests
> > >> > > > > that
> > >> > > > > > > >>> should
> > >> > > > > > > >>> > be
> > >> > > > > > > >>> > >> > > added
> > >> > > > > > > >>> > >> > > >> for
> > >> > > > > > > >>> > >> > > >> > > >> >> > deployment.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4.
> > >> Identifying
> > >> > the
> > >> > > > > test
> > >> > > > > > > >>> > artifacts
> > >> > > > > > > >>> > >> > which
> > >> > > > > > > >>> > >> > > >> goes
> > >> > > > > > > >>> > >> > > >> > > >> beyond
> > >> > > > > > > >>> > >> > > >> > > >> >> > > smoke
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > test
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> These
> > >> are
> > >> > > > > > basically
> > >> > > > > > > >>> the
> > >> > > > > > > >>> > >> > > integration
> > >> > > > > > > >>> > >> > > >> > > tests
> > >> > > > > > > >>> > >> > > >> > > >> for
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > verification
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > after
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> deployment.
> > >> This
> > >> > > is
> > >> > > > a
> > >> > > > > > soft
> > >> > > > > > > >>> > >> > requirement,
> > >> > > > > > > >>> > >> > > >> but
> > >> > > > > > > >>> > >> > > >> > > aim
> > >> > > > > > > >>> > >> > > >> > > >> is
> > >> > > > > > > >>> > >> > > >> > > >> >> to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > achieve
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > this
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > as
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at
> least
> > >> have
> > >> > > > > > > >>> explanation why
> > >> > > > > > > >>> > >> not
> > >> > > > > > > >>> > >> > to
> > >> > > > > > > >>> > >> > > >> > > include.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there
> > are
> > >> any
> > >> > > > from
> > >> > > > > > the
> > >> > > > > > > >>> link
> > >> > > > > > > >>> > >> which
> > >> > > > > > > >>> > >> > > >> > > explicitly
> > >> > > > > > > >>> > >> > > >> > > >> >> needs
> > >> > > > > > > >>> > >> > > >> > > >> >> > to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > be
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > checked
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than
> above
> > 4,
> > >> > > please
> > >> > > > > let
> > >> > > > > > > us
> > >> > > > > > > >>> > know.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2)
> > >> Adding
> > >> > > Apex
> > >> > > > > as
> > >> > > > > > > >>> > component
> > >> > > > > > > >>> > >> to
> > >> > > > > > > >>> > >> > > >> Bigtop*
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one
> of
> > >> the
> > >> > > mail
> > >> > > > > > > >>> archive of
> > >> > > > > > > >>> > >> > Bigtop,
> > >> > > > > > > >>> > >> > > >> it
> > >> > > > > > > >>> > >> > > >> > was
> > >> > > > > > > >>> > >> > > >> > > >> >> learnt
> > >> > > > > > > >>> > >> > > >> > > >> >> > > that
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community
> > >> want
> > >> > to
> > >> > > > see
> > >> > > > > > the
> > >> > > > > > > >>> > >> addition of
> > >> > > > > > > >>> > >> > > new
> > >> > > > > > > >>> > >> > > >> > > >> >> components
> > >> > > > > > > >>> > >> > > >> > > >> >> > in
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > phases.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Here
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > are
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the
> phases:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1.
> > Packaging
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> This
> > >> needs
> > >> > > > > > creating
> > >> > > > > > > of
> > >> > > > > > > >>> > >> package
> > >> > > > > > > >>> > >> > > i.e.
> > >> > > > > > > >>> > >> > > >> > rpm
> > >> > > > > > > >>> > >> > > >> > > &
> > >> > > > > > > >>> > >> > > >> > > >> deb
> > >> > > > > > > >>> > >> > > >> > > >> >> > > files.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > >> > > > > > > >>> documentations/READMEs,
> > >> > > > > > > >>> > >> > LICENSE,
> > >> > > > > > > >>> > >> > > >> > > >> DISCLAMER,
> > >> > > > > > > >>> > >> > > >> > > >> >> > NOTES
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > etc
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > if
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > any
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> > >> > > > > documentation
> > >> > > > > > > that
> > >> > > > > > > >>> > need
> > >> > > > > > > >>> > >> to
> > >> > > > > > > >>> > >> > be
> > >> > > > > > > >>> > >> > > >> > added
> > >> > > > > > > >>> > >> > > >> > > to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > distribution
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > our
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> > >> > license
> > >> > > > > > > >>> information of
> > >> > > > > > > >>> > >> > > >> > dependencies
> > >> > > > > > > >>> > >> > > >> > > >> >> required
> > >> > > > > > > >>> > >> > > >> > > >> >> > > to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > be
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > added
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke
> > >> tests
> > >> > (at
> > >> > > > > very
> > >> > > > > > > >>> least)
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > Adding
> > >> > smoke
> > >> > > > > test
> > >> > > > > > > for
> > >> > > > > > > >>> > >> > packaging.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet
> > >> > recipes
> > >> > > > for
> > >> > > > > > > >>> automatic
> > >> > > > > > > >>> > >> > > >> deployment
> > >> > > > > > > >>> > >> > > >> > and
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > configuration
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add
> > >> puppet
> > >> > > > > recipes
> > >> > > > > > > for
> > >> > > > > > > >>> > >> > automatic
> > >> > > > > > > >>> > >> > > >> > > >> deployment
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4.
> > >> Integration
> > >> > > tests
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For
> > >> > > > verification
> > >> > > > > > of
> > >> > > > > > > >>> > >> > deployments.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5.
> license
> > >> > > > clearance:
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run
> > >> 'gradle
> > >> > > rat'
> > >> > > > > to
> > >> > > > > > > make
> > >> > > > > > > >>> > sure
> > >> > > > > > > >>> > >> all
> > >> > > > > > > >>> > >> > > new
> > >> > > > > > > >>> > >> > > >> > > stuff
> > >> > > > > > > >>> > >> > > >> > > >> is
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > compliant
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > with
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license
> > >> > > > requirements.
> > >> > > > > If
> > >> > > > > > > >>> you add
> > >> > > > > > > >>> > >> code
> > >> > > > > > > >>> > >> > > >> > licenses
> > >> > > > > > > >>> > >> > > >> > > >> >> under
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > different
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those
> would
> > >> need
> > >> > > to
> > >> > > > be
> > >> > > > > > > >>> listed in
> > >> > > > > > > >>> > >> the
> > >> > > > > > > >>> > >> > > >> NOTICE.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please
> > share
> > >> > your
> > >> > > > > > thoughts
> > >> > > > > > > >>> on
> > >> > > > > > > >>> > the
> > >> > > > > > > >>> > >> > > >> approach.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll
> start
> > >> > > > > > corresponding
> > >> > > > > > > >>> > >> > communication
> > >> > > > > > > >>> > >> > > >> on
> > >> > > > > > > >>> > >> > > >> > > >> bigtop
> > >> > > > > > > >>> > >> > > >> > > >> >> > > mailing
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > list
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > as
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have
> > some
> > >> > > > specific
> > >> > > > > > > >>> > >> > > >> questions/suggestions
> > >> > > > > > > >>> > >> > > >> > > >> >> related to
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > what
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > should
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > be
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content
> of
> > >> the
> > >> > > > package
> > >> > > > > > and
> > >> > > > > > > >>> what
> > >> > > > > > > >>> > >> > should
> > >> > > > > > > >>> > >> > > be
> > >> > > > > > > >>> > >> > > >> > the
> > >> > > > > > > >>> > >> > > >> > > >> smoke
> > >> > > > > > > >>> > >> > > >> > > >> >> > > tests,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > but
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > in
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest
> of
> > >> not
> > >> > > > having
> > >> > > > > > too
> > >> > > > > > > >>> much
> > >> > > > > > > >>> > >> > content
> > >> > > > > > > >>> > >> > > >> > here,
> > >> > > > > > > >>> > >> > > >> > > >> we'll
> > >> > > > > > > >>> > >> > > >> > > >> >> > put
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > the
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > questions
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate
> > >> mail in
> > >> > > > this
> > >> > > > > > > >>> > mailthread.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > > >
> > >> > > > > > > >>> > >> > > >> > > >> >> > >
> > >> > > > > > > >>> > >> > > >> > > >> >> >
> > >> > > > > > > >>> > >> > > >> > > >> >>
> > >> > > > > > > >>> > >> > > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > > >> >
> > >> > > > > > > >>> > >> > > >> > > >>
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > > >
> > >> > > > > > > >>> > >> > > >> > >
> > >> > > > > > > >>> > >> > > >> >
> > >> > > > > > > >>> > >> > > >>
> > >> > > > > > > >>> > >> > > >
> > >> > > > > > > >>> > >> > > >
> > >> > > > > > > >>> > >> > >
> > >> > > > > > > >>> > >> >
> > >> > > > > > > >>> > >>
> > >> > > > > > > >>> > >
> > >> > > > > > > >>> > >
> > >> > > > > > > >>> >
> > >> > > > > > > >>>
> > >> > > > > > > >>
> > >> > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

Not only is there no objection, here is an enthusiastic +1

On Thu, Mar 3, 2016 at 10:47 AM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Hello Everyone,
>
> I believe there is no object on the smoke test to be added to bigtop for
> apex package.
> I've created smoke tests for bigtop as per above.
>
> If there no further comments, I'll create a pull request for the same.
>
> Thanks,
> Chinmay.
>
>
> On Fri, Feb 26, 2016 at 2:36 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Bigtop has consumed first patch for building apex packages into it's
> > master branch.
> > Commit mail:
> >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-commits/201602.mbox/%3C06c0ca62ac1b483d8ded090f0f2c8d73%40git.apache.org%3E
> >
> > What this means is bigtop master will be able to generate apex packages
> > (3.3.0-incubating), both deb & rpm, when it build.
> >
> > For the second phase, we need to add smoke tests for our package in
> bigtop
> > repo.
> > This is an important steps for bigtop to host the generate package in
> > bigtop repository during its 1.2.0 release.
> >
> > Here are my initial thoughts about smoke test:
> > TEST::
> > 1. Create mvn app with archetype
> > 2. Build app
> > 3. Run app with apex/dtcli
> >
> > VERIFICATION:
> > 1. stdout file gets has random numbers per line.
> >
> > I see app created by archetype is randomNumberGenerator -> Console.
> > I think this should be a good enough smoke test for apex package.
> >
> > Of course I need to check if we can do above with bigtop framework. But I
> > have a hunch that we can, because there are standalone frameworks like us
> > which has similar types of tests.
> >
> > Please share your opinion and thoughts on this.
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> > On Fri, Feb 19, 2016 at 11:38 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> >> We should not package any Hadoop dependencies, they are provided by the
> >> Bigtop stack.
> >>
> >>
> >> On Thu, Feb 18, 2016 at 8:56 PM, Tushar Gosavi <tu...@datatorrent.com>
> >> wrote:
> >>
> >> > I just did a quick install of bigtop 1.1.0 on ubuntu and observed that
> >> >  jersey-client-1.9.jar is already packaged in hadoop
> >> > /usr/lib/hadoop/client/jersey-client.jar, so I guess it will be ok to
> >> > include those in apex lib too. or If we are including hadoop jars
> >> > in classpath while launching the apex application then it may not be
> >> > needed.
> >> >
> >> > - Tushar.
> >> >
> >> >
> >> >
> >> > On Fri, Feb 19, 2016 at 8:44 AM, Chinmay Kolhatkar <
> chinmay@apache.org>
> >> > wrote:
> >> >
> >> > > No worries David. I've updated the manpage.
> >> > >
> >> > > Just an update: apex bigtop packaging for rpm+deb is done. Here is
> the
> >> > > branch on which I'm working if someone wants to look at:
> >> > > https://github.com/chinmaykolhatkar/bigtop/blob/BIGTOP-2314
> >> > >
> >> > > Only the 3rd questions below is the question that is left to be
> >> answered.
> >> > > Once this is resolved, I'll create the pull request in bigtop.
> >> > >
> >> > > Following questions are related to licenses and inclusion of certain
> >> > files
> >> > > in apex rpm+deb:
> >> > > 1. I'll be included LICENSE, DISCLAIMER, NOTICE from top folder of
> >> core
> >> > > repo in our rpm and deb package. Those will be present in standard
> >> > location
> >> > > i.e. /usr/lib/apex/
> >> > > Please let me know otherwise.
> >> > >
> >> > > 2. Most of our runtime dependencies has ASF license, but certain
> >> > libraries
> >> > > has MIT and BSD licenses, hence I've included those licenses too in
> >> rpm &
> >> > > deb package in same location as above.
> >> > > Please let me know otherwise.
> >> > >
> >> > > BSD - jline-2.11.jar, kryo-2.24.0.jar, minlog-1.2.jar
> >> > > MIT - mbassador-1.1.9.jar
> >> > >
> >> > > 3. There are 2 jars (jersey-apache-client4-1.9.jar,
> >> > jersey-client-1.9.jar)
> >> > > which has duel license of CDDL + GPLv2 with classpath exception:
> >> > > https://jersey.java.net/license.html
> >> > > I'm not an expert here. Hence can someone let me know whether these
> >> > > libraries is OK to be packaged in rpm+deb of apex?
> >> > >
> >> > > If yes, should we also include that license in our package?
> >> > >
> >> > > Thanks,
> >> > > Chinmay.
> >> > >
> >> > >
> >> > >
> >> > > On Thu, Feb 18, 2016 at 11:35 PM, David Yan <da...@datatorrent.com>
> >> > wrote:
> >> > >
> >> > > > I should've mentioned that I had a pull request that was merged
> >> > yesterday
> >> > > > that corrected the statement:
> >> > > >
> >> > > > https://github.com/DataTorrent/docs/pull/45
> >> > > >
> >> > > > On Thu, Feb 18, 2016 at 12:07 AM, Tushar Gosavi <
> >> > tushar@datatorrent.com>
> >> > > > wrote:
> >> > > >
> >> > > > > In following statement, can you replace dtGateway with
> >> > > ApplicationMaster
> >> > > > > and Yarn. or may be remove this statement entirely.
> >> > > > > apex is a wrapper around the REST API provided by dtGatway,
> >> > > > >
> >> > > > > - Tushar.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <
> >> > > chinmay@apache.org>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > I've attached man page for apex in Jira. Here is the link to
> it:
> >> > > > > >
> >> https://issues.apache.org/jira/secure/attachment/12788403/apex.1
> >> > > > > >
> >> > > > > > Please view it by: man -l apex.1
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Chinmay.
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <
> >> > > > tushar@datatorrent.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Chinmay,
> >> > > > > > >
> >> > > > > > > I am not able to find the attachment, may be attachments are
> >> not
> >> > > > > allowed
> >> > > > > > in
> >> > > > > > > the mailing list. Can you attach the man page to the Jira?
> >> > > > > > >
> >> > > > > > > - Tushar.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <
> >> > > > chinmay@apache.org
> >> > > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi All,
> >> > > > > > > >
> >> > > > > > > > I've attached man page file for the purpose of bigtop
> >> > > integration.
> >> > > > > > > > Please have a look and let me know if there any any
> feedback
> >> > > about
> >> > > > > it.
> >> > > > > > > >
> >> > > > > > > > You can see the man page content of this file with:
> >> > > > > > > > man -l apex.1
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > Chinmay.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> >> > > > > > chinmay@apache.org>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > >> Thanks. Will follow bigtop recommendation. i.e. remove
> >> > > > > "-incubating".
> >> > > > > > > >>
> >> > > > > > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
> >> > > > > thomas@datatorrent.com
> >> > > > > > >
> >> > > > > > > >> wrote:
> >> > > > > > > >>
> >> > > > > > > >>> Please follow the recommendation and continue discussion
> >> on
> >> > the
> >> > > > > > Bigtop
> >> > > > > > > >>> list.
> >> > > > > > > >>>
> >> > > > > > > >>> --
> >> > > > > > > >>> sent from mobile
> >> > > > > > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <
> >> > > chinmay@apache.org
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > >>>
> >> > > > > > > >>> > I got a recommendation from bigtop community to drop
> >> > > > > "-incubating"
> >> > > > > > > from
> >> > > > > > > >>> > version.
> >> > > > > > > >>> >
> >> > > > > > > >>> > Here is the mail thread:
> >> > > > > > > >>> >
> >> > > > > > > >>> >
> >> > > > > > > >>>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> >> > > > > > > >>> >
> >> > > > > > > >>> > Please let me know if removing "-incubating" is fine.
> >> > > > > > > >>> >
> >> > > > > > > >>> > Thanks,
> >> > > > > > > >>> > Chinmay.
> >> > > > > > > >>> >
> >> > > > > > > >>> >
> >> > > > > > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> >> > > > > > > chinmay@apache.org
> >> > > > > > > >>> >
> >> > > > > > > >>> > wrote:
> >> > > > > > > >>> >
> >> > > > > > > >>> > > I've already started communication there.
> >> > > > > > > >>> > >
> >> > > > > > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> >> > > > > > > >>> thomas@datatorrent.com>
> >> > > > > > > >>> > > wrote:
> >> > > > > > > >>> > >
> >> > > > > > > >>> > >> This is a question you should ask on the Bigtop
> >> mailing
> >> > > > list.
> >> > > > > > > >>> > >>
> >> > > > > > > >>> > >> --
> >> > > > > > > >>> > >> sent from mobile
> >> > > > > > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <
> >> > > > prad@apache.org>
> >> > > > > > > >>> wrote:
> >> > > > > > > >>> > >>
> >> > > > > > > >>> > >> > Yes. In case of RPM, the only restriction on
> >> version
> >> > is
> >> > > > that
> >> > > > > > it
> >> > > > > > > >>> can
> >> > > > > > > >>> > not
> >> > > > > > > >>> > >> > contain '-'.
> >> > > > > > > >>> > >> > However, as a workaround, one can use
> 'incubating'
> >> as
> >> > a
> >> > > > part
> >> > > > > > of
> >> > > > > > > >>> > >> %{release}.
> >> > > > > > > >>> > >> >
> >> > > > > > > >>> > >> > Version: 3.3.0
> >> > > > > > > >>> > >> > Release: incubating
> >> > > > > > > >>> > >> >
> >> > > > > > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> >> > > > > > > >>> > >> >
> >> > > > > > > >>> > >> > It is the same practice followed for pre-release
> >> > > packages
> >> > > > > e.g.
> >> > > > > > > >>> with
> >> > > > > > > >>> > rcX,
> >> > > > > > > >>> > >> > alphaX, betaX being used in %{release}.
> >> > > > > > > >>> > >> >
> >> > > > > > > >>> > >> > Regards,
> >> > > > > > > >>> > >> > Pradeep A. Dalvi
> >> > > > > > > >>> > >> >
> >> > > > > > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay
> Kolhatkar
> >> <
> >> > > > > > > >>> > chinmay@apache.org>
> >> > > > > > > >>> > >> > wrote:
> >> > > > > > > >>> > >> >
> >> > > > > > > >>> > >> > > Hi Everyone,
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> > > In one of the previous mails in this thread, we
> >> > > decided
> >> > > > > that
> >> > > > > > > >>> version
> >> > > > > > > >>> > >> of
> >> > > > > > > >>> > >> > the
> >> > > > > > > >>> > >> > > package will be "3.3.0-incubating".
> >> > > > > > > >>> > >> > > The problem that I'm facing is in RPM base
> >> version
> >> > "-"
> >> > > > is
> >> > > > > > not
> >> > > > > > > >>> > allowed.
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> > > Can we have package version to "3.3.0"?
> >> > > > > > > >>> > >> > > i.e. the packages created will be as follows:
> >> > > > > > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> >> > > > > > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> > > There is another integration for Zepplin in
> >> Bigtop
> >> > > which
> >> > > > > is
> >> > > > > > > >>> > incubating
> >> > > > > > > >>> > >> > and
> >> > > > > > > >>> > >> > > is having base version as "0.5.6".
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> > > Please let me know your thoughts.
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> > > Thanks,
> >> > > > > > > >>> > >> > > Chinmay.
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay
> >> Kolhatkar
> >> > <
> >> > > > > > > >>> > >> chinmay@apache.org>
> >> > > > > > > >>> > >> > > wrote:
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> > > > Thanks everyone for the feedback.
> >> > > > > > > >>> > >> > > >
> >> > > > > > > >>> > >> > > > Also as Ram and David mentioned, I'll start
> >> > creating
> >> > > > the
> >> > > > > > man
> >> > > > > > > >>> page
> >> > > > > > > >>> > as
> >> > > > > > > >>> > >> > > well.
> >> > > > > > > >>> > >> > > >
> >> > > > > > > >>> > >> > > > Thanks,
> >> > > > > > > >>> > >> > > > Chinmay.
> >> > > > > > > >>> > >> > > >
> >> > > > > > > >>> > >> > > >
> >> > > > > > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas
> Weise <
> >> > > > > > > >>> > >> thomas@datatorrent.com>
> >> > > > > > > >>> > >> > > > wrote:
> >> > > > > > > >>> > >> > > >
> >> > > > > > > >>> > >> > > >> I think we already agreed that we will
> retain
> >> the
> >> > > > > current
> >> > > > > > > >>> script
> >> > > > > > > >>> > >> name
> >> > > > > > > >>> > >> > in
> >> > > > > > > >>> > >> > > >> this bigtop iteration. There is a ticket for
> >> > > > renaming,
> >> > > > > > add
> >> > > > > > > >>> you
> >> > > > > > > >>> > >> > > suggestions
> >> > > > > > > >>> > >> > > >> there.
> >> > > > > > > >>> > >> > > >>
> >> > > > > > > >>> > >> > > >> --
> >> > > > > > > >>> > >> > > >> sent from mobile
> >> > > > > > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> >> > > > > > > >>> > sandesh@datatorrent.com>
> >> > > > > > > >>> > >> > > wrote:
> >> > > > > > > >>> > >> > > >>
> >> > > > > > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start
> >> the
> >> > > > > > transition
> >> > > > > > > >>> > >> process by
> >> > > > > > > >>> > >> > > >> making
> >> > > > > > > >>> > >> > > >> > dtcli as the symlink to apex cli.
> >> > > > > > > >>> > >> > > >> >
> >> > > > > > > >>> > >> > > >> > Renaming is a breaking change should be
> done
> >> > > during
> >> > > > > > major
> >> > > > > > > >>> > version
> >> > > > > > > >>> > >> > > number
> >> > > > > > > >>> > >> > > >> > change.
> >> > > > > > > >>> > >> > > >> >
> >> > > > > > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan
> <
> >> > > > > > > >>> > david@datatorrent.com
> >> > > > > > > >>> > >> >
> >> > > > > > > >>> > >> > > >> wrote:
> >> > > > > > > >>> > >> > > >> >
> >> > > > > > > >>> > >> > > >> > > Apologies for hitting the "Send" button
> >> too
> >> > > > early.
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > What is the scope of the name change?
> If
> >> we
> >> > > > simply
> >> > > > > > > have
> >> > > > > > > >>> a
> >> > > > > > > >>> > >> symlink
> >> > > > > > > >>> > >> > > >> from
> >> > > > > > > >>> > >> > > >> > > apex to dtcli, then it should have no
> >> > backward
> >> > > > > > > >>> compatibility
> >> > > > > > > >>> > >> > problem
> >> > > > > > > >>> > >> > > >> at
> >> > > > > > > >>> > >> > > >> > > all.
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > But the proper name change would
> involve:
> >> > > > > > > >>> > >> > > >> > > - Change the name of DTCli class to
> >> ApexCli
> >> > > > > > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> >> > > > > > > >>> > >> > > >> > > - Change the initial greeting when
> running
> >> > the
> >> > > > CLI
> >> > > > > > from
> >> > > > > > > >>> > >> something
> >> > > > > > > >>> > >> > > like
> >> > > > > > > >>> > >> > > >> > "DT
> >> > > > > > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI
> >> ...".
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > The above changes may break existing
> >> scripts
> >> > > that
> >> > > > > > rely
> >> > > > > > > >>> on the
> >> > > > > > > >>> > >> the
> >> > > > > > > >>> > >> > > >> string
> >> > > > > > > >>> > >> > > >> > > "DT".
> >> > > > > > > >>> > >> > > >> > > But if folks are okay with this
> potential
> >> > > > problem,
> >> > > > > I
> >> > > > > > am
> >> > > > > > > >>> okay
> >> > > > > > > >>> > >> with
> >> > > > > > > >>> > >> > > the
> >> > > > > > > >>> > >> > > >> > name
> >> > > > > > > >>> > >> > > >> > > change.
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > For the man page, we can start with the
> >> > content
> >> > > > of
> >> > > > > > this
> >> > > > > > > >>> page:
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > Plus the command line options when doing
> >> > dtcli
> >> > > > -h:
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > usage: DTCli
> >> > > > > > > >>> > >> > > >> > >  -e <arg>    Commands are read from the
> >> > > argument
> >> > > > > > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at
> >> all
> >> > > time
> >> > > > > > > >>> > >> > > >> > >  -h          Print this help
> >> > > > > > > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos
> >> > > principal
> >> > > > > > > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos
> >> > keytab
> >> > > > > > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> >> > > > > > > >>> > >> > > >> > >  -r          JSON Raw mode
> >> > > > > > > >>> > >> > > >> > >  -v          Verbose mode level 1
> >> > > > > > > >>> > >> > > >> > >  -vv         Verbose mode level 2
> >> > > > > > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> >> > > > > > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > David
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David
> >> Yan <
> >> > > > > > > >>> > >> david@datatorrent.com
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> > > >> > wrote:
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > > What is the scope of the name change?
> >> If
> >> > we
> >> > > > > simply
> >> > > > > > > >>> have a
> >> > > > > > > >>> > >> > symlink
> >> > > > > > > >>> > >> > > >> from
> >> > > > > > > >>> > >> > > >> > > > apex to dtcli, then it should have no
> >> > > backward
> >> > > > > > > >>> > compatibility
> >> > > > > > > >>> > >> > > >> problem at
> >> > > > > > > >>> > >> > > >> > > all.
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > > > But the proper name change would
> >> involve:
> >> > > > > > > >>> > >> > > >> > > > - Change the name of DTCli class to
> >> ApexCli
> >> > > > > > > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> >> > > > > > > >>> > >> > > >> > > > - Change the initial greeting when
> >> running
> >> > > the
> >> > > > > CLI
> >> > > > > > > from
> >> > > > > > > >>> > >> > something
> >> > > > > > > >>> > >> > > >> like
> >> > > > > > > >>> > >> > > >> > > "DT
> >> > > > > > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI
> >> > ...".
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > > > The above changes may break existing
> >> > scripts
> >> > > > that
> >> > > > > > > rely
> >> > > > > > > >>> on
> >> > > > > > > >>> > the
> >> > > > > > > >>> > >> > the
> >> > > > > > > >>> > >> > > >> > string
> >> > > > > > > >>> > >> > > >> > > > "DT".
> >> > > > > > > >>> > >> > > >> > > > But if folks are okay with this
> >> potential
> >> > > > > problem,
> >> > > > > > I
> >> > > > > > > am
> >> > > > > > > >>> > okay
> >> > > > > > > >>> > >> > with
> >> > > > > > > >>> > >> > > >> the
> >> > > > > > > >>> > >> > > >> > > name
> >> > > > > > > >>> > >> > > >> > > > change.
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > > > For the man page, we can start with
> the
> >> > > content
> >> > > > > of
> >> > > > > > > this
> >> > > > > > > >>> > page:
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM,
> Chinmay
> >> > > > > Kolhatkar
> >> > > > > > <
> >> > > > > > > >>> > >> > > >> chinmay@apache.org
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> > > > wrote:
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > > >> Here is a suggestion I have related
> to
> >> > name
> >> > > > > > change:
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > > >>> > >> > > >> > > >> 1. While packaging we change the name
> >> of
> >> > > > "dtcli"
> >> > > > > > > >>> script to
> >> > > > > > > >>> > >> > > "apex".
> >> > > > > > > >>> > >> > > >> > This
> >> > > > > > > >>> > >> > > >> > > is
> >> > > > > > > >>> > >> > > >> > > >> done so that in future when we change
> >> the
> >> > > name
> >> > > > > to
> >> > > > > > > >>> "apex",
> >> > > > > > > >>> > >> users
> >> > > > > > > >>> > >> > > of
> >> > > > > > > >>> > >> > > >> > > bigtop
> >> > > > > > > >>> > >> > > >> > > >> apex don't have to transition much.
> >> > > > > > > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli
> >> > which
> >> > > > > points
> >> > > > > > > to
> >> > > > > > > >>> apex
> >> > > > > > > >>> > >> > > script.
> >> > > > > > > >>> > >> > > >> > This
> >> > > > > > > >>> > >> > > >> > > is
> >> > > > > > > >>> > >> > > >> > > >> for backward compatibility.
> >> > > > > > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for
> >> first
> >> > > > > > > integration
> >> > > > > > > >>> in
> >> > > > > > > >>> > >> > bigtop.
> >> > > > > > > >>> > >> > > >> This
> >> > > > > > > >>> > >> > > >> > is
> >> > > > > > > >>> > >> > > >> > > >> in
> >> > > > > > > >>> > >> > > >> > > >> the interest of not changing the
> >> content
> >> > of
> >> > > > > dtcli
> >> > > > > > or
> >> > > > > > > >>> apex
> >> > > > > > > >>> > >> > script.
> >> > > > > > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all
> >> > these
> >> > > > > naming
> >> > > > > > > >>> related
> >> > > > > > > >>> > >> > > changes
> >> > > > > > > >>> > >> > > >> and
> >> > > > > > > >>> > >> > > >> > > >> update bigtop repository later to
> >> remove
> >> > > > > > references
> >> > > > > > > to
> >> > > > > > > >>> > dtcli
> >> > > > > > > >>> > >> > all
> >> > > > > > > >>> > >> > > >> > > together.
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > > >>> > >> > > >> > > >> Please share your thoughts on above
> >> > > approach.
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > > >>> > >> > > >> > > >> Also, please share what could be the
> >> man
> >> > > page
> >> > > > > > > content
> >> > > > > > > >>> for
> >> > > > > > > >>> > >> > "apex".
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > > >>> > >> > > >> > > >> Thanks,
> >> > > > > > > >>> > >> > > >> > > >> Chinmay.
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM,
> >> Chinmay
> >> > > > > > Kolhatkar <
> >> > > > > > > >>> > >> > > >> > chinmay@apache.org>
> >> > > > > > > >>> > >> > > >> > > >> wrote:
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > > >>> > >> > > >> > > >> > Yes.. It'll be build from source
> tar
> >> > > > > downloaded
> >> > > > > > > >>> from one
> >> > > > > > > >>> > >> of
> >> > > > > > > >>> > >> > the
> >> > > > > > > >>> > >> > > >> > apache
> >> > > > > > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> >> > > > > > > >>> > >> > > >> > > >> >
> >> > > > > > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's
> >> sourced
> >> > > > from
> >> > > > > > > dtcli.
> >> > > > > > > >>> > This
> >> > > > > > > >>> > >> > > leaves
> >> > > > > > > >>> > >> > > >> us
> >> > > > > > > >>> > >> > > >> > 3
> >> > > > > > > >>> > >> > > >> > > >> > options:
> >> > > > > > > >>> > >> > > >> > > >> > If we should not have any
> dt-env.sh,
> >> > dtcli
> >> > > > > would
> >> > > > > > > >>> need a
> >> > > > > > > >>> > >> > change
> >> > > > > > > >>> > >> > > >> while
> >> > > > > > > >>> > >> > > >> > > >> > packaging.
> >> > > > > > > >>> > >> > > >> > > >> >
> >> > > > > > > >>> > >> > > >> > > >> > But if we want to use dtcli as it
> is,
> >> > then
> >> > > > we
> >> > > > > > > would
> >> > > > > > > >>> need
> >> > > > > > > >>> > >> > > >> dt-env.sh
> >> > > > > > > >>> > >> > > >> > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >
> >> > > > > > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM,
> >> Thomas
> >> > > > Weise
> >> > > > > <
> >> > > > > > > >>> > >> > > >> > thomas@datatorrent.com
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > > >> > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> Looks good overall, though there
> >> > > shouldn't
> >> > > > be
> >> > > > > > any
> >> > > > > > > >>> > >> dt-env.sh
> >> > > > > > > >>> > >> > > >> > > >> >>
> >> > > > > > > >>> > >> > > >> > > >> >> What will the bigtop package be
> >> built
> >> > > from,
> >> > > > > the
> >> > > > > > > >>> source
> >> > > > > > > >>> > >> tar?
> >> > > > > > > >>> > >> > > >> > > >> >>
> >> > > > > > > >>> > >> > > >> > > >> >>
> >> > > > > > > >>> > >> > > >> > > >> >>
> >> > > > > > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM,
> >> > Chinmay
> >> > > > > > > Kolhatkar
> >> > > > > > > >>> <
> >> > > > > > > >>> > >> > > >> > > chinmay@apache.org
> >> > > > > > > >>> > >> > > >> > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> wrote:
> >> > > > > > > >>> > >> > > >> > > >> >>
> >> > > > > > > >>> > >> > > >> > > >> >> > Hi Thomas,
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > First, we're not changing the
> name
> >> > > > > anywhere.
> >> > > > > > > >>> We'll
> >> > > > > > > >>> > >> follow
> >> > > > > > > >>> > >> > > what
> >> > > > > > > >>> > >> > > >> > > >> >> currently is
> >> > > > > > > >>> > >> > > >> > > >> >> > in source tarball for
> >> > 3.3.0-incubating
> >> > > > > > version.
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a
> >> directory
> >> > > > > > structure
> >> > > > > > > >>> below
> >> > > > > > > >>> > >> which
> >> > > > > > > >>> > >> > > is
> >> > > > > > > >>> > >> > > >> > > inline
> >> > > > > > > >>> > >> > > >> > > >> >> with
> >> > > > > > > >>> > >> > > >> > > >> >> > other existing integrations with
> >> > > bigtop.
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > The need for each file is as
> >> follows:
> >> > > > > > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex
> >> picked
> >> > > from
> >> > > > > > > >>> > >> > > >> engine/src/main/scripts/
> >> > > > > > > >>> > >> > > >> > > of
> >> > > > > > > >>> > >> > > >> > > >> >> source
> >> > > > > > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is
> >> source
> >> > > from
> >> > > > > > dtcli.
> >> > > > > > > >>> This
> >> > > > > > > >>> > >> > > searches
> >> > > > > > > >>> > >> > > >> for
> >> > > > > > > >>> > >> > > >> > > >> hadoop
> >> > > > > > > >>> > >> > > >> > > >> >> > binary path and exports an env
> >> > variable
> >> > > > for
> >> > > > > > > >>> dtcli to
> >> > > > > > > >>> > >> use
> >> > > > > > > >>> > >> > it.
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI.
> This
> >> > list
> >> > > > is
> >> > > > > > > >>> extracted
> >> > > > > > > >>> > >> in a
> >> > > > > > > >>> > >> > > >> similar
> >> > > > > > > >>> > >> > > >> > > >> way to
> >> > > > > > > >>> > >> > > >> > > >> >> > how DT Community Edition finds
> >> > > dependency
> >> > > > > > jars.
> >> > > > > > > >>> > >> > > >> > > >> >> > Please note that this is the
> first
> >> > > > > iteration
> >> > > > > > > >>> list of
> >> > > > > > > >>> > >> jars.
> >> > > > > > > >>> > >> > > I'm
> >> > > > > > > >>> > >> > > >> > > >> trying to
> >> > > > > > > >>> > >> > > >> > > >> >> > narrow this down to only those
> >> which
> >> > > are
> >> > > > > > really
> >> > > > > > > >>> > >> required.
> >> > > > > > > >>> > >> > > >> > > >> >> > The test that I'm running to
> >> check if
> >> > > > dtcli
> >> > > > > > > runs
> >> > > > > > > >>> fine
> >> > > > > > > >>> > >> with
> >> > > > > > > >>> > >> > > >> given
> >> > > > > > > >>> > >> > > >> > > >> >> dependency
> >> > > > > > > >>> > >> > > >> > > >> >> > is to
> launch+shutdown+kill+status
> >> for
> >> > > pi
> >> > > > > > demo.
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > Above is the directory structure
> >> > which
> >> > > is
> >> > > > > > > >>> required
> >> > > > > > > >>> > for
> >> > > > > > > >>> > >> CLI
> >> > > > > > > >>> > >> > > to
> >> > > > > > > >>> > >> > > >> > work.
> >> > > > > > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has
> >> > > mentioned
> >> > > > > are
> >> > > > > > > some
> >> > > > > > > >>> > >> > additional
> >> > > > > > > >>> > >> > > >> > files
> >> > > > > > > >>> > >> > > >> > > >> which
> >> > > > > > > >>> > >> > > >> > > >> >> > makes the like of administrator
> >> > easier.
> >> > > > For
> >> > > > > > eg.
> >> > > > > > > >>> > >> /etc/skel
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > Please let me know if this is
> >> inline
> >> > > with
> >> > > > > > what
> >> > > > > > > >>> you're
> >> > > > > > > >>> > >> > > >> thinking.
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > Thanks,
> >> > > > > > > >>> > >> > > >> > > >> >> > Chinmay.
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > .
> >> > > > > > > >>> > >> > > >> > > >> >> > |-- bin
> >> > > > > > > >>> > >> > > >> > > >> >> > |   `-- dtcli
> >> > > > > > > >>> > >> > > >> > > >> >> > |-- conf
> >> > > > > > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> >> > > > > > > >>> > >> > > >> > > >> >> > `-- lib
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> apex-api-3.3.0-incubating.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> > > > > > apex-bufferserver-3.3.0-incubating.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> > > apex-common-3.3.0-incubating.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> async-http-client-1.7.20.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> commons-beanutils-1.8.3.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> commons-logging-1.1.3.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> > grizzly-http-servlet-2.1.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> hadoop-common-2.2.0-tests.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> jackson-core-asl-1.9.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> jackson-mapper-asl-1.9.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> javax.servlet-api-3.0.1.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> jersey-apache-client4-1.9.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> > jetty-http-8.1.10.v20130312.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> jetty-io-8.1.10.v20130312.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> > jetty-util-8.1.10.v20130312.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> > > > > jetty-websocket-8.1.10.v20130312.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> >> > validation-api-1.1.0.Final.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     |--
> xbean-asm5-shaded-4.3.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM,
> >> > Thomas
> >> > > > > > Weise <
> >> > > > > > > >>> > >> > > >> > > >> thomas@datatorrent.com>
> >> > > > > > > >>> > >> > > >> > > >> >> > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > Chinmay,
> >> > > > > > > >>> > >> > > >> > > >> >> > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > Before discussing where to put
> >> the
> >> > > > files,
> >> > > > > > > let's
> >> > > > > > > >>> > make
> >> > > > > > > >>> > >> > sure
> >> > > > > > > >>> > >> > > >> they
> >> > > > > > > >>> > >> > > >> > > are
> >> > > > > > > >>> > >> > > >> > > >> >> really
> >> > > > > > > >>> > >> > > >> > > >> >> > > needed for the operation of
> the
> >> > CLI.
> >> > > As
> >> > > > > for
> >> > > > > > > >>> names,
> >> > > > > > > >>> > >> > > anything
> >> > > > > > > >>> > >> > > >> new
> >> > > > > > > >>> > >> > > >> > > >> needs
> >> > > > > > > >>> > >> > > >> > > >> >> to
> >> > > > > > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and
> >> should
> >> > > > > follow
> >> > > > > > > >>> common
> >> > > > > > > >>> > >> > > >> conventions,
> >> > > > > > > >>> > >> > > >> > > >> >> especially
> >> > > > > > > >>> > >> > > >> > > >> >> > > in Bigtop where there are many
> >> > > existing
> >> > > > > > > >>> > integrations
> >> > > > > > > >>> > >> to
> >> > > > > > > >>> > >> > > look
> >> > > > > > > >>> > >> > > >> > at.
> >> > > > > > > >>> > >> > > >> > > >> >> > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > Thanks,
> >> > > > > > > >>> > >> > > >> > > >> >> > > Thomas
> >> > > > > > > >>> > >> > > >> > > >> >> > >
> >> > > > > > > >>> > >> > > >> > > >> >> > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07
> AM,
> >> > > > Chinmay
> >> > > > > > > >>> Kolhatkar
> >> > > > > > > >>> > <
> >> > > > > > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> >> > > > > > > >>> > >> > > >> > > >> >> > > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep
> and
> >> > > > > Aniruddha.
> >> > > > > > > >>> > >> > > >> > > >> >> > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need
> to
> >> > > change
> >> > > > > the
> >> > > > > > > >>> dtcli
> >> > > > > > > >>> > >> > > >> considering it
> >> > > > > > > >>> > >> > > >> > > >> works
> >> > > > > > > >>> > >> > > >> > > >> >> > with
> >> > > > > > > >>> > >> > > >> > > >> >> > > DT
> >> > > > > > > >>> > >> > > >> > > >> >> > > > Community edition. We can
> keep
> >> > the
> >> > > > > > > directory
> >> > > > > > > >>> > >> structure
> >> > > > > > > >>> > >> > > >> > similar
> >> > > > > > > >>> > >> > > >> > > to
> >> > > > > > > >>> > >> > > >> > > >> >> that.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables
> which
> >> > > > contains
> >> > > > > > > >>> > information
> >> > > > > > > >>> > >> > > >> required
> >> > > > > > > >>> > >> > > >> > for
> >> > > > > > > >>> > >> > > >> > > >> >> dtcli
> >> > > > > > > >>> > >> > > >> > > >> >> > to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > launch.
> >> > > > > > > >>> > >> > > >> > > >> >> > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop
> >> > > community
> >> > > > > > that
> >> > > > > > > >>> > whether
> >> > > > > > > >>> > >> > they
> >> > > > > > > >>> > >> > > >> > > >> facilitate
> >> > > > > > > >>> > >> > > >> > > >> >> the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs
> >> before
> >> > > user
> >> > > > > is
> >> > > > > > > >>> created.
> >> > > > > > > >>> > >> In
> >> > > > > > > >>> > >> > > >> either
> >> > > > > > > >>> > >> > > >> > > case,
> >> > > > > > > >>> > >> > > >> > > >> >> > current
> >> > > > > > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder
> in
> >> > home
> >> > > > > > > directory.
> >> > > > > > > >>> > Also
> >> > > > > > > >>> > >> > > before
> >> > > > > > > >>> > >> > > >> > > putting
> >> > > > > > > >>> > >> > > >> > > >> >> > > anything
> >> > > > > > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to
> define
> >> > what
> >> > > > are
> >> > > > > > the
> >> > > > > > > >>> > default
> >> > > > > > > >>> > >> > > >> contents
> >> > > > > > > >>> > >> > > >> > > that
> >> > > > > > > >>> > >> > > >> > > >> >> > should
> >> > > > > > > >>> > >> > > >> > > >> >> > > go
> >> > > > > > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there
> is
> >> no
> >> > > > > > defaults,
> >> > > > > > > >>> > >> probably we
> >> > > > > > > >>> > >> > > >> should
> >> > > > > > > >>> > >> > > >> > > not
> >> > > > > > > >>> > >> > > >> > > >> >> > > > explicitly add it in
> >> /etc/skel.
> >> > > > > > > >>> > >> > > >> > > >> >> > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach
> >> looks
> >> > > > nice.
> >> > > > > > > This
> >> > > > > > > >>> way
> >> > > > > > > >>> > >> > > contents
> >> > > > > > > >>> > >> > > >> of
> >> > > > > > > >>> > >> > > >> > > >> >> dt-env.sh
> >> > > > > > > >>> > >> > > >> > > >> >> > > are
> >> > > > > > > >>> > >> > > >> > > >> >> > > > present as env variables.
> But
> >> I
> >> > > see a
> >> > > > > > catch
> >> > > > > > > >>> > there.
> >> > > > > > > >>> > >> > > Adding
> >> > > > > > > >>> > >> > > >> > > >> dt-env.sh
> >> > > > > > > >>> > >> > > >> > > >> >> to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make
> all
> >> the
> >> > > > > > variables
> >> > > > > > > >>> > >> available
> >> > > > > > > >>> > >> > at
> >> > > > > > > >>> > >> > > >> > > runtime
> >> > > > > > > >>> > >> > > >> > > >> all
> >> > > > > > > >>> > >> > > >> > > >> >> > the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > time. I feel a little
> >> skeptical
> >> > > about
> >> > > > > > that.
> >> > > > > > > >>> > Maybe a
> >> > > > > > > >>> > >> > > >> possible
> >> > > > > > > >>> > >> > > >> > > >> >> collision
> >> > > > > > > >>> > >> > > >> > > >> >> > > can
> >> > > > > > > >>> > >> > > >> > > >> >> > > > occur with other application
> >> > vars.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does
> >> > source
> >> > > > > > > >>> > >> > "../conf/dt-env.sh"
> >> > > > > > > >>> > >> > > >> and
> >> > > > > > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those
> >> > > variables
> >> > > > > are
> >> > > > > > > >>> anyway
> >> > > > > > > >>> > >> > > >> available.
> >> > > > > > > >>> > >> > > >> > > >> >> > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR
> >> > effort
> >> > > is
> >> > > > > > > >>> happening
> >> > > > > > > >>> > at
> >> > > > > > > >>> > >> > > Bigtop
> >> > > > > > > >>> > >> > > >> > > >> >> > > Jira/repository.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > Packaging related code
> usually
> >> > goes
> >> > > > > > there.
> >> > > > > > > >>> > (That's
> >> > > > > > > >>> > >> > what
> >> > > > > > > >>> > >> > > >> all
> >> > > > > > > >>> > >> > > >> > the
> >> > > > > > > >>> > >> > > >> > > >> >> > > components
> >> > > > > > > >>> > >> > > >> > > >> >> > > > in bigtop does).
> >> > > > > > > >>> > >> > > >> > > >> >> > > > Having said that, once a PR
> is
> >> > > > created,
> >> > > > > > > I'll
> >> > > > > > > >>> be
> >> > > > > > > >>> > >> > sharing
> >> > > > > > > >>> > >> > > >> the
> >> > > > > > > >>> > >> > > >> > > link
> >> > > > > > > >>> > >> > > >> > > >> of
> >> > > > > > > >>> > >> > > >> > > >> >> PR
> >> > > > > > > >>> > >> > > >> > > >> >> > > > here, so that apex community
> >> as
> >> > > well
> >> > > > > can
> >> > > > > > > >>> review
> >> > > > > > > >>> > it.
> >> > > > > > > >>> > >> > > >> > > >> >> > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > Thanks,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > Chinmay.
> >> > > > > > > >>> > >> > > >> > > >> >> > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57
> >> PM,
> >> > > > > > Aniruddha
> >> > > > > > > >>> > >> Thombare <
> >> > > > > > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com>
> >> > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and
> >> approach.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > We may need to iron out
> >> details
> >> > > > about
> >> > > > > > > exact
> >> > > > > > > >>> > paths
> >> > > > > > > >>> > >> > etc.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > Which can be done on jira
> /
> >> PR
> >> > > > > > comments.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53
> pm
> >> > > Pradeep
> >> > > > > A.
> >> > > > > > > >>> Dalvi <
> >> > > > > > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at
> >> 2:51
> >> > > PM,
> >> > > > > > > Chinmay
> >> > > > > > > >>> > >> > Kolhatkar <
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com
> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the
> jar
> >> > > > present
> >> > > > > in
> >> > > > > > > DT
> >> > > > > > > >>> > >> community
> >> > > > > > > >>> > >> > > >> > edition
> >> > > > > > > >>> > >> > > >> > > >> will
> >> > > > > > > >>> > >> > > >> > > >> >> be
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > included
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > there.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of
> >> jars I
> >> > > > found
> >> > > > > > from
> >> > > > > > > >>> > common
> >> > > > > > > >>> > >> > > >> > dependencies
> >> > > > > > > >>> > >> > > >> > > >> >> between
> >> > > > > > > >>> > >> > > >> > > >> >> > > of
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > apex
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > and DT Community
> >> edition:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> jackson-core-asl-1.9.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > jackson-mapper-asl-1.9.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > async-http-client-1.7.20.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > validation-api-1.1.0.Final.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> commons-beanutils-1.8.3.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > jetty-websocket-8.1.10.v20130312.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> xbean-asm5-shaded-4.3.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > jersey-apache-client4-1.9.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> jackson-jaxrs-1.9.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > hadoop-common-2.2.0-tests.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be
> >> running
> >> > > some
> >> > > > > > tests
> >> > > > > > > do
> >> > > > > > > >>> > check
> >> > > > > > > >>> > >> > that
> >> > > > > > > >>> > >> > > >> > dtcli
> >> > > > > > > >>> > >> > > >> > > >> works
> >> > > > > > > >>> > >> > > >> > > >> >> > > > properly
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > for
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill
> of
> >> > apps
> >> > > > with
> >> > > > > > > only
> >> > > > > > > >>> > these
> >> > > > > > > >>> > >> > > >> libraries
> >> > > > > > > >>> > >> > > >> > > >> >> present in
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > isolation
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > without dependency on
> >> local
> >> > > m2.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there
> >> are
> >> > > > > unwanted
> >> > > > > > > >>> jars
> >> > > > > > > >>> > >> which
> >> > > > > > > >>> > >> > are
> >> > > > > > > >>> > >> > > >> used
> >> > > > > > > >>> > >> > > >> > > for
> >> > > > > > > >>> > >> > > >> > > >> >> > compile
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > time
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > dependency and not
> >> runtime
> >> > in
> >> > > > > above
> >> > > > > > > >>> list
> >> > > > > > > >>> > >> which I
> >> > > > > > > >>> > >> > > can
> >> > > > > > > >>> > >> > > >> > drop
> >> > > > > > > >>> > >> > > >> > > >> to
> >> > > > > > > >>> > >> > > >> > > >> >> keep
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > package
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm
> not
> >> > sure
> >> > > if
> >> > > > > all
> >> > > > > > > of
> >> > > > > > > >>> the
> >> > > > > > > >>> > >> > > >> dt-env.sh is
> >> > > > > > > >>> > >> > > >> > > >> >> required.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > Only
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > required values I see
> >> in it
> >> > > are
> >> > > > > > > >>> classpath.
> >> > > > > > > >>> > >> and
> >> > > > > > > >>> > >> > > >> dtcli is
> >> > > > > > > >>> > >> > > >> > > >> >> capable
> >> > > > > > > >>> > >> > > >> > > >> >> > of
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > building
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes,
> >> > we'll
> >> > > > have
> >> > > > > > to
> >> > > > > > > >>> see
> >> > > > > > > >>> > how
> >> > > > > > > >>> > >> > dtcli
> >> > > > > > > >>> > >> > > >> can
> >> > > > > > > >>> > >> > > >> > > find
> >> > > > > > > >>> > >> > > >> > > >> >> the
> >> > > > > > > >>> > >> > > >> > > >> >> > jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > path,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such
> >> conf
> >> > > > > files.
> >> > > > > > If
> >> > > > > > > >>> we
> >> > > > > > > >>> > >> need a
> >> > > > > > > >>> > >> > > >> change
> >> > > > > > > >>> > >> > > >> > in
> >> > > > > > > >>> > >> > > >> > > >> >> dtcli,
> >> > > > > > > >>> > >> > > >> > > >> >> > > then
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > community's opinion is
> >> > > required
> >> > > > > for
> >> > > > > > > >>> whether
> >> > > > > > > >>> > >> > dtcli
> >> > > > > > > >>> > >> > > >> > should
> >> > > > > > > >>> > >> > > >> > > >> >> change
> >> > > > > > > >>> > >> > > >> > > >> >> > in
> >> > > > > > > >>> > >> > > >> > > >> >> > > > our
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > repo
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with
> >> > > required
> >> > > > > > > changes
> >> > > > > > > >>> > >> exist in
> >> > > > > > > >>> > >> > > >> bigtop
> >> > > > > > > >>> > >> > > >> > > >> repo
> >> > > > > > > >>> > >> > > >> > > >> >> > until
> >> > > > > > > >>> > >> > > >> > > >> >> > > we
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > make
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic
> >> enough.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Can we have these file
> >> paths
> >> > > set
> >> > > > > > using
> >> > > > > > > >>> > >> environment
> >> > > > > > > >>> > >> > > >> > > variables
> >> > > > > > > >>> > >> > > >> > > >> >> with
> >> > > > > > > >>> > >> > > >> > > >> >> > > some
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Then we can set such
> >> params
> >> > in
> >> > > > > > > dt-env.sh.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How
> >> is
> >> > > this
> >> > > > > file
> >> > > > > > > >>> > different
> >> > > > > > > >>> > >> > from
> >> > > > > > > >>> > >> > > >> > > >> >> dt-site.xml?
> >> > > > > > > >>> > >> > > >> > > >> >> > Not
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > sure
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > if
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel
> >> would
> >> > > help.
> >> > > > > > > Files &
> >> > > > > > > >>> > Dirs
> >> > > > > > > >>> > >> > from
> >> > > > > > > >>> > >> > > >> > > /etc/skel
> >> > > > > > > >>> > >> > > >> > > >> >> are
> >> > > > > > > >>> > >> > > >> > > >> >> > > > copied
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > home of new user when
> >> > useradd
> >> > > > > > program
> >> > > > > > > >>> is
> >> > > > > > > >>> > >> called.
> >> > > > > > > >>> > >> > > But
> >> > > > > > > >>> > >> > > >> > for
> >> > > > > > > >>> > >> > > >> > > >> >> existing
> >> > > > > > > >>> > >> > > >> > > >> >> > > > users
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any
> >> use.
> >> > > > > Moreover,
> >> > > > > > > >>> dtcli
> >> > > > > > > >>> > >> > creates
> >> > > > > > > >>> > >> > > >> > ~/.dt/
> >> > > > > > > >>> > >> > > >> > > on
> >> > > > > > > >>> > >> > > >> > > >> >> the
> >> > > > > > > >>> > >> > > >> > > >> >> > fly
> >> > > > > > > >>> > >> > > >> > > >> >> > > > if
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > not
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > encountered for the
> >> first
> >> > > time.
> >> > > > > > Again
> >> > > > > > > >>> > >> correct me
> >> > > > > > > >>> > >> > > if
> >> > > > > > > >>> > >> > > >> I'm
> >> > > > > > > >>> > >> > > >> > > >> wrong.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right.
> >> However
> >> > > this
> >> > > > > > would
> >> > > > > > > >>> > >> probably be
> >> > > > > > > >>> > >> > > >> > > necessary
> >> > > > > > > >>> > >> > > >> > > >> >> step
> >> > > > > > > >>> > >> > > >> > > >> >> > for
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may
> also
> >> get
> >> > > > > > installed
> >> > > > > > > >>> > during
> >> > > > > > > >>> > >> OS
> >> > > > > > > >>> > >> > > >> install
> >> > > > > > > >>> > >> > > >> > > and
> >> > > > > > > >>> > >> > > >> > > >> >> > before
> >> > > > > > > >>> > >> > > >> > > >> >> > > > user
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc &
> >> > > > bash_profile
> >> > > > > > ->
> >> > > > > > > >>> I'm
> >> > > > > > > >>> > not
> >> > > > > > > >>> > >> > sure
> >> > > > > > > >>> > >> > > >> its
> >> > > > > > > >>> > >> > > >> > the
> >> > > > > > > >>> > >> > > >> > > >> best
> >> > > > > > > >>> > >> > > >> > > >> >> > > idea.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > We'll
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in
> >> > location
> >> > > > > which
> >> > > > > > is
> >> > > > > > > >>> by
> >> > > > > > > >>> > >> default
> >> > > > > > > >>> > >> > > >> > present
> >> > > > > > > >>> > >> > > >> > > in
> >> > > > > > > >>> > >> > > >> > > >> >> path
> >> > > > > > > >>> > >> > > >> > > >> >> > > i.e.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > For other env variables
> >> > > specific
> >> > > > to
> >> > > > > > > apex,
> >> > > > > > > >>> > I'll
> >> > > > > > > >>> > >> > > prefer
> >> > > > > > > >>> > >> > > >> to
> >> > > > > > > >>> > >> > > >> > > use
> >> > > > > > > >>> > >> > > >> > > >> >> > > dt-env.sh
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > and
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli
> >> rather
> >> > > than
> >> > > > > > > changing
> >> > > > > > > >>> > >> bashrc
> >> > > > > > > >>> > >> > > >> etc...
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Instead, coping
> dt-env.sh
> >> to
> >> > > > > > > >>> /etc/profile.d/
> >> > > > > > > >>> > >> shall
> >> > > > > > > >>> > >> > > be
> >> > > > > > > >>> > >> > > >> the
> >> > > > > > > >>> > >> > > >> > > >> >> cleaner
> >> > > > > > > >>> > >> > > >> > > >> >> > way
> >> > > > > > > >>> > >> > > >> > > >> >> > > > to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Thanks,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016
> at
> >> > 1:07
> >> > > > PM,
> >> > > > > > > >>> Aniruddha
> >> > > > > > > >>> > >> > > Thombare
> >> > > > > > > >>> > >> > > >> <
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> aniruddha@datatorrent.com>
> >> > > > > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > We need to consider
> >> > > > following:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > System wide default
> >> > config
> >> > > > > files
> >> > > > > > > can
> >> > > > > > > >>> be
> >> > > > > > > >>> > >> > located
> >> > > > > > > >>> > >> > > at
> >> > > > > > > >>> > >> > > >> > > >> following
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > locations:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh
> >> (we
> >> > > may
> >> > > > > have
> >> > > > > > > to
> >> > > > > > > >>> > change
> >> > > > > > > >>> > >> > > dtcli
> >> > > > > > > >>> > >> > > >> > > >> behaviour
> >> > > > > > > >>> > >> > > >> > > >> >> on
> >> > > > > > > >>> > >> > > >> > > >> >> > > how
> >> > > > > > > >>> > >> > > >> > > >> >> > > > it
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > finds
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> /etc/apex/dt-site.xml
> >> (we
> >> > > may
> >> > > > > > have
> >> > > > > > > to
> >> > > > > > > >>> > >> change
> >> > > > > > > >>> > >> > > dtcli
> >> > > > > > > >>> > >> > > >> > > >> >> behaviour on
> >> > > > > > > >>> > >> > > >> > > >> >> > > how
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > it
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > finds those
> locations)
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > /etc/skel/.dt/dt-sited.xml
> >> > > > and
> >> > > > > > > other
> >> > > > > > > >>> > files
> >> > > > > > > >>> > >> > (for
> >> > > > > > > >>> > >> > > >> new
> >> > > > > > > >>> > >> > > >> > > users
> >> > > > > > > >>> > >> > > >> > > >> >> that
> >> > > > > > > >>> > >> > > >> > > >> >> > > will
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > be
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > created in system in
> >> > > future.)
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > We may also have to
> >> > modify
> >> > > > > > bashrc /
> >> > > > > > > >>> > >> > bashprofile
> >> > > > > > > >>> > >> > > >> for
> >> > > > > > > >>> > >> > > >> > > >> >> population
> >> > > > > > > >>> > >> > > >> > > >> >> > > any
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > variables that are
> >> > > required.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in
> >> your
> >> > > > > > comments /
> >> > > > > > > >>> > >> > suggestion.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016
> >> at
> >> > > 1:16
> >> > > > > AM,
> >> > > > > > > >>> Thomas
> >> > > > > > > >>> > >> Weise
> >> > > > > > > >>> > >> > <
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > -->
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9,
> 2016
> >> at
> >> > > 9:05
> >> > > > > AM,
> >> > > > > > > >>> Chinmay
> >> > > > > > > >>> > >> > > >> Kolhatkar <
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> chinmay@datatorrent.com
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with
> >> > > packaging
> >> > > > > > > effort
> >> > > > > > > >>> > >> (rpm+deb)
> >> > > > > > > >>> > >> > > of
> >> > > > > > > >>> > >> > > >> > apex,
> >> > > > > > > >>> > >> > > >> > > >> here
> >> > > > > > > >>> > >> > > >> > > >> >> > are
> >> > > > > > > >>> > >> > > >> > > >> >> > > > some
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > about package
> >> > structure
> >> > > > > etc..
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting
> it
> >> on
> >> > > > > bbigtop
> >> > > > > > > >>> mailing
> >> > > > > > > >>> > >> > list, I
> >> > > > > > > >>> > >> > > >> have
> >> > > > > > > >>> > >> > > >> > > >> some
> >> > > > > > > >>> > >> > > >> > > >> >> > > question
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > for
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > apex
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed
> Directory
> >> > > > > structure
> >> > > > > > of
> >> > > > > > > >>> apex
> >> > > > > > > >>> > >> > package
> >> > > > > > > >>> > >> > > >> for
> >> > > > > > > >>> > >> > > >> > > both
> >> > > > > > > >>> > >> > > >> > > >> >> deb &
> >> > > > > > > >>> > >> > > >> > > >> >> > > > rpm:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > /usr/lib/apex/bin/dtcli
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >>
> >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > /usr/lib/apex/lib/<other
> >> > > > > > > >>> dependent
> >> > > > > > > >>> > >> jars>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> >> > > > > > > >>> > /usr/lib/apex/bin/dtcli
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> /usr/share/doc/man/man1/dtcli.1.gz
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > /usr/share/doc/apex/license/LICENSE.txt
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >>
> >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > /usr/share/doc/apex/CHANGELOG
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > /usr/share/doc/apex/NOTICE
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
> >> > > > > 3.3.0-incubating.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent
> >> > jars>
> >> > > =
> >> > > > > All
> >> > > > > > > the
> >> > > > > > > >>> 3rd
> >> > > > > > > >>> > >> party
> >> > > > > > > >>> > >> > > >> jars
> >> > > > > > > >>> > >> > > >> > > which
> >> > > > > > > >>> > >> > > >> > > >> >> are
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > required
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > for
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > apex
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually
> >> the
> >> > > > > > > dependencies
> >> > > > > > > >>> are
> >> > > > > > > >>> > >> > > packaged
> >> > > > > > > >>> > >> > > >> as
> >> > > > > > > >>> > >> > > >> > > >> part of
> >> > > > > > > >>> > >> > > >> > > >> >> > > > rpm/deb
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > by
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > any
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > software in
> >> bigtop.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify
> what
> >> > > those
> >> > > > > jars
> >> > > > > > > >>> are in
> >> > > > > > > >>> > >> > Bigtop
> >> > > > > > > >>> > >> > > >> > > context?
> >> > > > > > > >>> > >> > > >> > > >> >> Same
> >> > > > > > > >>> > >> > > >> > > >> >> > as
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > shipped
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > with
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > DT community
> >> addition
> >> > > under
> >> > > > > > lib/
> >> > > > > > > ?
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <package
> LICENSE>
> >> =
> >> > > > > Licenses
> >> > > > > > of
> >> > > > > > > >>> > >> > > corresponding
> >> > > > > > > >>> > >> > > >> of
> >> > > > > > > >>> > >> > > >> > > 3rd
> >> > > > > > > >>> > >> > > >> > > >> >> party
> >> > > > > > > >>> > >> > > >> > > >> >> > > jars
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > which
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > needs
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to included
> while
> >> > > > > packaging.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions
> related
> >> to
> >> > > > this:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we
> call
> >> the
> >> > > cli
> >> > > > > of
> >> > > > > > > >>> apex as
> >> > > > > > > >>> > >> > "apex"
> >> > > > > > > >>> > >> > > >> > instead
> >> > > > > > > >>> > >> > > >> > > >> of
> >> > > > > > > >>> > >> > > >> > > >> >> > > "dtcli"
> >> > > > > > > >>> > >> > > >> > > >> >> > > > in
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > I think we should
> >> keep
> >> > > the
> >> > > > > name
> >> > > > > > > >>> until
> >> > > > > > > >>> > we
> >> > > > > > > >>> > >> are
> >> > > > > > > >>> > >> > > >> able
> >> > > > > > > >>> > >> > > >> > to
> >> > > > > > > >>> > >> > > >> > > >> >> change
> >> > > > > > > >>> > >> > > >> > > >> >> > it
> >> > > > > > > >>> > >> > > >> > > >> >> > > in
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > Apex.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may
> >> get
> >> > > > > > confusing.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that
> all
> >> > > > softwares
> >> > > > > > in
> >> > > > > > > >>> bigtop
> >> > > > > > > >>> > >> have
> >> > > > > > > >>> > >> > > man
> >> > > > > > > >>> > >> > > >> > page
> >> > > > > > > >>> > >> > > >> > > >> for
> >> > > > > > > >>> > >> > > >> > > >> >> > their
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we
> should
> >> > have
> >> > > it
> >> > > > > too
> >> > > > > > > for
> >> > > > > > > >>> > >> dtcli.
> >> > > > > > > >>> > >> > Is
> >> > > > > > > >>> > >> > > >> there
> >> > > > > > > >>> > >> > > >> > > any
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > documentation
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > which I
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to
> man
> >> > > page?
> >> > > > or
> >> > > > > > can
> >> > > > > > > >>> I use
> >> > > > > > > >>> > >> > output
> >> > > > > > > >>> > >> > > >> of
> >> > > > > > > >>> > >> > > >> > > >> "dtcli
> >> > > > > > > >>> > >> > > >> > > >> >> > > --help"?
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to
> >> call
> >> > > > > version
> >> > > > > > > of
> >> > > > > > > >>> apex
> >> > > > > > > >>> > >> in
> >> > > > > > > >>> > >> > > >> Bigtop
> >> > > > > > > >>> > >> > > >> > as
> >> > > > > > > >>> > >> > > >> > > >> >> 3.3.0 OR
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> 3.3.0-incubating?
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be
> >> -incubating
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for
> >> apex
> >> > > > > package
> >> > > > > > of
> >> > > > > > > >>> > bigtop
> >> > > > > > > >>> > >> to
> >> > > > > > > >>> > >> > > >> depend
> >> > > > > > > >>> > >> > > >> > on
> >> > > > > > > >>> > >> > > >> > > >> >> 2.7.1
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > version
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop?
> Any
> >> > > > problems
> >> > > > > > > that
> >> > > > > > > >>> we
> >> > > > > > > >>> > see
> >> > > > > > > >>> > >> > with
> >> > > > > > > >>> > >> > > >> this
> >> > > > > > > >>> > >> > > >> > > >> >> > dependency?
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Should work
> without
> >> > > > changed.
> >> > > > > I
> >> > > > > > > >>> thought
> >> > > > > > > >>> > we
> >> > > > > > > >>> > >> > > >> certified
> >> > > > > > > >>> > >> > > >> > > >> >> against
> >> > > > > > > >>> > >> > > >> > > >> >> > > > 2.7.0?
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is
> >> the
> >> > > > apache
> >> > > > > > > mirror
> >> > > > > > > >>> > from
> >> > > > > > > >>> > >> > which
> >> > > > > > > >>> > >> > > >> > bigtop
> >> > > > > > > >>> > >> > > >> > > >> will
> >> > > > > > > >>> > >> > > >> > > >> >> > pick
> >> > > > > > > >>> > >> > > >> > > >> >> > > > the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > apex
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > source for
> >> > compilation
> >> > > > and
> >> > > > > > > >>> packaging.
> >> > > > > > > >>> > >> > Please
> >> > > > > > > >>> > >> > > >> > > correct
> >> > > > > > > >>> > >> > > >> > > >> if
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > incorrect:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > >
> >> > > > > > > >>> > >> > >
> >> > > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
> >> > > > > apache.osuosl.org
> >> > > > > > is
> >> > > > > > > >>> the
> >> > > > > > > >>> > >> mirror
> >> > > > > > > >>> > >> > > >> used
> >> > > > > > > >>> > >> > > >> > by
> >> > > > > > > >>> > >> > > >> > > >> all
> >> > > > > > > >>> > >> > > >> > > >> >> the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > softwares
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > in
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share
> your
> >> > > > opinion.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8,
> >> 2016
> >> > at
> >> > > > > 11:19
> >> > > > > > > AM,
> >> > > > > > > >>> > >> Chinmay
> >> > > > > > > >>> > >> > > >> > Kolhatkar
> >> > > > > > > >>> > >> > > >> > > <
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > chinmay@datatorrent.com>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting
> >> to
> >> > > work
> >> > > > on
> >> > > > > > the
> >> > > > > > > >>> > >> packaging.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already
> >> > started
> >> > > > > > > >>> discussion on
> >> > > > > > > >>> > >> > bigtop
> >> > > > > > > >>> > >> > > >> dev
> >> > > > > > > >>> > >> > > >> > > >> mailing
> >> > > > > > > >>> > >> > > >> > > >> >> > list
> >> > > > > > > >>> > >> > > >> > > >> >> > > > for
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration.
> >> Also
> >> > > > > created a
> >> > > > > > > >>> Jira
> >> > > > > > > >>> > for
> >> > > > > > > >>> > >> the
> >> > > > > > > >>> > >> > > >> same.
> >> > > > > > > >>> > >> > > >> > > For
> >> > > > > > > >>> > >> > > >> > > >> >> this
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > communities
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference,
> here
> >> is
> >> > > the
> >> > > > > > bigtop
> >> > > > > > > >>> Jira:
> >> > > > > > > >>> > >> > > >> > BIGTOP-2313.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8,
> >> 2016
> >> > > at
> >> > > > > 1:13
> >> > > > > > > AM,
> >> > > > > > > >>> > >> Thomas
> >> > > > > > > >>> > >> > > >> Weise <
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> thomas@datatorrent.com
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see
> >> > anything
> >> > > > > under
> >> > > > > > > >>> > >> > prerequisites
> >> > > > > > > >>> > >> > > >> that
> >> > > > > > > >>> > >> > > >> > > >> would
> >> > > > > > > >>> > >> > > >> > > >> >> be a
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > problem.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > We
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF
> >> > licencing
> >> > > > > > > >>> compatibility
> >> > > > > > > >>> > >> as
> >> > > > > > > >>> > >> > > part
> >> > > > > > > >>> > >> > > >> of
> >> > > > > > > >>> > >> > > >> > > >> >> becoming
> >> > > > > > > >>> > >> > > >> > > >> >> > an
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > incubator
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus
> on
> >> > the
> >> > > > > > > packaging
> >> > > > > > > >>> > during
> >> > > > > > > >>> > >> > the
> >> > > > > > > >>> > >> > > >> next
> >> > > > > > > >>> > >> > > >> > > >> weeks.
> >> > > > > > > >>> > >> > > >> > > >> >> > Since
> >> > > > > > > >>> > >> > > >> > > >> >> > > > the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > work
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > will
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of
> Bigtop,
> >> > > > related
> >> > > > > > > >>> > discussions
> >> > > > > > > >>> > >> and
> >> > > > > > > >>> > >> > > >> JIRAs
> >> > > > > > > >>> > >> > > >> > > >> should
> >> > > > > > > >>> > >> > > >> > > >> >> > also
> >> > > > > > > >>> > >> > > >> > > >> >> > > be
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > there.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good
> >> to
> >> > > have
> >> > > > > the
> >> > > > > > > >>> > packaging
> >> > > > > > > >>> > >> in
> >> > > > > > > >>> > >> > > >> place
> >> > > > > > > >>> > >> > > >> > by
> >> > > > > > > >>> > >> > > >> > > >> end
> >> > > > > > > >>> > >> > > >> > > >> >> Feb.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb
> 4,
> >> > 2016
> >> > > at
> >> > > > > > 10:14
> >> > > > > > > >>> AM,
> >> > > > > > > >>> > >> > Chinmay
> >> > > > > > > >>> > >> > > >> > > >> Kolhatkar <
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >> > > > chinmay@datatorrent.com
> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're
> >> planning a
> >> > > > work
> >> > > > > on
> >> > > > > > > >>> adding
> >> > > > > > > >>> > >> > Apache
> >> > > > > > > >>> > >> > > >> Apex
> >> > > > > > > >>> > >> > > >> > > as a
> >> > > > > > > >>> > >> > > >> > > >> >> > > component
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Apache
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is
> the
> >> > > > > packaging
> >> > > > > > > >>> system
> >> > > > > > > >>> > for
> >> > > > > > > >>> > >> > the
> >> > > > > > > >>> > >> > > >> > Apache
> >> > > > > > > >>> > >> > > >> > > >> big
> >> > > > > > > >>> > >> > > >> > > >> >> data
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop
> >> distros
> >> > use
> >> > > > it,
> >> > > > > > > most
> >> > > > > > > >>> > >> recently
> >> > > > > > > >>> > >> > > EMR.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the
> >> > > tracking
> >> > > > > > Jira
> >> > > > > > > >>> task
> >> > > > > > > >>> > in
> >> > > > > > > >>> > >> > > >> APEXCORE
> >> > > > > > > >>> > >> > > >> > for
> >> > > > > > > >>> > >> > > >> > > >> the
> >> > > > > > > >>> > >> > > >> > > >> >> > same:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > https://issues.apache.org/jira/browse/APEXCORE-331
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed
> >> plan of
> >> > > > > > execution
> >> > > > > > > >>> is as
> >> > > > > > > >>> > >> > > follows:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1)
> >> Handle
> >> > > > > > > >>> prerequisites*
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache
> bigtop
> >> > has
> >> > > > some
> >> > > > > > > hard
> >> > > > > > > >>> and
> >> > > > > > > >>> > >> soft
> >> > > > > > > >>> > >> > > >> > > expectation
> >> > > > > > > >>> > >> > > >> > > >> >> for
> >> > > > > > > >>> > >> > > >> > > >> >> > new
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > components
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated
> >> into
> >> > > > > Bigtop.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the
> >> list
> >> > > of
> >> > > > > it:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > >
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >>
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> >
> >> > > > > > > >>> > >> > > >>
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> >
> >> > > > > > > >>> > >>
> >> > > > > > > >>> >
> >> > > > > > > >>>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of
> them
> >> > seems
> >> > > > to
> >> > > > > be
> >> > > > > > > >>> > standard
> >> > > > > > > >>> > >> ASF
> >> > > > > > > >>> > >> > > >> based
> >> > > > > > > >>> > >> > > >> > > >> >> > > requirements,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > but
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > few
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > need
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked
> >> for:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software
> >> > > projects
> >> > > > > are
> >> > > > > > > >>> > expected
> >> > > > > > > >>> > >> to
> >> > > > > > > >>> > >> > be
> >> > > > > > > >>> > >> > > >> > > Licensed
> >> > > > > > > >>> > >> > > >> > > >> >> under
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > Apache
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > License,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0
> >> (and
> >> > > > their
> >> > > > > > > >>> > >> dependencies
> >> > > > > > > >>> > >> > are
> >> > > > > > > >>> > >> > > >> > > expected
> >> > > > > > > >>> > >> > > >> > > >> >> to be
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > compatible
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > with
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex
> is
> >> > > under
> >> > > > > ASL
> >> > > > > > > 2.0
> >> > > > > > > >>> but
> >> > > > > > > >>> > >> need
> >> > > > > > > >>> > >> > to
> >> > > > > > > >>> > >> > > >> > check
> >> > > > > > > >>> > >> > > >> > > if
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > dependencies
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible
> >> with
> >> > > ASL
> >> > > > > 2.0.
> >> > > > > > > >>> This I
> >> > > > > > > >>> > >> guess
> >> > > > > > > >>> > >> > > >> would
> >> > > > > > > >>> > >> > > >> > > be a
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > verification
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > check.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software
> >> > > projects
> >> > > > > are
> >> > > > > > > >>> > expected
> >> > > > > > > >>> > >> to
> >> > > > > > > >>> > >> > be
> >> > > > > > > >>> > >> > > >> > > >> compatible
> >> > > > > > > >>> > >> > > >> > > >> >> > with
> >> > > > > > > >>> > >> > > >> > > >> >> > > > all
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > of
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported
> >> > > platforms
> >> > > > > that
> >> > > > > > > >>> Bigtop
> >> > > > > > > >>> > >> > > >> distribution
> >> > > > > > > >>> > >> > > >> > > is
> >> > > > > > > >>> > >> > > >> > > >> >> > > targeting
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This
> >> needs
> >> > > > > > verifying
> >> > > > > > > >>> > whether
> >> > > > > > > >>> > >> > our
> >> > > > > > > >>> > >> > > >> > > software
> >> > > > > > > >>> > >> > > >> > > >> >> runs
> >> > > > > > > >>> > >> > > >> > > >> >> > > fine
> >> > > > > > > >>> > >> > > >> > > >> >> > > > in
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7
> >> > fedora-20
> >> > > > > > > >>> ubuntu-14.04
> >> > > > > > > >>> > >> > > debian-8
> >> > > > > > > >>> > >> > > >> > > >> >> > opensuse-13.2.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What
> smoke
> >> > > tests
> >> > > > > that
> >> > > > > > > >>> should
> >> > > > > > > >>> > be
> >> > > > > > > >>> > >> > > added
> >> > > > > > > >>> > >> > > >> for
> >> > > > > > > >>> > >> > > >> > > >> >> > deployment.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4.
> >> Identifying
> >> > the
> >> > > > > test
> >> > > > > > > >>> > artifacts
> >> > > > > > > >>> > >> > which
> >> > > > > > > >>> > >> > > >> goes
> >> > > > > > > >>> > >> > > >> > > >> beyond
> >> > > > > > > >>> > >> > > >> > > >> >> > > smoke
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > test
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These
> >> are
> >> > > > > > basically
> >> > > > > > > >>> the
> >> > > > > > > >>> > >> > > integration
> >> > > > > > > >>> > >> > > >> > > tests
> >> > > > > > > >>> > >> > > >> > > >> for
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > verification
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > after
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment.
> >> This
> >> > > is
> >> > > > a
> >> > > > > > soft
> >> > > > > > > >>> > >> > requirement,
> >> > > > > > > >>> > >> > > >> but
> >> > > > > > > >>> > >> > > >> > > aim
> >> > > > > > > >>> > >> > > >> > > >> is
> >> > > > > > > >>> > >> > > >> > > >> >> to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > achieve
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > this
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > as
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least
> >> have
> >> > > > > > > >>> explanation why
> >> > > > > > > >>> > >> not
> >> > > > > > > >>> > >> > to
> >> > > > > > > >>> > >> > > >> > > include.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there
> are
> >> any
> >> > > > from
> >> > > > > > the
> >> > > > > > > >>> link
> >> > > > > > > >>> > >> which
> >> > > > > > > >>> > >> > > >> > > explicitly
> >> > > > > > > >>> > >> > > >> > > >> >> needs
> >> > > > > > > >>> > >> > > >> > > >> >> > to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > be
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > checked
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above
> 4,
> >> > > please
> >> > > > > let
> >> > > > > > > us
> >> > > > > > > >>> > know.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2)
> >> Adding
> >> > > Apex
> >> > > > > as
> >> > > > > > > >>> > component
> >> > > > > > > >>> > >> to
> >> > > > > > > >>> > >> > > >> Bigtop*
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of
> >> the
> >> > > mail
> >> > > > > > > >>> archive of
> >> > > > > > > >>> > >> > Bigtop,
> >> > > > > > > >>> > >> > > >> it
> >> > > > > > > >>> > >> > > >> > was
> >> > > > > > > >>> > >> > > >> > > >> >> learnt
> >> > > > > > > >>> > >> > > >> > > >> >> > > that
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community
> >> want
> >> > to
> >> > > > see
> >> > > > > > the
> >> > > > > > > >>> > >> addition of
> >> > > > > > > >>> > >> > > new
> >> > > > > > > >>> > >> > > >> > > >> >> components
> >> > > > > > > >>> > >> > > >> > > >> >> > in
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > phases.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Here
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > are
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1.
> Packaging
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This
> >> needs
> >> > > > > > creating
> >> > > > > > > of
> >> > > > > > > >>> > >> package
> >> > > > > > > >>> > >> > > i.e.
> >> > > > > > > >>> > >> > > >> > rpm
> >> > > > > > > >>> > >> > > >> > > &
> >> > > > > > > >>> > >> > > >> > > >> deb
> >> > > > > > > >>> > >> > > >> > > >> >> > > files.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> >> > > > > > > >>> documentations/READMEs,
> >> > > > > > > >>> > >> > LICENSE,
> >> > > > > > > >>> > >> > > >> > > >> DISCLAMER,
> >> > > > > > > >>> > >> > > >> > > >> >> > NOTES
> >> > > > > > > >>> > >> > > >> > > >> >> > > > etc
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > if
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > any
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> >> > > > > documentation
> >> > > > > > > that
> >> > > > > > > >>> > need
> >> > > > > > > >>> > >> to
> >> > > > > > > >>> > >> > be
> >> > > > > > > >>> > >> > > >> > added
> >> > > > > > > >>> > >> > > >> > > to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > distribution
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > our
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> >> > license
> >> > > > > > > >>> information of
> >> > > > > > > >>> > >> > > >> > dependencies
> >> > > > > > > >>> > >> > > >> > > >> >> required
> >> > > > > > > >>> > >> > > >> > > >> >> > > to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > be
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > added
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke
> >> tests
> >> > (at
> >> > > > > very
> >> > > > > > > >>> least)
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> Adding
> >> > smoke
> >> > > > > test
> >> > > > > > > for
> >> > > > > > > >>> > >> > packaging.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet
> >> > recipes
> >> > > > for
> >> > > > > > > >>> automatic
> >> > > > > > > >>> > >> > > >> deployment
> >> > > > > > > >>> > >> > > >> > and
> >> > > > > > > >>> > >> > > >> > > >> >> > > > configuration
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add
> >> puppet
> >> > > > > recipes
> >> > > > > > > for
> >> > > > > > > >>> > >> > automatic
> >> > > > > > > >>> > >> > > >> > > >> deployment
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4.
> >> Integration
> >> > > tests
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For
> >> > > > verification
> >> > > > > > of
> >> > > > > > > >>> > >> > deployments.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license
> >> > > > clearance:
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run
> >> 'gradle
> >> > > rat'
> >> > > > > to
> >> > > > > > > make
> >> > > > > > > >>> > sure
> >> > > > > > > >>> > >> all
> >> > > > > > > >>> > >> > > new
> >> > > > > > > >>> > >> > > >> > > stuff
> >> > > > > > > >>> > >> > > >> > > >> is
> >> > > > > > > >>> > >> > > >> > > >> >> > > > compliant
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > with
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license
> >> > > > requirements.
> >> > > > > If
> >> > > > > > > >>> you add
> >> > > > > > > >>> > >> code
> >> > > > > > > >>> > >> > > >> > licenses
> >> > > > > > > >>> > >> > > >> > > >> >> under
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > different
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would
> >> need
> >> > > to
> >> > > > be
> >> > > > > > > >>> listed in
> >> > > > > > > >>> > >> the
> >> > > > > > > >>> > >> > > >> NOTICE.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please
> share
> >> > your
> >> > > > > > thoughts
> >> > > > > > > >>> on
> >> > > > > > > >>> > the
> >> > > > > > > >>> > >> > > >> approach.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
> >> > > > > > corresponding
> >> > > > > > > >>> > >> > communication
> >> > > > > > > >>> > >> > > >> on
> >> > > > > > > >>> > >> > > >> > > >> bigtop
> >> > > > > > > >>> > >> > > >> > > >> >> > > mailing
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > list
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > as
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have
> some
> >> > > > specific
> >> > > > > > > >>> > >> > > >> questions/suggestions
> >> > > > > > > >>> > >> > > >> > > >> >> related to
> >> > > > > > > >>> > >> > > >> > > >> >> > > > what
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > should
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > be
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of
> >> the
> >> > > > package
> >> > > > > > and
> >> > > > > > > >>> what
> >> > > > > > > >>> > >> > should
> >> > > > > > > >>> > >> > > be
> >> > > > > > > >>> > >> > > >> > the
> >> > > > > > > >>> > >> > > >> > > >> smoke
> >> > > > > > > >>> > >> > > >> > > >> >> > > tests,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > but
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > in
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of
> >> not
> >> > > > having
> >> > > > > > too
> >> > > > > > > >>> much
> >> > > > > > > >>> > >> > content
> >> > > > > > > >>> > >> > > >> > here,
> >> > > > > > > >>> > >> > > >> > > >> we'll
> >> > > > > > > >>> > >> > > >> > > >> >> > put
> >> > > > > > > >>> > >> > > >> > > >> >> > > > the
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > questions
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate
> >> mail in
> >> > > > this
> >> > > > > > > >>> > mailthread.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > > >
> >> > > > > > > >>> > >> > > >> > > >> >> > >
> >> > > > > > > >>> > >> > > >> > > >> >> >
> >> > > > > > > >>> > >> > > >> > > >> >>
> >> > > > > > > >>> > >> > > >> > > >> >
> >> > > > > > > >>> > >> > > >> > > >> >
> >> > > > > > > >>> > >> > > >> > > >>
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > > >
> >> > > > > > > >>> > >> > > >> > >
> >> > > > > > > >>> > >> > > >> >
> >> > > > > > > >>> > >> > > >>
> >> > > > > > > >>> > >> > > >
> >> > > > > > > >>> > >> > > >
> >> > > > > > > >>> > >> > >
> >> > > > > > > >>> > >> >
> >> > > > > > > >>> > >>
> >> > > > > > > >>> > >
> >> > > > > > > >>> > >
> >> > > > > > > >>> >
> >> > > > > > > >>>
> >> > > > > > > >>
> >> > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Hello Everyone,

I believe there is no object on the smoke test to be added to bigtop for
apex package.
I've created smoke tests for bigtop as per above.

If there no further comments, I'll create a pull request for the same.

Thanks,
Chinmay.


On Fri, Feb 26, 2016 at 2:36 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Bigtop has consumed first patch for building apex packages into it's
> master branch.
> Commit mail:
>
> http://mail-archives.apache.org/mod_mbox/bigtop-commits/201602.mbox/%3C06c0ca62ac1b483d8ded090f0f2c8d73%40git.apache.org%3E
>
> What this means is bigtop master will be able to generate apex packages
> (3.3.0-incubating), both deb & rpm, when it build.
>
> For the second phase, we need to add smoke tests for our package in bigtop
> repo.
> This is an important steps for bigtop to host the generate package in
> bigtop repository during its 1.2.0 release.
>
> Here are my initial thoughts about smoke test:
> TEST::
> 1. Create mvn app with archetype
> 2. Build app
> 3. Run app with apex/dtcli
>
> VERIFICATION:
> 1. stdout file gets has random numbers per line.
>
> I see app created by archetype is randomNumberGenerator -> Console.
> I think this should be a good enough smoke test for apex package.
>
> Of course I need to check if we can do above with bigtop framework. But I
> have a hunch that we can, because there are standalone frameworks like us
> which has similar types of tests.
>
> Please share your opinion and thoughts on this.
>
> Thanks,
> Chinmay.
>
>
>
> On Fri, Feb 19, 2016 at 11:38 AM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
>> We should not package any Hadoop dependencies, they are provided by the
>> Bigtop stack.
>>
>>
>> On Thu, Feb 18, 2016 at 8:56 PM, Tushar Gosavi <tu...@datatorrent.com>
>> wrote:
>>
>> > I just did a quick install of bigtop 1.1.0 on ubuntu and observed that
>> >  jersey-client-1.9.jar is already packaged in hadoop
>> > /usr/lib/hadoop/client/jersey-client.jar, so I guess it will be ok to
>> > include those in apex lib too. or If we are including hadoop jars
>> > in classpath while launching the apex application then it may not be
>> > needed.
>> >
>> > - Tushar.
>> >
>> >
>> >
>> > On Fri, Feb 19, 2016 at 8:44 AM, Chinmay Kolhatkar <ch...@apache.org>
>> > wrote:
>> >
>> > > No worries David. I've updated the manpage.
>> > >
>> > > Just an update: apex bigtop packaging for rpm+deb is done. Here is the
>> > > branch on which I'm working if someone wants to look at:
>> > > https://github.com/chinmaykolhatkar/bigtop/blob/BIGTOP-2314
>> > >
>> > > Only the 3rd questions below is the question that is left to be
>> answered.
>> > > Once this is resolved, I'll create the pull request in bigtop.
>> > >
>> > > Following questions are related to licenses and inclusion of certain
>> > files
>> > > in apex rpm+deb:
>> > > 1. I'll be included LICENSE, DISCLAIMER, NOTICE from top folder of
>> core
>> > > repo in our rpm and deb package. Those will be present in standard
>> > location
>> > > i.e. /usr/lib/apex/
>> > > Please let me know otherwise.
>> > >
>> > > 2. Most of our runtime dependencies has ASF license, but certain
>> > libraries
>> > > has MIT and BSD licenses, hence I've included those licenses too in
>> rpm &
>> > > deb package in same location as above.
>> > > Please let me know otherwise.
>> > >
>> > > BSD - jline-2.11.jar, kryo-2.24.0.jar, minlog-1.2.jar
>> > > MIT - mbassador-1.1.9.jar
>> > >
>> > > 3. There are 2 jars (jersey-apache-client4-1.9.jar,
>> > jersey-client-1.9.jar)
>> > > which has duel license of CDDL + GPLv2 with classpath exception:
>> > > https://jersey.java.net/license.html
>> > > I'm not an expert here. Hence can someone let me know whether these
>> > > libraries is OK to be packaged in rpm+deb of apex?
>> > >
>> > > If yes, should we also include that license in our package?
>> > >
>> > > Thanks,
>> > > Chinmay.
>> > >
>> > >
>> > >
>> > > On Thu, Feb 18, 2016 at 11:35 PM, David Yan <da...@datatorrent.com>
>> > wrote:
>> > >
>> > > > I should've mentioned that I had a pull request that was merged
>> > yesterday
>> > > > that corrected the statement:
>> > > >
>> > > > https://github.com/DataTorrent/docs/pull/45
>> > > >
>> > > > On Thu, Feb 18, 2016 at 12:07 AM, Tushar Gosavi <
>> > tushar@datatorrent.com>
>> > > > wrote:
>> > > >
>> > > > > In following statement, can you replace dtGateway with
>> > > ApplicationMaster
>> > > > > and Yarn. or may be remove this statement entirely.
>> > > > > apex is a wrapper around the REST API provided by dtGatway,
>> > > > >
>> > > > > - Tushar.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <
>> > > chinmay@apache.org>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > I've attached man page for apex in Jira. Here is the link to it:
>> > > > > >
>> https://issues.apache.org/jira/secure/attachment/12788403/apex.1
>> > > > > >
>> > > > > > Please view it by: man -l apex.1
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Chinmay.
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <
>> > > > tushar@datatorrent.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Chinmay,
>> > > > > > >
>> > > > > > > I am not able to find the attachment, may be attachments are
>> not
>> > > > > allowed
>> > > > > > in
>> > > > > > > the mailing list. Can you attach the man page to the Jira?
>> > > > > > >
>> > > > > > > - Tushar.
>> > > > > > >
>> > > > > > >
>> > > > > > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <
>> > > > chinmay@apache.org
>> > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi All,
>> > > > > > > >
>> > > > > > > > I've attached man page file for the purpose of bigtop
>> > > integration.
>> > > > > > > > Please have a look and let me know if there any any feedback
>> > > about
>> > > > > it.
>> > > > > > > >
>> > > > > > > > You can see the man page content of this file with:
>> > > > > > > > man -l apex.1
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Chinmay.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
>> > > > > > chinmay@apache.org>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > >> Thanks. Will follow bigtop recommendation. i.e. remove
>> > > > > "-incubating".
>> > > > > > > >>
>> > > > > > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
>> > > > > thomas@datatorrent.com
>> > > > > > >
>> > > > > > > >> wrote:
>> > > > > > > >>
>> > > > > > > >>> Please follow the recommendation and continue discussion
>> on
>> > the
>> > > > > > Bigtop
>> > > > > > > >>> list.
>> > > > > > > >>>
>> > > > > > > >>> --
>> > > > > > > >>> sent from mobile
>> > > > > > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <
>> > > chinmay@apache.org
>> > > > >
>> > > > > > > wrote:
>> > > > > > > >>>
>> > > > > > > >>> > I got a recommendation from bigtop community to drop
>> > > > > "-incubating"
>> > > > > > > from
>> > > > > > > >>> > version.
>> > > > > > > >>> >
>> > > > > > > >>> > Here is the mail thread:
>> > > > > > > >>> >
>> > > > > > > >>> >
>> > > > > > > >>>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
>> > > > > > > >>> >
>> > > > > > > >>> > Please let me know if removing "-incubating" is fine.
>> > > > > > > >>> >
>> > > > > > > >>> > Thanks,
>> > > > > > > >>> > Chinmay.
>> > > > > > > >>> >
>> > > > > > > >>> >
>> > > > > > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
>> > > > > > > chinmay@apache.org
>> > > > > > > >>> >
>> > > > > > > >>> > wrote:
>> > > > > > > >>> >
>> > > > > > > >>> > > I've already started communication there.
>> > > > > > > >>> > >
>> > > > > > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
>> > > > > > > >>> thomas@datatorrent.com>
>> > > > > > > >>> > > wrote:
>> > > > > > > >>> > >
>> > > > > > > >>> > >> This is a question you should ask on the Bigtop
>> mailing
>> > > > list.
>> > > > > > > >>> > >>
>> > > > > > > >>> > >> --
>> > > > > > > >>> > >> sent from mobile
>> > > > > > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <
>> > > > prad@apache.org>
>> > > > > > > >>> wrote:
>> > > > > > > >>> > >>
>> > > > > > > >>> > >> > Yes. In case of RPM, the only restriction on
>> version
>> > is
>> > > > that
>> > > > > > it
>> > > > > > > >>> can
>> > > > > > > >>> > not
>> > > > > > > >>> > >> > contain '-'.
>> > > > > > > >>> > >> > However, as a workaround, one can use 'incubating'
>> as
>> > a
>> > > > part
>> > > > > > of
>> > > > > > > >>> > >> %{release}.
>> > > > > > > >>> > >> >
>> > > > > > > >>> > >> > Version: 3.3.0
>> > > > > > > >>> > >> > Release: incubating
>> > > > > > > >>> > >> >
>> > > > > > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
>> > > > > > > >>> > >> >
>> > > > > > > >>> > >> > It is the same practice followed for pre-release
>> > > packages
>> > > > > e.g.
>> > > > > > > >>> with
>> > > > > > > >>> > rcX,
>> > > > > > > >>> > >> > alphaX, betaX being used in %{release}.
>> > > > > > > >>> > >> >
>> > > > > > > >>> > >> > Regards,
>> > > > > > > >>> > >> > Pradeep A. Dalvi
>> > > > > > > >>> > >> >
>> > > > > > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar
>> <
>> > > > > > > >>> > chinmay@apache.org>
>> > > > > > > >>> > >> > wrote:
>> > > > > > > >>> > >> >
>> > > > > > > >>> > >> > > Hi Everyone,
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> > > In one of the previous mails in this thread, we
>> > > decided
>> > > > > that
>> > > > > > > >>> version
>> > > > > > > >>> > >> of
>> > > > > > > >>> > >> > the
>> > > > > > > >>> > >> > > package will be "3.3.0-incubating".
>> > > > > > > >>> > >> > > The problem that I'm facing is in RPM base
>> version
>> > "-"
>> > > > is
>> > > > > > not
>> > > > > > > >>> > allowed.
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> > > Can we have package version to "3.3.0"?
>> > > > > > > >>> > >> > > i.e. the packages created will be as follows:
>> > > > > > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
>> > > > > > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> > > There is another integration for Zepplin in
>> Bigtop
>> > > which
>> > > > > is
>> > > > > > > >>> > incubating
>> > > > > > > >>> > >> > and
>> > > > > > > >>> > >> > > is having base version as "0.5.6".
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> > > Please let me know your thoughts.
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> > > Thanks,
>> > > > > > > >>> > >> > > Chinmay.
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay
>> Kolhatkar
>> > <
>> > > > > > > >>> > >> chinmay@apache.org>
>> > > > > > > >>> > >> > > wrote:
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> > > > Thanks everyone for the feedback.
>> > > > > > > >>> > >> > > >
>> > > > > > > >>> > >> > > > Also as Ram and David mentioned, I'll start
>> > creating
>> > > > the
>> > > > > > man
>> > > > > > > >>> page
>> > > > > > > >>> > as
>> > > > > > > >>> > >> > > well.
>> > > > > > > >>> > >> > > >
>> > > > > > > >>> > >> > > > Thanks,
>> > > > > > > >>> > >> > > > Chinmay.
>> > > > > > > >>> > >> > > >
>> > > > > > > >>> > >> > > >
>> > > > > > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
>> > > > > > > >>> > >> thomas@datatorrent.com>
>> > > > > > > >>> > >> > > > wrote:
>> > > > > > > >>> > >> > > >
>> > > > > > > >>> > >> > > >> I think we already agreed that we will retain
>> the
>> > > > > current
>> > > > > > > >>> script
>> > > > > > > >>> > >> name
>> > > > > > > >>> > >> > in
>> > > > > > > >>> > >> > > >> this bigtop iteration. There is a ticket for
>> > > > renaming,
>> > > > > > add
>> > > > > > > >>> you
>> > > > > > > >>> > >> > > suggestions
>> > > > > > > >>> > >> > > >> there.
>> > > > > > > >>> > >> > > >>
>> > > > > > > >>> > >> > > >> --
>> > > > > > > >>> > >> > > >> sent from mobile
>> > > > > > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
>> > > > > > > >>> > sandesh@datatorrent.com>
>> > > > > > > >>> > >> > > wrote:
>> > > > > > > >>> > >> > > >>
>> > > > > > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start
>> the
>> > > > > > transition
>> > > > > > > >>> > >> process by
>> > > > > > > >>> > >> > > >> making
>> > > > > > > >>> > >> > > >> > dtcli as the symlink to apex cli.
>> > > > > > > >>> > >> > > >> >
>> > > > > > > >>> > >> > > >> > Renaming is a breaking change should be done
>> > > during
>> > > > > > major
>> > > > > > > >>> > version
>> > > > > > > >>> > >> > > number
>> > > > > > > >>> > >> > > >> > change.
>> > > > > > > >>> > >> > > >> >
>> > > > > > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
>> > > > > > > >>> > david@datatorrent.com
>> > > > > > > >>> > >> >
>> > > > > > > >>> > >> > > >> wrote:
>> > > > > > > >>> > >> > > >> >
>> > > > > > > >>> > >> > > >> > > Apologies for hitting the "Send" button
>> too
>> > > > early.
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > What is the scope of the name change?  If
>> we
>> > > > simply
>> > > > > > > have
>> > > > > > > >>> a
>> > > > > > > >>> > >> symlink
>> > > > > > > >>> > >> > > >> from
>> > > > > > > >>> > >> > > >> > > apex to dtcli, then it should have no
>> > backward
>> > > > > > > >>> compatibility
>> > > > > > > >>> > >> > problem
>> > > > > > > >>> > >> > > >> at
>> > > > > > > >>> > >> > > >> > > all.
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > But the proper name change would involve:
>> > > > > > > >>> > >> > > >> > > - Change the name of DTCli class to
>> ApexCli
>> > > > > > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
>> > > > > > > >>> > >> > > >> > > - Change the initial greeting when running
>> > the
>> > > > CLI
>> > > > > > from
>> > > > > > > >>> > >> something
>> > > > > > > >>> > >> > > like
>> > > > > > > >>> > >> > > >> > "DT
>> > > > > > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI
>> ...".
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > The above changes may break existing
>> scripts
>> > > that
>> > > > > > rely
>> > > > > > > >>> on the
>> > > > > > > >>> > >> the
>> > > > > > > >>> > >> > > >> string
>> > > > > > > >>> > >> > > >> > > "DT".
>> > > > > > > >>> > >> > > >> > > But if folks are okay with this potential
>> > > > problem,
>> > > > > I
>> > > > > > am
>> > > > > > > >>> okay
>> > > > > > > >>> > >> with
>> > > > > > > >>> > >> > > the
>> > > > > > > >>> > >> > > >> > name
>> > > > > > > >>> > >> > > >> > > change.
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > For the man page, we can start with the
>> > content
>> > > > of
>> > > > > > this
>> > > > > > > >>> page:
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > Plus the command line options when doing
>> > dtcli
>> > > > -h:
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > usage: DTCli
>> > > > > > > >>> > >> > > >> > >  -e <arg>    Commands are read from the
>> > > argument
>> > > > > > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at
>> all
>> > > time
>> > > > > > > >>> > >> > > >> > >  -h          Print this help
>> > > > > > > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos
>> > > principal
>> > > > > > > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos
>> > keytab
>> > > > > > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
>> > > > > > > >>> > >> > > >> > >  -r          JSON Raw mode
>> > > > > > > >>> > >> > > >> > >  -v          Verbose mode level 1
>> > > > > > > >>> > >> > > >> > >  -vv         Verbose mode level 2
>> > > > > > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
>> > > > > > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > David
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David
>> Yan <
>> > > > > > > >>> > >> david@datatorrent.com
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> > > >> > wrote:
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > > What is the scope of the name change?
>> If
>> > we
>> > > > > simply
>> > > > > > > >>> have a
>> > > > > > > >>> > >> > symlink
>> > > > > > > >>> > >> > > >> from
>> > > > > > > >>> > >> > > >> > > > apex to dtcli, then it should have no
>> > > backward
>> > > > > > > >>> > compatibility
>> > > > > > > >>> > >> > > >> problem at
>> > > > > > > >>> > >> > > >> > > all.
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > > > But the proper name change would
>> involve:
>> > > > > > > >>> > >> > > >> > > > - Change the name of DTCli class to
>> ApexCli
>> > > > > > > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
>> > > > > > > >>> > >> > > >> > > > - Change the initial greeting when
>> running
>> > > the
>> > > > > CLI
>> > > > > > > from
>> > > > > > > >>> > >> > something
>> > > > > > > >>> > >> > > >> like
>> > > > > > > >>> > >> > > >> > > "DT
>> > > > > > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI
>> > ...".
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > > > The above changes may break existing
>> > scripts
>> > > > that
>> > > > > > > rely
>> > > > > > > >>> on
>> > > > > > > >>> > the
>> > > > > > > >>> > >> > the
>> > > > > > > >>> > >> > > >> > string
>> > > > > > > >>> > >> > > >> > > > "DT".
>> > > > > > > >>> > >> > > >> > > > But if folks are okay with this
>> potential
>> > > > > problem,
>> > > > > > I
>> > > > > > > am
>> > > > > > > >>> > okay
>> > > > > > > >>> > >> > with
>> > > > > > > >>> > >> > > >> the
>> > > > > > > >>> > >> > > >> > > name
>> > > > > > > >>> > >> > > >> > > > change.
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > > > For the man page, we can start with the
>> > > content
>> > > > > of
>> > > > > > > this
>> > > > > > > >>> > page:
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay
>> > > > > Kolhatkar
>> > > > > > <
>> > > > > > > >>> > >> > > >> chinmay@apache.org
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> > > > wrote:
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > > >> Here is a suggestion I have related to
>> > name
>> > > > > > change:
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > > >>> > >> > > >> > > >> 1. While packaging we change the name
>> of
>> > > > "dtcli"
>> > > > > > > >>> script to
>> > > > > > > >>> > >> > > "apex".
>> > > > > > > >>> > >> > > >> > This
>> > > > > > > >>> > >> > > >> > > is
>> > > > > > > >>> > >> > > >> > > >> done so that in future when we change
>> the
>> > > name
>> > > > > to
>> > > > > > > >>> "apex",
>> > > > > > > >>> > >> users
>> > > > > > > >>> > >> > > of
>> > > > > > > >>> > >> > > >> > > bigtop
>> > > > > > > >>> > >> > > >> > > >> apex don't have to transition much.
>> > > > > > > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli
>> > which
>> > > > > points
>> > > > > > > to
>> > > > > > > >>> apex
>> > > > > > > >>> > >> > > script.
>> > > > > > > >>> > >> > > >> > This
>> > > > > > > >>> > >> > > >> > > is
>> > > > > > > >>> > >> > > >> > > >> for backward compatibility.
>> > > > > > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for
>> first
>> > > > > > > integration
>> > > > > > > >>> in
>> > > > > > > >>> > >> > bigtop.
>> > > > > > > >>> > >> > > >> This
>> > > > > > > >>> > >> > > >> > is
>> > > > > > > >>> > >> > > >> > > >> in
>> > > > > > > >>> > >> > > >> > > >> the interest of not changing the
>> content
>> > of
>> > > > > dtcli
>> > > > > > or
>> > > > > > > >>> apex
>> > > > > > > >>> > >> > script.
>> > > > > > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all
>> > these
>> > > > > naming
>> > > > > > > >>> related
>> > > > > > > >>> > >> > > changes
>> > > > > > > >>> > >> > > >> and
>> > > > > > > >>> > >> > > >> > > >> update bigtop repository later to
>> remove
>> > > > > > references
>> > > > > > > to
>> > > > > > > >>> > dtcli
>> > > > > > > >>> > >> > all
>> > > > > > > >>> > >> > > >> > > together.
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > > >>> > >> > > >> > > >> Please share your thoughts on above
>> > > approach.
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > > >>> > >> > > >> > > >> Also, please share what could be the
>> man
>> > > page
>> > > > > > > content
>> > > > > > > >>> for
>> > > > > > > >>> > >> > "apex".
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > > >>> > >> > > >> > > >> Thanks,
>> > > > > > > >>> > >> > > >> > > >> Chinmay.
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM,
>> Chinmay
>> > > > > > Kolhatkar <
>> > > > > > > >>> > >> > > >> > chinmay@apache.org>
>> > > > > > > >>> > >> > > >> > > >> wrote:
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > > >>> > >> > > >> > > >> > Yes.. It'll be build from source tar
>> > > > > downloaded
>> > > > > > > >>> from one
>> > > > > > > >>> > >> of
>> > > > > > > >>> > >> > the
>> > > > > > > >>> > >> > > >> > apache
>> > > > > > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
>> > > > > > > >>> > >> > > >> > > >> >
>> > > > > > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's
>> sourced
>> > > > from
>> > > > > > > dtcli.
>> > > > > > > >>> > This
>> > > > > > > >>> > >> > > leaves
>> > > > > > > >>> > >> > > >> us
>> > > > > > > >>> > >> > > >> > 3
>> > > > > > > >>> > >> > > >> > > >> > options:
>> > > > > > > >>> > >> > > >> > > >> > If we should not have any dt-env.sh,
>> > dtcli
>> > > > > would
>> > > > > > > >>> need a
>> > > > > > > >>> > >> > change
>> > > > > > > >>> > >> > > >> while
>> > > > > > > >>> > >> > > >> > > >> > packaging.
>> > > > > > > >>> > >> > > >> > > >> >
>> > > > > > > >>> > >> > > >> > > >> > But if we want to use dtcli as it is,
>> > then
>> > > > we
>> > > > > > > would
>> > > > > > > >>> need
>> > > > > > > >>> > >> > > >> dt-env.sh
>> > > > > > > >>> > >> > > >> > > >> >
>> > > > > > > >>> > >> > > >> > > >> >
>> > > > > > > >>> > >> > > >> > > >> >
>> > > > > > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM,
>> Thomas
>> > > > Weise
>> > > > > <
>> > > > > > > >>> > >> > > >> > thomas@datatorrent.com
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > > >> > wrote:
>> > > > > > > >>> > >> > > >> > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> Looks good overall, though there
>> > > shouldn't
>> > > > be
>> > > > > > any
>> > > > > > > >>> > >> dt-env.sh
>> > > > > > > >>> > >> > > >> > > >> >>
>> > > > > > > >>> > >> > > >> > > >> >> What will the bigtop package be
>> built
>> > > from,
>> > > > > the
>> > > > > > > >>> source
>> > > > > > > >>> > >> tar?
>> > > > > > > >>> > >> > > >> > > >> >>
>> > > > > > > >>> > >> > > >> > > >> >>
>> > > > > > > >>> > >> > > >> > > >> >>
>> > > > > > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM,
>> > Chinmay
>> > > > > > > Kolhatkar
>> > > > > > > >>> <
>> > > > > > > >>> > >> > > >> > > chinmay@apache.org
>> > > > > > > >>> > >> > > >> > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> wrote:
>> > > > > > > >>> > >> > > >> > > >> >>
>> > > > > > > >>> > >> > > >> > > >> >> > Hi Thomas,
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > First, we're not changing the name
>> > > > > anywhere.
>> > > > > > > >>> We'll
>> > > > > > > >>> > >> follow
>> > > > > > > >>> > >> > > what
>> > > > > > > >>> > >> > > >> > > >> >> currently is
>> > > > > > > >>> > >> > > >> > > >> >> > in source tarball for
>> > 3.3.0-incubating
>> > > > > > version.
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a
>> directory
>> > > > > > structure
>> > > > > > > >>> below
>> > > > > > > >>> > >> which
>> > > > > > > >>> > >> > > is
>> > > > > > > >>> > >> > > >> > > inline
>> > > > > > > >>> > >> > > >> > > >> >> with
>> > > > > > > >>> > >> > > >> > > >> >> > other existing integrations with
>> > > bigtop.
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > The need for each file is as
>> follows:
>> > > > > > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex
>> picked
>> > > from
>> > > > > > > >>> > >> > > >> engine/src/main/scripts/
>> > > > > > > >>> > >> > > >> > > of
>> > > > > > > >>> > >> > > >> > > >> >> source
>> > > > > > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is
>> source
>> > > from
>> > > > > > dtcli.
>> > > > > > > >>> This
>> > > > > > > >>> > >> > > searches
>> > > > > > > >>> > >> > > >> for
>> > > > > > > >>> > >> > > >> > > >> hadoop
>> > > > > > > >>> > >> > > >> > > >> >> > binary path and exports an env
>> > variable
>> > > > for
>> > > > > > > >>> dtcli to
>> > > > > > > >>> > >> use
>> > > > > > > >>> > >> > it.
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This
>> > list
>> > > > is
>> > > > > > > >>> extracted
>> > > > > > > >>> > >> in a
>> > > > > > > >>> > >> > > >> similar
>> > > > > > > >>> > >> > > >> > > >> way to
>> > > > > > > >>> > >> > > >> > > >> >> > how DT Community Edition finds
>> > > dependency
>> > > > > > jars.
>> > > > > > > >>> > >> > > >> > > >> >> > Please note that this is the first
>> > > > > iteration
>> > > > > > > >>> list of
>> > > > > > > >>> > >> jars.
>> > > > > > > >>> > >> > > I'm
>> > > > > > > >>> > >> > > >> > > >> trying to
>> > > > > > > >>> > >> > > >> > > >> >> > narrow this down to only those
>> which
>> > > are
>> > > > > > really
>> > > > > > > >>> > >> required.
>> > > > > > > >>> > >> > > >> > > >> >> > The test that I'm running to
>> check if
>> > > > dtcli
>> > > > > > > runs
>> > > > > > > >>> fine
>> > > > > > > >>> > >> with
>> > > > > > > >>> > >> > > >> given
>> > > > > > > >>> > >> > > >> > > >> >> dependency
>> > > > > > > >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status
>> for
>> > > pi
>> > > > > > demo.
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > Above is the directory structure
>> > which
>> > > is
>> > > > > > > >>> required
>> > > > > > > >>> > for
>> > > > > > > >>> > >> CLI
>> > > > > > > >>> > >> > > to
>> > > > > > > >>> > >> > > >> > work.
>> > > > > > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has
>> > > mentioned
>> > > > > are
>> > > > > > > some
>> > > > > > > >>> > >> > additional
>> > > > > > > >>> > >> > > >> > files
>> > > > > > > >>> > >> > > >> > > >> which
>> > > > > > > >>> > >> > > >> > > >> >> > makes the like of administrator
>> > easier.
>> > > > For
>> > > > > > eg.
>> > > > > > > >>> > >> /etc/skel
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > Please let me know if this is
>> inline
>> > > with
>> > > > > > what
>> > > > > > > >>> you're
>> > > > > > > >>> > >> > > >> thinking.
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > Thanks,
>> > > > > > > >>> > >> > > >> > > >> >> > Chinmay.
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > .
>> > > > > > > >>> > >> > > >> > > >> >> > |-- bin
>> > > > > > > >>> > >> > > >> > > >> >> > |   `-- dtcli
>> > > > > > > >>> > >> > > >> > > >> >> > |-- conf
>> > > > > > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
>> > > > > > > >>> > >> > > >> > > >> >> > `-- lib
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> apex-api-3.3.0-incubating.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> > > > > > apex-bufferserver-3.3.0-incubating.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> > > apex-common-3.3.0-incubating.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> async-http-client-1.7.20.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> commons-beanutils-1.8.3.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> > grizzly-http-servlet-2.1.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> hadoop-common-2.2.0-tests.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> jackson-mapper-asl-1.9.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> javax.servlet-api-3.0.1.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> jersey-apache-client4-1.9.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> > jetty-http-8.1.10.v20130312.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> jetty-io-8.1.10.v20130312.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> > jetty-util-8.1.10.v20130312.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> > > > > jetty-websocket-8.1.10.v20130312.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |--
>> > validation-api-1.1.0.Final.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
>> > > > > > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM,
>> > Thomas
>> > > > > > Weise <
>> > > > > > > >>> > >> > > >> > > >> thomas@datatorrent.com>
>> > > > > > > >>> > >> > > >> > > >> >> > wrote:
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > Chinmay,
>> > > > > > > >>> > >> > > >> > > >> >> > >
>> > > > > > > >>> > >> > > >> > > >> >> > > Before discussing where to put
>> the
>> > > > files,
>> > > > > > > let's
>> > > > > > > >>> > make
>> > > > > > > >>> > >> > sure
>> > > > > > > >>> > >> > > >> they
>> > > > > > > >>> > >> > > >> > > are
>> > > > > > > >>> > >> > > >> > > >> >> really
>> > > > > > > >>> > >> > > >> > > >> >> > > needed for the operation of the
>> > CLI.
>> > > As
>> > > > > for
>> > > > > > > >>> names,
>> > > > > > > >>> > >> > > anything
>> > > > > > > >>> > >> > > >> new
>> > > > > > > >>> > >> > > >> > > >> needs
>> > > > > > > >>> > >> > > >> > > >> >> to
>> > > > > > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and
>> should
>> > > > > follow
>> > > > > > > >>> common
>> > > > > > > >>> > >> > > >> conventions,
>> > > > > > > >>> > >> > > >> > > >> >> especially
>> > > > > > > >>> > >> > > >> > > >> >> > > in Bigtop where there are many
>> > > existing
>> > > > > > > >>> > integrations
>> > > > > > > >>> > >> to
>> > > > > > > >>> > >> > > look
>> > > > > > > >>> > >> > > >> > at.
>> > > > > > > >>> > >> > > >> > > >> >> > >
>> > > > > > > >>> > >> > > >> > > >> >> > > Thanks,
>> > > > > > > >>> > >> > > >> > > >> >> > > Thomas
>> > > > > > > >>> > >> > > >> > > >> >> > >
>> > > > > > > >>> > >> > > >> > > >> >> > >
>> > > > > > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM,
>> > > > Chinmay
>> > > > > > > >>> Kolhatkar
>> > > > > > > >>> > <
>> > > > > > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
>> > > > > > > >>> > >> > > >> > > >> >> > > wrote:
>> > > > > > > >>> > >> > > >> > > >> >> > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep and
>> > > > > Aniruddha.
>> > > > > > > >>> > >> > > >> > > >> >> > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to
>> > > change
>> > > > > the
>> > > > > > > >>> dtcli
>> > > > > > > >>> > >> > > >> considering it
>> > > > > > > >>> > >> > > >> > > >> works
>> > > > > > > >>> > >> > > >> > > >> >> > with
>> > > > > > > >>> > >> > > >> > > >> >> > > DT
>> > > > > > > >>> > >> > > >> > > >> >> > > > Community edition. We can keep
>> > the
>> > > > > > > directory
>> > > > > > > >>> > >> structure
>> > > > > > > >>> > >> > > >> > similar
>> > > > > > > >>> > >> > > >> > > to
>> > > > > > > >>> > >> > > >> > > >> >> that.
>> > > > > > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which
>> > > > contains
>> > > > > > > >>> > information
>> > > > > > > >>> > >> > > >> required
>> > > > > > > >>> > >> > > >> > for
>> > > > > > > >>> > >> > > >> > > >> >> dtcli
>> > > > > > > >>> > >> > > >> > > >> >> > to
>> > > > > > > >>> > >> > > >> > > >> >> > > > launch.
>> > > > > > > >>> > >> > > >> > > >> >> > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop
>> > > community
>> > > > > > that
>> > > > > > > >>> > whether
>> > > > > > > >>> > >> > they
>> > > > > > > >>> > >> > > >> > > >> facilitate
>> > > > > > > >>> > >> > > >> > > >> >> the
>> > > > > > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs
>> before
>> > > user
>> > > > > is
>> > > > > > > >>> created.
>> > > > > > > >>> > >> In
>> > > > > > > >>> > >> > > >> either
>> > > > > > > >>> > >> > > >> > > case,
>> > > > > > > >>> > >> > > >> > > >> >> > current
>> > > > > > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in
>> > home
>> > > > > > > directory.
>> > > > > > > >>> > Also
>> > > > > > > >>> > >> > > before
>> > > > > > > >>> > >> > > >> > > putting
>> > > > > > > >>> > >> > > >> > > >> >> > > anything
>> > > > > > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define
>> > what
>> > > > are
>> > > > > > the
>> > > > > > > >>> > default
>> > > > > > > >>> > >> > > >> contents
>> > > > > > > >>> > >> > > >> > > that
>> > > > > > > >>> > >> > > >> > > >> >> > should
>> > > > > > > >>> > >> > > >> > > >> >> > > go
>> > > > > > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is
>> no
>> > > > > > defaults,
>> > > > > > > >>> > >> probably we
>> > > > > > > >>> > >> > > >> should
>> > > > > > > >>> > >> > > >> > > not
>> > > > > > > >>> > >> > > >> > > >> >> > > > explicitly add it in
>> /etc/skel.
>> > > > > > > >>> > >> > > >> > > >> >> > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach
>> looks
>> > > > nice.
>> > > > > > > This
>> > > > > > > >>> way
>> > > > > > > >>> > >> > > contents
>> > > > > > > >>> > >> > > >> of
>> > > > > > > >>> > >> > > >> > > >> >> dt-env.sh
>> > > > > > > >>> > >> > > >> > > >> >> > > are
>> > > > > > > >>> > >> > > >> > > >> >> > > > present as env variables. But
>> I
>> > > see a
>> > > > > > catch
>> > > > > > > >>> > there.
>> > > > > > > >>> > >> > > Adding
>> > > > > > > >>> > >> > > >> > > >> dt-env.sh
>> > > > > > > >>> > >> > > >> > > >> >> to
>> > > > > > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all
>> the
>> > > > > > variables
>> > > > > > > >>> > >> available
>> > > > > > > >>> > >> > at
>> > > > > > > >>> > >> > > >> > > runtime
>> > > > > > > >>> > >> > > >> > > >> all
>> > > > > > > >>> > >> > > >> > > >> >> > the
>> > > > > > > >>> > >> > > >> > > >> >> > > > time. I feel a little
>> skeptical
>> > > about
>> > > > > > that.
>> > > > > > > >>> > Maybe a
>> > > > > > > >>> > >> > > >> possible
>> > > > > > > >>> > >> > > >> > > >> >> collision
>> > > > > > > >>> > >> > > >> > > >> >> > > can
>> > > > > > > >>> > >> > > >> > > >> >> > > > occur with other application
>> > vars.
>> > > > > > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does
>> > source
>> > > > > > > >>> > >> > "../conf/dt-env.sh"
>> > > > > > > >>> > >> > > >> and
>> > > > > > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those
>> > > variables
>> > > > > are
>> > > > > > > >>> anyway
>> > > > > > > >>> > >> > > >> available.
>> > > > > > > >>> > >> > > >> > > >> >> > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR
>> > effort
>> > > is
>> > > > > > > >>> happening
>> > > > > > > >>> > at
>> > > > > > > >>> > >> > > Bigtop
>> > > > > > > >>> > >> > > >> > > >> >> > > Jira/repository.
>> > > > > > > >>> > >> > > >> > > >> >> > > > Packaging related code usually
>> > goes
>> > > > > > there.
>> > > > > > > >>> > (That's
>> > > > > > > >>> > >> > what
>> > > > > > > >>> > >> > > >> all
>> > > > > > > >>> > >> > > >> > the
>> > > > > > > >>> > >> > > >> > > >> >> > > components
>> > > > > > > >>> > >> > > >> > > >> >> > > > in bigtop does).
>> > > > > > > >>> > >> > > >> > > >> >> > > > Having said that, once a PR is
>> > > > created,
>> > > > > > > I'll
>> > > > > > > >>> be
>> > > > > > > >>> > >> > sharing
>> > > > > > > >>> > >> > > >> the
>> > > > > > > >>> > >> > > >> > > link
>> > > > > > > >>> > >> > > >> > > >> of
>> > > > > > > >>> > >> > > >> > > >> >> PR
>> > > > > > > >>> > >> > > >> > > >> >> > > > here, so that apex community
>> as
>> > > well
>> > > > > can
>> > > > > > > >>> review
>> > > > > > > >>> > it.
>> > > > > > > >>> > >> > > >> > > >> >> > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > Thanks,
>> > > > > > > >>> > >> > > >> > > >> >> > > > Chinmay.
>> > > > > > > >>> > >> > > >> > > >> >> > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57
>> PM,
>> > > > > > Aniruddha
>> > > > > > > >>> > >> Thombare <
>> > > > > > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com>
>> > wrote:
>> > > > > > > >>> > >> > > >> > > >> >> > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and
>> approach.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > We may need to iron out
>> details
>> > > > about
>> > > > > > > exact
>> > > > > > > >>> > paths
>> > > > > > > >>> > >> > etc.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > Which can be done on jira /
>> PR
>> > > > > > comments.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
>> > > > > > > >>> > >> > > >> > > >> >> > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm
>> > > Pradeep
>> > > > > A.
>> > > > > > > >>> Dalvi <
>> > > > > > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > wrote:
>> > > > > > > >>> > >> > > >> > > >> >> > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at
>> 2:51
>> > > PM,
>> > > > > > > Chinmay
>> > > > > > > >>> > >> > Kolhatkar <
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > wrote:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar
>> > > > present
>> > > > > in
>> > > > > > > DT
>> > > > > > > >>> > >> community
>> > > > > > > >>> > >> > > >> > edition
>> > > > > > > >>> > >> > > >> > > >> will
>> > > > > > > >>> > >> > > >> > > >> >> be
>> > > > > > > >>> > >> > > >> > > >> >> > > > > included
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > there.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of
>> jars I
>> > > > found
>> > > > > > from
>> > > > > > > >>> > common
>> > > > > > > >>> > >> > > >> > dependencies
>> > > > > > > >>> > >> > > >> > > >> >> between
>> > > > > > > >>> > >> > > >> > > >> >> > > of
>> > > > > > > >>> > >> > > >> > > >> >> > > > > apex
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > and DT Community
>> edition:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> jackson-core-asl-1.9.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > jackson-mapper-asl-1.9.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > async-http-client-1.7.20.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > validation-api-1.1.0.Final.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> commons-beanutils-1.8.3.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > jetty-websocket-8.1.10.v20130312.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> xbean-asm5-shaded-4.3.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > jersey-apache-client4-1.9.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > hadoop-common-2.2.0-tests.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be
>> running
>> > > some
>> > > > > > tests
>> > > > > > > do
>> > > > > > > >>> > check
>> > > > > > > >>> > >> > that
>> > > > > > > >>> > >> > > >> > dtcli
>> > > > > > > >>> > >> > > >> > > >> works
>> > > > > > > >>> > >> > > >> > > >> >> > > > properly
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > for
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of
>> > apps
>> > > > with
>> > > > > > > only
>> > > > > > > >>> > these
>> > > > > > > >>> > >> > > >> libraries
>> > > > > > > >>> > >> > > >> > > >> >> present in
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > isolation
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > without dependency on
>> local
>> > > m2.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there
>> are
>> > > > > unwanted
>> > > > > > > >>> jars
>> > > > > > > >>> > >> which
>> > > > > > > >>> > >> > are
>> > > > > > > >>> > >> > > >> used
>> > > > > > > >>> > >> > > >> > > for
>> > > > > > > >>> > >> > > >> > > >> >> > compile
>> > > > > > > >>> > >> > > >> > > >> >> > > > > time
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > dependency and not
>> runtime
>> > in
>> > > > > above
>> > > > > > > >>> list
>> > > > > > > >>> > >> which I
>> > > > > > > >>> > >> > > can
>> > > > > > > >>> > >> > > >> > drop
>> > > > > > > >>> > >> > > >> > > >> to
>> > > > > > > >>> > >> > > >> > > >> >> keep
>> > > > > > > >>> > >> > > >> > > >> >> > > > > package
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not
>> > sure
>> > > if
>> > > > > all
>> > > > > > > of
>> > > > > > > >>> the
>> > > > > > > >>> > >> > > >> dt-env.sh is
>> > > > > > > >>> > >> > > >> > > >> >> required.
>> > > > > > > >>> > >> > > >> > > >> >> > > > Only
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > required values I see
>> in it
>> > > are
>> > > > > > > >>> classpath.
>> > > > > > > >>> > >> and
>> > > > > > > >>> > >> > > >> dtcli is
>> > > > > > > >>> > >> > > >> > > >> >> capable
>> > > > > > > >>> > >> > > >> > > >> >> > of
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > building
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes,
>> > we'll
>> > > > have
>> > > > > > to
>> > > > > > > >>> see
>> > > > > > > >>> > how
>> > > > > > > >>> > >> > dtcli
>> > > > > > > >>> > >> > > >> can
>> > > > > > > >>> > >> > > >> > > find
>> > > > > > > >>> > >> > > >> > > >> >> the
>> > > > > > > >>> > >> > > >> > > >> >> > jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > path,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such
>> conf
>> > > > > files.
>> > > > > > If
>> > > > > > > >>> we
>> > > > > > > >>> > >> need a
>> > > > > > > >>> > >> > > >> change
>> > > > > > > >>> > >> > > >> > in
>> > > > > > > >>> > >> > > >> > > >> >> dtcli,
>> > > > > > > >>> > >> > > >> > > >> >> > > then
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > community's opinion is
>> > > required
>> > > > > for
>> > > > > > > >>> whether
>> > > > > > > >>> > >> > dtcli
>> > > > > > > >>> > >> > > >> > should
>> > > > > > > >>> > >> > > >> > > >> >> change
>> > > > > > > >>> > >> > > >> > > >> >> > in
>> > > > > > > >>> > >> > > >> > > >> >> > > > our
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > repo
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with
>> > > required
>> > > > > > > changes
>> > > > > > > >>> > >> exist in
>> > > > > > > >>> > >> > > >> bigtop
>> > > > > > > >>> > >> > > >> > > >> repo
>> > > > > > > >>> > >> > > >> > > >> >> > until
>> > > > > > > >>> > >> > > >> > > >> >> > > we
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > make
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic
>> enough.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > Can we have these file
>> paths
>> > > set
>> > > > > > using
>> > > > > > > >>> > >> environment
>> > > > > > > >>> > >> > > >> > > variables
>> > > > > > > >>> > >> > > >> > > >> >> with
>> > > > > > > >>> > >> > > >> > > >> >> > > some
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > Then we can set such
>> params
>> > in
>> > > > > > > dt-env.sh.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How
>> is
>> > > this
>> > > > > file
>> > > > > > > >>> > different
>> > > > > > > >>> > >> > from
>> > > > > > > >>> > >> > > >> > > >> >> dt-site.xml?
>> > > > > > > >>> > >> > > >> > > >> >> > Not
>> > > > > > > >>> > >> > > >> > > >> >> > > > > sure
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > if
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel
>> would
>> > > help.
>> > > > > > > Files &
>> > > > > > > >>> > Dirs
>> > > > > > > >>> > >> > from
>> > > > > > > >>> > >> > > >> > > /etc/skel
>> > > > > > > >>> > >> > > >> > > >> >> are
>> > > > > > > >>> > >> > > >> > > >> >> > > > copied
>> > > > > > > >>> > >> > > >> > > >> >> > > > > to
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > home of new user when
>> > useradd
>> > > > > > program
>> > > > > > > >>> is
>> > > > > > > >>> > >> called.
>> > > > > > > >>> > >> > > But
>> > > > > > > >>> > >> > > >> > for
>> > > > > > > >>> > >> > > >> > > >> >> existing
>> > > > > > > >>> > >> > > >> > > >> >> > > > users
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any
>> use.
>> > > > > Moreover,
>> > > > > > > >>> dtcli
>> > > > > > > >>> > >> > creates
>> > > > > > > >>> > >> > > >> > ~/.dt/
>> > > > > > > >>> > >> > > >> > > on
>> > > > > > > >>> > >> > > >> > > >> >> the
>> > > > > > > >>> > >> > > >> > > >> >> > fly
>> > > > > > > >>> > >> > > >> > > >> >> > > > if
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > not
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > encountered for the
>> first
>> > > time.
>> > > > > > Again
>> > > > > > > >>> > >> correct me
>> > > > > > > >>> > >> > > if
>> > > > > > > >>> > >> > > >> I'm
>> > > > > > > >>> > >> > > >> > > >> wrong.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right.
>> However
>> > > this
>> > > > > > would
>> > > > > > > >>> > >> probably be
>> > > > > > > >>> > >> > > >> > > necessary
>> > > > > > > >>> > >> > > >> > > >> >> step
>> > > > > > > >>> > >> > > >> > > >> >> > for
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also
>> get
>> > > > > > installed
>> > > > > > > >>> > during
>> > > > > > > >>> > >> OS
>> > > > > > > >>> > >> > > >> install
>> > > > > > > >>> > >> > > >> > > and
>> > > > > > > >>> > >> > > >> > > >> >> > before
>> > > > > > > >>> > >> > > >> > > >> >> > > > user
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc &
>> > > > bash_profile
>> > > > > > ->
>> > > > > > > >>> I'm
>> > > > > > > >>> > not
>> > > > > > > >>> > >> > sure
>> > > > > > > >>> > >> > > >> its
>> > > > > > > >>> > >> > > >> > the
>> > > > > > > >>> > >> > > >> > > >> best
>> > > > > > > >>> > >> > > >> > > >> >> > > idea.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > We'll
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in
>> > location
>> > > > > which
>> > > > > > is
>> > > > > > > >>> by
>> > > > > > > >>> > >> default
>> > > > > > > >>> > >> > > >> > present
>> > > > > > > >>> > >> > > >> > > in
>> > > > > > > >>> > >> > > >> > > >> >> path
>> > > > > > > >>> > >> > > >> > > >> >> > > i.e.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > For other env variables
>> > > specific
>> > > > to
>> > > > > > > apex,
>> > > > > > > >>> > I'll
>> > > > > > > >>> > >> > > prefer
>> > > > > > > >>> > >> > > >> to
>> > > > > > > >>> > >> > > >> > > use
>> > > > > > > >>> > >> > > >> > > >> >> > > dt-env.sh
>> > > > > > > >>> > >> > > >> > > >> >> > > > > and
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli
>> rather
>> > > than
>> > > > > > > changing
>> > > > > > > >>> > >> bashrc
>> > > > > > > >>> > >> > > >> etc...
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh
>> to
>> > > > > > > >>> /etc/profile.d/
>> > > > > > > >>> > >> shall
>> > > > > > > >>> > >> > > be
>> > > > > > > >>> > >> > > >> the
>> > > > > > > >>> > >> > > >> > > >> >> cleaner
>> > > > > > > >>> > >> > > >> > > >> >> > way
>> > > > > > > >>> > >> > > >> > > >> >> > > > to
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > Thanks,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at
>> > 1:07
>> > > > PM,
>> > > > > > > >>> Aniruddha
>> > > > > > > >>> > >> > > Thombare
>> > > > > > > >>> > >> > > >> <
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> aniruddha@datatorrent.com>
>> > > > > wrote:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > We need to consider
>> > > > following:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > System wide default
>> > config
>> > > > > files
>> > > > > > > can
>> > > > > > > >>> be
>> > > > > > > >>> > >> > located
>> > > > > > > >>> > >> > > at
>> > > > > > > >>> > >> > > >> > > >> following
>> > > > > > > >>> > >> > > >> > > >> >> > > > > locations:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh
>> (we
>> > > may
>> > > > > have
>> > > > > > > to
>> > > > > > > >>> > change
>> > > > > > > >>> > >> > > dtcli
>> > > > > > > >>> > >> > > >> > > >> behaviour
>> > > > > > > >>> > >> > > >> > > >> >> on
>> > > > > > > >>> > >> > > >> > > >> >> > > how
>> > > > > > > >>> > >> > > >> > > >> >> > > > it
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > finds
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml
>> (we
>> > > may
>> > > > > > have
>> > > > > > > to
>> > > > > > > >>> > >> change
>> > > > > > > >>> > >> > > dtcli
>> > > > > > > >>> > >> > > >> > > >> >> behaviour on
>> > > > > > > >>> > >> > > >> > > >> >> > > how
>> > > > > > > >>> > >> > > >> > > >> >> > > > > it
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > /etc/skel/.dt/dt-sited.xml
>> > > > and
>> > > > > > > other
>> > > > > > > >>> > files
>> > > > > > > >>> > >> > (for
>> > > > > > > >>> > >> > > >> new
>> > > > > > > >>> > >> > > >> > > users
>> > > > > > > >>> > >> > > >> > > >> >> that
>> > > > > > > >>> > >> > > >> > > >> >> > > will
>> > > > > > > >>> > >> > > >> > > >> >> > > > > be
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > created in system in
>> > > future.)
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > We may also have to
>> > modify
>> > > > > > bashrc /
>> > > > > > > >>> > >> > bashprofile
>> > > > > > > >>> > >> > > >> for
>> > > > > > > >>> > >> > > >> > > >> >> population
>> > > > > > > >>> > >> > > >> > > >> >> > > any
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > variables that are
>> > > required.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in
>> your
>> > > > > > comments /
>> > > > > > > >>> > >> > suggestion.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016
>> at
>> > > 1:16
>> > > > > AM,
>> > > > > > > >>> Thomas
>> > > > > > > >>> > >> Weise
>> > > > > > > >>> > >> > <
>> > > > > > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > -->
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016
>> at
>> > > 9:05
>> > > > > AM,
>> > > > > > > >>> Chinmay
>> > > > > > > >>> > >> > > >> Kolhatkar <
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> chinmay@datatorrent.com
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with
>> > > packaging
>> > > > > > > effort
>> > > > > > > >>> > >> (rpm+deb)
>> > > > > > > >>> > >> > > of
>> > > > > > > >>> > >> > > >> > apex,
>> > > > > > > >>> > >> > > >> > > >> here
>> > > > > > > >>> > >> > > >> > > >> >> > are
>> > > > > > > >>> > >> > > >> > > >> >> > > > some
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > about package
>> > structure
>> > > > > etc..
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it
>> on
>> > > > > bbigtop
>> > > > > > > >>> mailing
>> > > > > > > >>> > >> > list, I
>> > > > > > > >>> > >> > > >> have
>> > > > > > > >>> > >> > > >> > > >> some
>> > > > > > > >>> > >> > > >> > > >> >> > > question
>> > > > > > > >>> > >> > > >> > > >> >> > > > > for
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > apex
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory
>> > > > > structure
>> > > > > > of
>> > > > > > > >>> apex
>> > > > > > > >>> > >> > package
>> > > > > > > >>> > >> > > >> for
>> > > > > > > >>> > >> > > >> > > both
>> > > > > > > >>> > >> > > >> > > >> >> deb &
>> > > > > > > >>> > >> > > >> > > >> >> > > > rpm:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > /usr/lib/apex/bin/dtcli
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >>
>> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > /usr/lib/apex/lib/<other
>> > > > > > > >>> dependent
>> > > > > > > >>> > >> jars>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
>> > > > > > > >>> > /usr/lib/apex/bin/dtcli
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> /usr/share/doc/man/man1/dtcli.1.gz
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > /usr/share/doc/apex/license/LICENSE.txt
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >>
>> /usr/share/doc/apex/license/<package>-LICENSE.txt
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > /usr/share/doc/apex/CHANGELOG
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > /usr/share/doc/apex/NOTICE
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
>> > > > > 3.3.0-incubating.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent
>> > jars>
>> > > =
>> > > > > All
>> > > > > > > the
>> > > > > > > >>> 3rd
>> > > > > > > >>> > >> party
>> > > > > > > >>> > >> > > >> jars
>> > > > > > > >>> > >> > > >> > > which
>> > > > > > > >>> > >> > > >> > > >> >> are
>> > > > > > > >>> > >> > > >> > > >> >> > > > > required
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > for
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > apex
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually
>> the
>> > > > > > > dependencies
>> > > > > > > >>> are
>> > > > > > > >>> > >> > > packaged
>> > > > > > > >>> > >> > > >> as
>> > > > > > > >>> > >> > > >> > > >> part of
>> > > > > > > >>> > >> > > >> > > >> >> > > > rpm/deb
>> > > > > > > >>> > >> > > >> > > >> >> > > > > by
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > any
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > software in
>> bigtop.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what
>> > > those
>> > > > > jars
>> > > > > > > >>> are in
>> > > > > > > >>> > >> > Bigtop
>> > > > > > > >>> > >> > > >> > > context?
>> > > > > > > >>> > >> > > >> > > >> >> Same
>> > > > > > > >>> > >> > > >> > > >> >> > as
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > shipped
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > with
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > DT community
>> addition
>> > > under
>> > > > > > lib/
>> > > > > > > ?
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE>
>> =
>> > > > > Licenses
>> > > > > > of
>> > > > > > > >>> > >> > > corresponding
>> > > > > > > >>> > >> > > >> of
>> > > > > > > >>> > >> > > >> > > 3rd
>> > > > > > > >>> > >> > > >> > > >> >> party
>> > > > > > > >>> > >> > > >> > > >> >> > > jars
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > which
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > needs
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to included while
>> > > > > packaging.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions related
>> to
>> > > > this:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call
>> the
>> > > cli
>> > > > > of
>> > > > > > > >>> apex as
>> > > > > > > >>> > >> > "apex"
>> > > > > > > >>> > >> > > >> > instead
>> > > > > > > >>> > >> > > >> > > >> of
>> > > > > > > >>> > >> > > >> > > >> >> > > "dtcli"
>> > > > > > > >>> > >> > > >> > > >> >> > > > in
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > I think we should
>> keep
>> > > the
>> > > > > name
>> > > > > > > >>> until
>> > > > > > > >>> > we
>> > > > > > > >>> > >> are
>> > > > > > > >>> > >> > > >> able
>> > > > > > > >>> > >> > > >> > to
>> > > > > > > >>> > >> > > >> > > >> >> change
>> > > > > > > >>> > >> > > >> > > >> >> > it
>> > > > > > > >>> > >> > > >> > > >> >> > > in
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > Apex.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may
>> get
>> > > > > > confusing.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all
>> > > > softwares
>> > > > > > in
>> > > > > > > >>> bigtop
>> > > > > > > >>> > >> have
>> > > > > > > >>> > >> > > man
>> > > > > > > >>> > >> > > >> > page
>> > > > > > > >>> > >> > > >> > > >> for
>> > > > > > > >>> > >> > > >> > > >> >> > their
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we should
>> > have
>> > > it
>> > > > > too
>> > > > > > > for
>> > > > > > > >>> > >> dtcli.
>> > > > > > > >>> > >> > Is
>> > > > > > > >>> > >> > > >> there
>> > > > > > > >>> > >> > > >> > > any
>> > > > > > > >>> > >> > > >> > > >> >> > > > > documentation
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > which I
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man
>> > > page?
>> > > > or
>> > > > > > can
>> > > > > > > >>> I use
>> > > > > > > >>> > >> > output
>> > > > > > > >>> > >> > > >> of
>> > > > > > > >>> > >> > > >> > > >> "dtcli
>> > > > > > > >>> > >> > > >> > > >> >> > > --help"?
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to
>> call
>> > > > > version
>> > > > > > > of
>> > > > > > > >>> apex
>> > > > > > > >>> > >> in
>> > > > > > > >>> > >> > > >> Bigtop
>> > > > > > > >>> > >> > > >> > as
>> > > > > > > >>> > >> > > >> > > >> >> 3.3.0 OR
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be
>> -incubating
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for
>> apex
>> > > > > package
>> > > > > > of
>> > > > > > > >>> > bigtop
>> > > > > > > >>> > >> to
>> > > > > > > >>> > >> > > >> depend
>> > > > > > > >>> > >> > > >> > on
>> > > > > > > >>> > >> > > >> > > >> >> 2.7.1
>> > > > > > > >>> > >> > > >> > > >> >> > > > > version
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any
>> > > > problems
>> > > > > > > that
>> > > > > > > >>> we
>> > > > > > > >>> > see
>> > > > > > > >>> > >> > with
>> > > > > > > >>> > >> > > >> this
>> > > > > > > >>> > >> > > >> > > >> >> > dependency?
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Should work without
>> > > > changed.
>> > > > > I
>> > > > > > > >>> thought
>> > > > > > > >>> > we
>> > > > > > > >>> > >> > > >> certified
>> > > > > > > >>> > >> > > >> > > >> >> against
>> > > > > > > >>> > >> > > >> > > >> >> > > > 2.7.0?
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is
>> the
>> > > > apache
>> > > > > > > mirror
>> > > > > > > >>> > from
>> > > > > > > >>> > >> > which
>> > > > > > > >>> > >> > > >> > bigtop
>> > > > > > > >>> > >> > > >> > > >> will
>> > > > > > > >>> > >> > > >> > > >> >> > pick
>> > > > > > > >>> > >> > > >> > > >> >> > > > the
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > apex
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > source for
>> > compilation
>> > > > and
>> > > > > > > >>> packaging.
>> > > > > > > >>> > >> > Please
>> > > > > > > >>> > >> > > >> > > correct
>> > > > > > > >>> > >> > > >> > > >> if
>> > > > > > > >>> > >> > > >> > > >> >> > > > > incorrect:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > >
>> > > > > > > >>> > >> > >
>> > > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
>> > > > > apache.osuosl.org
>> > > > > > is
>> > > > > > > >>> the
>> > > > > > > >>> > >> mirror
>> > > > > > > >>> > >> > > >> used
>> > > > > > > >>> > >> > > >> > by
>> > > > > > > >>> > >> > > >> > > >> all
>> > > > > > > >>> > >> > > >> > > >> >> the
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > softwares
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > in
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share your
>> > > > opinion.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8,
>> 2016
>> > at
>> > > > > 11:19
>> > > > > > > AM,
>> > > > > > > >>> > >> Chinmay
>> > > > > > > >>> > >> > > >> > Kolhatkar
>> > > > > > > >>> > >> > > >> > > <
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > chinmay@datatorrent.com>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting
>> to
>> > > work
>> > > > on
>> > > > > > the
>> > > > > > > >>> > >> packaging.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already
>> > started
>> > > > > > > >>> discussion on
>> > > > > > > >>> > >> > bigtop
>> > > > > > > >>> > >> > > >> dev
>> > > > > > > >>> > >> > > >> > > >> mailing
>> > > > > > > >>> > >> > > >> > > >> >> > list
>> > > > > > > >>> > >> > > >> > > >> >> > > > for
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration.
>> Also
>> > > > > created a
>> > > > > > > >>> Jira
>> > > > > > > >>> > for
>> > > > > > > >>> > >> the
>> > > > > > > >>> > >> > > >> same.
>> > > > > > > >>> > >> > > >> > > For
>> > > > > > > >>> > >> > > >> > > >> >> this
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > communities
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here
>> is
>> > > the
>> > > > > > bigtop
>> > > > > > > >>> Jira:
>> > > > > > > >>> > >> > > >> > BIGTOP-2313.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8,
>> 2016
>> > > at
>> > > > > 1:13
>> > > > > > > AM,
>> > > > > > > >>> > >> Thomas
>> > > > > > > >>> > >> > > >> Weise <
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see
>> > anything
>> > > > > under
>> > > > > > > >>> > >> > prerequisites
>> > > > > > > >>> > >> > > >> that
>> > > > > > > >>> > >> > > >> > > >> would
>> > > > > > > >>> > >> > > >> > > >> >> be a
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > problem.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > We
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF
>> > licencing
>> > > > > > > >>> compatibility
>> > > > > > > >>> > >> as
>> > > > > > > >>> > >> > > part
>> > > > > > > >>> > >> > > >> of
>> > > > > > > >>> > >> > > >> > > >> >> becoming
>> > > > > > > >>> > >> > > >> > > >> >> > an
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > incubator
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on
>> > the
>> > > > > > > packaging
>> > > > > > > >>> > during
>> > > > > > > >>> > >> > the
>> > > > > > > >>> > >> > > >> next
>> > > > > > > >>> > >> > > >> > > >> weeks.
>> > > > > > > >>> > >> > > >> > > >> >> > Since
>> > > > > > > >>> > >> > > >> > > >> >> > > > the
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > work
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > will
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop,
>> > > > related
>> > > > > > > >>> > discussions
>> > > > > > > >>> > >> and
>> > > > > > > >>> > >> > > >> JIRAs
>> > > > > > > >>> > >> > > >> > > >> should
>> > > > > > > >>> > >> > > >> > > >> >> > also
>> > > > > > > >>> > >> > > >> > > >> >> > > be
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > there.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good
>> to
>> > > have
>> > > > > the
>> > > > > > > >>> > packaging
>> > > > > > > >>> > >> in
>> > > > > > > >>> > >> > > >> place
>> > > > > > > >>> > >> > > >> > by
>> > > > > > > >>> > >> > > >> > > >> end
>> > > > > > > >>> > >> > > >> > > >> >> Feb.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4,
>> > 2016
>> > > at
>> > > > > > 10:14
>> > > > > > > >>> AM,
>> > > > > > > >>> > >> > Chinmay
>> > > > > > > >>> > >> > > >> > > >> Kolhatkar <
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > > > chinmay@datatorrent.com
>> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're
>> planning a
>> > > > work
>> > > > > on
>> > > > > > > >>> adding
>> > > > > > > >>> > >> > Apache
>> > > > > > > >>> > >> > > >> Apex
>> > > > > > > >>> > >> > > >> > > as a
>> > > > > > > >>> > >> > > >> > > >> >> > > component
>> > > > > > > >>> > >> > > >> > > >> >> > > > > to
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Apache
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the
>> > > > > packaging
>> > > > > > > >>> system
>> > > > > > > >>> > for
>> > > > > > > >>> > >> > the
>> > > > > > > >>> > >> > > >> > Apache
>> > > > > > > >>> > >> > > >> > > >> big
>> > > > > > > >>> > >> > > >> > > >> >> data
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop
>> distros
>> > use
>> > > > it,
>> > > > > > > most
>> > > > > > > >>> > >> recently
>> > > > > > > >>> > >> > > EMR.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the
>> > > tracking
>> > > > > > Jira
>> > > > > > > >>> task
>> > > > > > > >>> > in
>> > > > > > > >>> > >> > > >> APEXCORE
>> > > > > > > >>> > >> > > >> > for
>> > > > > > > >>> > >> > > >> > > >> the
>> > > > > > > >>> > >> > > >> > > >> >> > same:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > https://issues.apache.org/jira/browse/APEXCORE-331
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed
>> plan of
>> > > > > > execution
>> > > > > > > >>> is as
>> > > > > > > >>> > >> > > follows:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1)
>> Handle
>> > > > > > > >>> prerequisites*
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop
>> > has
>> > > > some
>> > > > > > > hard
>> > > > > > > >>> and
>> > > > > > > >>> > >> soft
>> > > > > > > >>> > >> > > >> > > expectation
>> > > > > > > >>> > >> > > >> > > >> >> for
>> > > > > > > >>> > >> > > >> > > >> >> > new
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > components
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated
>> into
>> > > > > Bigtop.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the
>> list
>> > > of
>> > > > > it:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > >
>> > > > > > > >>> > >> > > >> > > >> >> > >
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >>
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> >
>> > > > > > > >>> > >> > > >>
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> >
>> > > > > > > >>> > >>
>> > > > > > > >>> >
>> > > > > > > >>>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them
>> > seems
>> > > > to
>> > > > > be
>> > > > > > > >>> > standard
>> > > > > > > >>> > >> ASF
>> > > > > > > >>> > >> > > >> based
>> > > > > > > >>> > >> > > >> > > >> >> > > requirements,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > but
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > few
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > need
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked
>> for:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software
>> > > projects
>> > > > > are
>> > > > > > > >>> > expected
>> > > > > > > >>> > >> to
>> > > > > > > >>> > >> > be
>> > > > > > > >>> > >> > > >> > > Licensed
>> > > > > > > >>> > >> > > >> > > >> >> under
>> > > > > > > >>> > >> > > >> > > >> >> > > > > Apache
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > License,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0
>> (and
>> > > > their
>> > > > > > > >>> > >> dependencies
>> > > > > > > >>> > >> > are
>> > > > > > > >>> > >> > > >> > > expected
>> > > > > > > >>> > >> > > >> > > >> >> to be
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > compatible
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > with
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is
>> > > under
>> > > > > ASL
>> > > > > > > 2.0
>> > > > > > > >>> but
>> > > > > > > >>> > >> need
>> > > > > > > >>> > >> > to
>> > > > > > > >>> > >> > > >> > check
>> > > > > > > >>> > >> > > >> > > if
>> > > > > > > >>> > >> > > >> > > >> >> > > > > dependencies
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible
>> with
>> > > ASL
>> > > > > 2.0.
>> > > > > > > >>> This I
>> > > > > > > >>> > >> guess
>> > > > > > > >>> > >> > > >> would
>> > > > > > > >>> > >> > > >> > > be a
>> > > > > > > >>> > >> > > >> > > >> >> > > > > verification
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > check.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software
>> > > projects
>> > > > > are
>> > > > > > > >>> > expected
>> > > > > > > >>> > >> to
>> > > > > > > >>> > >> > be
>> > > > > > > >>> > >> > > >> > > >> compatible
>> > > > > > > >>> > >> > > >> > > >> >> > with
>> > > > > > > >>> > >> > > >> > > >> >> > > > all
>> > > > > > > >>> > >> > > >> > > >> >> > > > > of
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > the
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported
>> > > platforms
>> > > > > that
>> > > > > > > >>> Bigtop
>> > > > > > > >>> > >> > > >> distribution
>> > > > > > > >>> > >> > > >> > > is
>> > > > > > > >>> > >> > > >> > > >> >> > > targeting
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This
>> needs
>> > > > > > verifying
>> > > > > > > >>> > whether
>> > > > > > > >>> > >> > our
>> > > > > > > >>> > >> > > >> > > software
>> > > > > > > >>> > >> > > >> > > >> >> runs
>> > > > > > > >>> > >> > > >> > > >> >> > > fine
>> > > > > > > >>> > >> > > >> > > >> >> > > > in
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7
>> > fedora-20
>> > > > > > > >>> ubuntu-14.04
>> > > > > > > >>> > >> > > debian-8
>> > > > > > > >>> > >> > > >> > > >> >> > opensuse-13.2.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke
>> > > tests
>> > > > > that
>> > > > > > > >>> should
>> > > > > > > >>> > be
>> > > > > > > >>> > >> > > added
>> > > > > > > >>> > >> > > >> for
>> > > > > > > >>> > >> > > >> > > >> >> > deployment.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4.
>> Identifying
>> > the
>> > > > > test
>> > > > > > > >>> > artifacts
>> > > > > > > >>> > >> > which
>> > > > > > > >>> > >> > > >> goes
>> > > > > > > >>> > >> > > >> > > >> beyond
>> > > > > > > >>> > >> > > >> > > >> >> > > smoke
>> > > > > > > >>> > >> > > >> > > >> >> > > > > test
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These
>> are
>> > > > > > basically
>> > > > > > > >>> the
>> > > > > > > >>> > >> > > integration
>> > > > > > > >>> > >> > > >> > > tests
>> > > > > > > >>> > >> > > >> > > >> for
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > verification
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > after
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment.
>> This
>> > > is
>> > > > a
>> > > > > > soft
>> > > > > > > >>> > >> > requirement,
>> > > > > > > >>> > >> > > >> but
>> > > > > > > >>> > >> > > >> > > aim
>> > > > > > > >>> > >> > > >> > > >> is
>> > > > > > > >>> > >> > > >> > > >> >> to
>> > > > > > > >>> > >> > > >> > > >> >> > > > > achieve
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > this
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > as
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least
>> have
>> > > > > > > >>> explanation why
>> > > > > > > >>> > >> not
>> > > > > > > >>> > >> > to
>> > > > > > > >>> > >> > > >> > > include.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are
>> any
>> > > > from
>> > > > > > the
>> > > > > > > >>> link
>> > > > > > > >>> > >> which
>> > > > > > > >>> > >> > > >> > > explicitly
>> > > > > > > >>> > >> > > >> > > >> >> needs
>> > > > > > > >>> > >> > > >> > > >> >> > to
>> > > > > > > >>> > >> > > >> > > >> >> > > > be
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > checked
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4,
>> > > please
>> > > > > let
>> > > > > > > us
>> > > > > > > >>> > know.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2)
>> Adding
>> > > Apex
>> > > > > as
>> > > > > > > >>> > component
>> > > > > > > >>> > >> to
>> > > > > > > >>> > >> > > >> Bigtop*
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of
>> the
>> > > mail
>> > > > > > > >>> archive of
>> > > > > > > >>> > >> > Bigtop,
>> > > > > > > >>> > >> > > >> it
>> > > > > > > >>> > >> > > >> > was
>> > > > > > > >>> > >> > > >> > > >> >> learnt
>> > > > > > > >>> > >> > > >> > > >> >> > > that
>> > > > > > > >>> > >> > > >> > > >> >> > > > > the
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community
>> want
>> > to
>> > > > see
>> > > > > > the
>> > > > > > > >>> > >> addition of
>> > > > > > > >>> > >> > > new
>> > > > > > > >>> > >> > > >> > > >> >> components
>> > > > > > > >>> > >> > > >> > > >> >> > in
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > phases.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Here
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > are
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This
>> needs
>> > > > > > creating
>> > > > > > > of
>> > > > > > > >>> > >> package
>> > > > > > > >>> > >> > > i.e.
>> > > > > > > >>> > >> > > >> > rpm
>> > > > > > > >>> > >> > > >> > > &
>> > > > > > > >>> > >> > > >> > > >> deb
>> > > > > > > >>> > >> > > >> > > >> >> > > files.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
>> > > > > > > >>> documentations/READMEs,
>> > > > > > > >>> > >> > LICENSE,
>> > > > > > > >>> > >> > > >> > > >> DISCLAMER,
>> > > > > > > >>> > >> > > >> > > >> >> > NOTES
>> > > > > > > >>> > >> > > >> > > >> >> > > > etc
>> > > > > > > >>> > >> > > >> > > >> >> > > > > if
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > any
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
>> > > > > documentation
>> > > > > > > that
>> > > > > > > >>> > need
>> > > > > > > >>> > >> to
>> > > > > > > >>> > >> > be
>> > > > > > > >>> > >> > > >> > added
>> > > > > > > >>> > >> > > >> > > to
>> > > > > > > >>> > >> > > >> > > >> >> > > > > distribution
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > our
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
>> > license
>> > > > > > > >>> information of
>> > > > > > > >>> > >> > > >> > dependencies
>> > > > > > > >>> > >> > > >> > > >> >> required
>> > > > > > > >>> > >> > > >> > > >> >> > > to
>> > > > > > > >>> > >> > > >> > > >> >> > > > be
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > added
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke
>> tests
>> > (at
>> > > > > very
>> > > > > > > >>> least)
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding
>> > smoke
>> > > > > test
>> > > > > > > for
>> > > > > > > >>> > >> > packaging.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet
>> > recipes
>> > > > for
>> > > > > > > >>> automatic
>> > > > > > > >>> > >> > > >> deployment
>> > > > > > > >>> > >> > > >> > and
>> > > > > > > >>> > >> > > >> > > >> >> > > > configuration
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add
>> puppet
>> > > > > recipes
>> > > > > > > for
>> > > > > > > >>> > >> > automatic
>> > > > > > > >>> > >> > > >> > > >> deployment
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4.
>> Integration
>> > > tests
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For
>> > > > verification
>> > > > > > of
>> > > > > > > >>> > >> > deployments.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license
>> > > > clearance:
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run
>> 'gradle
>> > > rat'
>> > > > > to
>> > > > > > > make
>> > > > > > > >>> > sure
>> > > > > > > >>> > >> all
>> > > > > > > >>> > >> > > new
>> > > > > > > >>> > >> > > >> > > stuff
>> > > > > > > >>> > >> > > >> > > >> is
>> > > > > > > >>> > >> > > >> > > >> >> > > > compliant
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > with
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license
>> > > > requirements.
>> > > > > If
>> > > > > > > >>> you add
>> > > > > > > >>> > >> code
>> > > > > > > >>> > >> > > >> > licenses
>> > > > > > > >>> > >> > > >> > > >> >> under
>> > > > > > > >>> > >> > > >> > > >> >> > > > > different
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would
>> need
>> > > to
>> > > > be
>> > > > > > > >>> listed in
>> > > > > > > >>> > >> the
>> > > > > > > >>> > >> > > >> NOTICE.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share
>> > your
>> > > > > > thoughts
>> > > > > > > >>> on
>> > > > > > > >>> > the
>> > > > > > > >>> > >> > > >> approach.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
>> > > > > > corresponding
>> > > > > > > >>> > >> > communication
>> > > > > > > >>> > >> > > >> on
>> > > > > > > >>> > >> > > >> > > >> bigtop
>> > > > > > > >>> > >> > > >> > > >> >> > > mailing
>> > > > > > > >>> > >> > > >> > > >> >> > > > > list
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > as
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some
>> > > > specific
>> > > > > > > >>> > >> > > >> questions/suggestions
>> > > > > > > >>> > >> > > >> > > >> >> related to
>> > > > > > > >>> > >> > > >> > > >> >> > > > what
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > should
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > be
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of
>> the
>> > > > package
>> > > > > > and
>> > > > > > > >>> what
>> > > > > > > >>> > >> > should
>> > > > > > > >>> > >> > > be
>> > > > > > > >>> > >> > > >> > the
>> > > > > > > >>> > >> > > >> > > >> smoke
>> > > > > > > >>> > >> > > >> > > >> >> > > tests,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > but
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > in
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > the
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of
>> not
>> > > > having
>> > > > > > too
>> > > > > > > >>> much
>> > > > > > > >>> > >> > content
>> > > > > > > >>> > >> > > >> > here,
>> > > > > > > >>> > >> > > >> > > >> we'll
>> > > > > > > >>> > >> > > >> > > >> >> > put
>> > > > > > > >>> > >> > > >> > > >> >> > > > the
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > questions
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate
>> mail in
>> > > > this
>> > > > > > > >>> > mailthread.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > > >
>> > > > > > > >>> > >> > > >> > > >> >> > > >
>> > > > > > > >>> > >> > > >> > > >> >> > >
>> > > > > > > >>> > >> > > >> > > >> >> >
>> > > > > > > >>> > >> > > >> > > >> >>
>> > > > > > > >>> > >> > > >> > > >> >
>> > > > > > > >>> > >> > > >> > > >> >
>> > > > > > > >>> > >> > > >> > > >>
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > > >
>> > > > > > > >>> > >> > > >> > >
>> > > > > > > >>> > >> > > >> >
>> > > > > > > >>> > >> > > >>
>> > > > > > > >>> > >> > > >
>> > > > > > > >>> > >> > > >
>> > > > > > > >>> > >> > >
>> > > > > > > >>> > >> >
>> > > > > > > >>> > >>
>> > > > > > > >>> > >
>> > > > > > > >>> > >
>> > > > > > > >>> >
>> > > > > > > >>>
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Bigtop has consumed first patch for building apex packages into it's master
branch.
Commit mail:
http://mail-archives.apache.org/mod_mbox/bigtop-commits/201602.mbox/%3C06c0ca62ac1b483d8ded090f0f2c8d73%40git.apache.org%3E

What this means is bigtop master will be able to generate apex packages
(3.3.0-incubating), both deb & rpm, when it build.

For the second phase, we need to add smoke tests for our package in bigtop
repo.
This is an important steps for bigtop to host the generate package in
bigtop repository during its 1.2.0 release.

Here are my initial thoughts about smoke test:
TEST::
1. Create mvn app with archetype
2. Build app
3. Run app with apex/dtcli

VERIFICATION:
1. stdout file gets has random numbers per line.

I see app created by archetype is randomNumberGenerator -> Console.
I think this should be a good enough smoke test for apex package.

Of course I need to check if we can do above with bigtop framework. But I
have a hunch that we can, because there are standalone frameworks like us
which has similar types of tests.

Please share your opinion and thoughts on this.

Thanks,
Chinmay.



On Fri, Feb 19, 2016 at 11:38 AM, Thomas Weise <th...@datatorrent.com>
wrote:

> We should not package any Hadoop dependencies, they are provided by the
> Bigtop stack.
>
>
> On Thu, Feb 18, 2016 at 8:56 PM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > I just did a quick install of bigtop 1.1.0 on ubuntu and observed that
> >  jersey-client-1.9.jar is already packaged in hadoop
> > /usr/lib/hadoop/client/jersey-client.jar, so I guess it will be ok to
> > include those in apex lib too. or If we are including hadoop jars
> > in classpath while launching the apex application then it may not be
> > needed.
> >
> > - Tushar.
> >
> >
> >
> > On Fri, Feb 19, 2016 at 8:44 AM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > No worries David. I've updated the manpage.
> > >
> > > Just an update: apex bigtop packaging for rpm+deb is done. Here is the
> > > branch on which I'm working if someone wants to look at:
> > > https://github.com/chinmaykolhatkar/bigtop/blob/BIGTOP-2314
> > >
> > > Only the 3rd questions below is the question that is left to be
> answered.
> > > Once this is resolved, I'll create the pull request in bigtop.
> > >
> > > Following questions are related to licenses and inclusion of certain
> > files
> > > in apex rpm+deb:
> > > 1. I'll be included LICENSE, DISCLAIMER, NOTICE from top folder of core
> > > repo in our rpm and deb package. Those will be present in standard
> > location
> > > i.e. /usr/lib/apex/
> > > Please let me know otherwise.
> > >
> > > 2. Most of our runtime dependencies has ASF license, but certain
> > libraries
> > > has MIT and BSD licenses, hence I've included those licenses too in
> rpm &
> > > deb package in same location as above.
> > > Please let me know otherwise.
> > >
> > > BSD - jline-2.11.jar, kryo-2.24.0.jar, minlog-1.2.jar
> > > MIT - mbassador-1.1.9.jar
> > >
> > > 3. There are 2 jars (jersey-apache-client4-1.9.jar,
> > jersey-client-1.9.jar)
> > > which has duel license of CDDL + GPLv2 with classpath exception:
> > > https://jersey.java.net/license.html
> > > I'm not an expert here. Hence can someone let me know whether these
> > > libraries is OK to be packaged in rpm+deb of apex?
> > >
> > > If yes, should we also include that license in our package?
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > >
> > > On Thu, Feb 18, 2016 at 11:35 PM, David Yan <da...@datatorrent.com>
> > wrote:
> > >
> > > > I should've mentioned that I had a pull request that was merged
> > yesterday
> > > > that corrected the statement:
> > > >
> > > > https://github.com/DataTorrent/docs/pull/45
> > > >
> > > > On Thu, Feb 18, 2016 at 12:07 AM, Tushar Gosavi <
> > tushar@datatorrent.com>
> > > > wrote:
> > > >
> > > > > In following statement, can you replace dtGateway with
> > > ApplicationMaster
> > > > > and Yarn. or may be remove this statement entirely.
> > > > > apex is a wrapper around the REST API provided by dtGatway,
> > > > >
> > > > > - Tushar.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <
> > > chinmay@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I've attached man page for apex in Jira. Here is the link to it:
> > > > > > https://issues.apache.org/jira/secure/attachment/12788403/apex.1
> > > > > >
> > > > > > Please view it by: man -l apex.1
> > > > > >
> > > > > > Thanks,
> > > > > > Chinmay.
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <
> > > > tushar@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Chinmay,
> > > > > > >
> > > > > > > I am not able to find the attachment, may be attachments are
> not
> > > > > allowed
> > > > > > in
> > > > > > > the mailing list. Can you attach the man page to the Jira?
> > > > > > >
> > > > > > > - Tushar.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <
> > > > chinmay@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > I've attached man page file for the purpose of bigtop
> > > integration.
> > > > > > > > Please have a look and let me know if there any any feedback
> > > about
> > > > > it.
> > > > > > > >
> > > > > > > > You can see the man page content of this file with:
> > > > > > > > man -l apex.1
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chinmay.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> > > > > > chinmay@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks. Will follow bigtop recommendation. i.e. remove
> > > > > "-incubating".
> > > > > > > >>
> > > > > > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
> > > > > thomas@datatorrent.com
> > > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Please follow the recommendation and continue discussion on
> > the
> > > > > > Bigtop
> > > > > > > >>> list.
> > > > > > > >>>
> > > > > > > >>> --
> > > > > > > >>> sent from mobile
> > > > > > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <
> > > chinmay@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >>>
> > > > > > > >>> > I got a recommendation from bigtop community to drop
> > > > > "-incubating"
> > > > > > > from
> > > > > > > >>> > version.
> > > > > > > >>> >
> > > > > > > >>> > Here is the mail thread:
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> > > > > > > >>> >
> > > > > > > >>> > Please let me know if removing "-incubating" is fine.
> > > > > > > >>> >
> > > > > > > >>> > Thanks,
> > > > > > > >>> > Chinmay.
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> > > > > > > chinmay@apache.org
> > > > > > > >>> >
> > > > > > > >>> > wrote:
> > > > > > > >>> >
> > > > > > > >>> > > I've already started communication there.
> > > > > > > >>> > >
> > > > > > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> > > > > > > >>> thomas@datatorrent.com>
> > > > > > > >>> > > wrote:
> > > > > > > >>> > >
> > > > > > > >>> > >> This is a question you should ask on the Bigtop
> mailing
> > > > list.
> > > > > > > >>> > >>
> > > > > > > >>> > >> --
> > > > > > > >>> > >> sent from mobile
> > > > > > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <
> > > > prad@apache.org>
> > > > > > > >>> wrote:
> > > > > > > >>> > >>
> > > > > > > >>> > >> > Yes. In case of RPM, the only restriction on version
> > is
> > > > that
> > > > > > it
> > > > > > > >>> can
> > > > > > > >>> > not
> > > > > > > >>> > >> > contain '-'.
> > > > > > > >>> > >> > However, as a workaround, one can use 'incubating'
> as
> > a
> > > > part
> > > > > > of
> > > > > > > >>> > >> %{release}.
> > > > > > > >>> > >> >
> > > > > > > >>> > >> > Version: 3.3.0
> > > > > > > >>> > >> > Release: incubating
> > > > > > > >>> > >> >
> > > > > > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> > > > > > > >>> > >> >
> > > > > > > >>> > >> > It is the same practice followed for pre-release
> > > packages
> > > > > e.g.
> > > > > > > >>> with
> > > > > > > >>> > rcX,
> > > > > > > >>> > >> > alphaX, betaX being used in %{release}.
> > > > > > > >>> > >> >
> > > > > > > >>> > >> > Regards,
> > > > > > > >>> > >> > Pradeep A. Dalvi
> > > > > > > >>> > >> >
> > > > > > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> > > > > > > >>> > chinmay@apache.org>
> > > > > > > >>> > >> > wrote:
> > > > > > > >>> > >> >
> > > > > > > >>> > >> > > Hi Everyone,
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> > > In one of the previous mails in this thread, we
> > > decided
> > > > > that
> > > > > > > >>> version
> > > > > > > >>> > >> of
> > > > > > > >>> > >> > the
> > > > > > > >>> > >> > > package will be "3.3.0-incubating".
> > > > > > > >>> > >> > > The problem that I'm facing is in RPM base version
> > "-"
> > > > is
> > > > > > not
> > > > > > > >>> > allowed.
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> > > Can we have package version to "3.3.0"?
> > > > > > > >>> > >> > > i.e. the packages created will be as follows:
> > > > > > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > > > > > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> > > There is another integration for Zepplin in Bigtop
> > > which
> > > > > is
> > > > > > > >>> > incubating
> > > > > > > >>> > >> > and
> > > > > > > >>> > >> > > is having base version as "0.5.6".
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> > > Please let me know your thoughts.
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> > > Thanks,
> > > > > > > >>> > >> > > Chinmay.
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay
> Kolhatkar
> > <
> > > > > > > >>> > >> chinmay@apache.org>
> > > > > > > >>> > >> > > wrote:
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> > > > Thanks everyone for the feedback.
> > > > > > > >>> > >> > > >
> > > > > > > >>> > >> > > > Also as Ram and David mentioned, I'll start
> > creating
> > > > the
> > > > > > man
> > > > > > > >>> page
> > > > > > > >>> > as
> > > > > > > >>> > >> > > well.
> > > > > > > >>> > >> > > >
> > > > > > > >>> > >> > > > Thanks,
> > > > > > > >>> > >> > > > Chinmay.
> > > > > > > >>> > >> > > >
> > > > > > > >>> > >> > > >
> > > > > > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> > > > > > > >>> > >> thomas@datatorrent.com>
> > > > > > > >>> > >> > > > wrote:
> > > > > > > >>> > >> > > >
> > > > > > > >>> > >> > > >> I think we already agreed that we will retain
> the
> > > > > current
> > > > > > > >>> script
> > > > > > > >>> > >> name
> > > > > > > >>> > >> > in
> > > > > > > >>> > >> > > >> this bigtop iteration. There is a ticket for
> > > > renaming,
> > > > > > add
> > > > > > > >>> you
> > > > > > > >>> > >> > > suggestions
> > > > > > > >>> > >> > > >> there.
> > > > > > > >>> > >> > > >>
> > > > > > > >>> > >> > > >> --
> > > > > > > >>> > >> > > >> sent from mobile
> > > > > > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > > > > > > >>> > sandesh@datatorrent.com>
> > > > > > > >>> > >> > > wrote:
> > > > > > > >>> > >> > > >>
> > > > > > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start
> the
> > > > > > transition
> > > > > > > >>> > >> process by
> > > > > > > >>> > >> > > >> making
> > > > > > > >>> > >> > > >> > dtcli as the symlink to apex cli.
> > > > > > > >>> > >> > > >> >
> > > > > > > >>> > >> > > >> > Renaming is a breaking change should be done
> > > during
> > > > > > major
> > > > > > > >>> > version
> > > > > > > >>> > >> > > number
> > > > > > > >>> > >> > > >> > change.
> > > > > > > >>> > >> > > >> >
> > > > > > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> > > > > > > >>> > david@datatorrent.com
> > > > > > > >>> > >> >
> > > > > > > >>> > >> > > >> wrote:
> > > > > > > >>> > >> > > >> >
> > > > > > > >>> > >> > > >> > > Apologies for hitting the "Send" button too
> > > > early.
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > What is the scope of the name change?  If
> we
> > > > simply
> > > > > > > have
> > > > > > > >>> a
> > > > > > > >>> > >> symlink
> > > > > > > >>> > >> > > >> from
> > > > > > > >>> > >> > > >> > > apex to dtcli, then it should have no
> > backward
> > > > > > > >>> compatibility
> > > > > > > >>> > >> > problem
> > > > > > > >>> > >> > > >> at
> > > > > > > >>> > >> > > >> > > all.
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > But the proper name change would involve:
> > > > > > > >>> > >> > > >> > > - Change the name of DTCli class to ApexCli
> > > > > > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> > > > > > > >>> > >> > > >> > > - Change the initial greeting when running
> > the
> > > > CLI
> > > > > > from
> > > > > > > >>> > >> something
> > > > > > > >>> > >> > > like
> > > > > > > >>> > >> > > >> > "DT
> > > > > > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI
> ...".
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > The above changes may break existing
> scripts
> > > that
> > > > > > rely
> > > > > > > >>> on the
> > > > > > > >>> > >> the
> > > > > > > >>> > >> > > >> string
> > > > > > > >>> > >> > > >> > > "DT".
> > > > > > > >>> > >> > > >> > > But if folks are okay with this potential
> > > > problem,
> > > > > I
> > > > > > am
> > > > > > > >>> okay
> > > > > > > >>> > >> with
> > > > > > > >>> > >> > > the
> > > > > > > >>> > >> > > >> > name
> > > > > > > >>> > >> > > >> > > change.
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > For the man page, we can start with the
> > content
> > > > of
> > > > > > this
> > > > > > > >>> page:
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > Plus the command line options when doing
> > dtcli
> > > > -h:
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > usage: DTCli
> > > > > > > >>> > >> > > >> > >  -e <arg>    Commands are read from the
> > > argument
> > > > > > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at
> all
> > > time
> > > > > > > >>> > >> > > >> > >  -h          Print this help
> > > > > > > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos
> > > principal
> > > > > > > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos
> > keytab
> > > > > > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> > > > > > > >>> > >> > > >> > >  -r          JSON Raw mode
> > > > > > > >>> > >> > > >> > >  -v          Verbose mode level 1
> > > > > > > >>> > >> > > >> > >  -vv         Verbose mode level 2
> > > > > > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> > > > > > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > David
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan
> <
> > > > > > > >>> > >> david@datatorrent.com
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> > > >> > wrote:
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > > What is the scope of the name change?  If
> > we
> > > > > simply
> > > > > > > >>> have a
> > > > > > > >>> > >> > symlink
> > > > > > > >>> > >> > > >> from
> > > > > > > >>> > >> > > >> > > > apex to dtcli, then it should have no
> > > backward
> > > > > > > >>> > compatibility
> > > > > > > >>> > >> > > >> problem at
> > > > > > > >>> > >> > > >> > > all.
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > > > But the proper name change would involve:
> > > > > > > >>> > >> > > >> > > > - Change the name of DTCli class to
> ApexCli
> > > > > > > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> > > > > > > >>> > >> > > >> > > > - Change the initial greeting when
> running
> > > the
> > > > > CLI
> > > > > > > from
> > > > > > > >>> > >> > something
> > > > > > > >>> > >> > > >> like
> > > > > > > >>> > >> > > >> > > "DT
> > > > > > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI
> > ...".
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > > > The above changes may break existing
> > scripts
> > > > that
> > > > > > > rely
> > > > > > > >>> on
> > > > > > > >>> > the
> > > > > > > >>> > >> > the
> > > > > > > >>> > >> > > >> > string
> > > > > > > >>> > >> > > >> > > > "DT".
> > > > > > > >>> > >> > > >> > > > But if folks are okay with this potential
> > > > > problem,
> > > > > > I
> > > > > > > am
> > > > > > > >>> > okay
> > > > > > > >>> > >> > with
> > > > > > > >>> > >> > > >> the
> > > > > > > >>> > >> > > >> > > name
> > > > > > > >>> > >> > > >> > > > change.
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > > > For the man page, we can start with the
> > > content
> > > > > of
> > > > > > > this
> > > > > > > >>> > page:
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay
> > > > > Kolhatkar
> > > > > > <
> > > > > > > >>> > >> > > >> chinmay@apache.org
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> > > > wrote:
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > > >> Here is a suggestion I have related to
> > name
> > > > > > change:
> > > > > > > >>> > >> > > >> > > >>
> > > > > > > >>> > >> > > >> > > >> 1. While packaging we change the name of
> > > > "dtcli"
> > > > > > > >>> script to
> > > > > > > >>> > >> > > "apex".
> > > > > > > >>> > >> > > >> > This
> > > > > > > >>> > >> > > >> > > is
> > > > > > > >>> > >> > > >> > > >> done so that in future when we change
> the
> > > name
> > > > > to
> > > > > > > >>> "apex",
> > > > > > > >>> > >> users
> > > > > > > >>> > >> > > of
> > > > > > > >>> > >> > > >> > > bigtop
> > > > > > > >>> > >> > > >> > > >> apex don't have to transition much.
> > > > > > > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli
> > which
> > > > > points
> > > > > > > to
> > > > > > > >>> apex
> > > > > > > >>> > >> > > script.
> > > > > > > >>> > >> > > >> > This
> > > > > > > >>> > >> > > >> > > is
> > > > > > > >>> > >> > > >> > > >> for backward compatibility.
> > > > > > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for
> first
> > > > > > > integration
> > > > > > > >>> in
> > > > > > > >>> > >> > bigtop.
> > > > > > > >>> > >> > > >> This
> > > > > > > >>> > >> > > >> > is
> > > > > > > >>> > >> > > >> > > >> in
> > > > > > > >>> > >> > > >> > > >> the interest of not changing the content
> > of
> > > > > dtcli
> > > > > > or
> > > > > > > >>> apex
> > > > > > > >>> > >> > script.
> > > > > > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all
> > these
> > > > > naming
> > > > > > > >>> related
> > > > > > > >>> > >> > > changes
> > > > > > > >>> > >> > > >> and
> > > > > > > >>> > >> > > >> > > >> update bigtop repository later to remove
> > > > > > references
> > > > > > > to
> > > > > > > >>> > dtcli
> > > > > > > >>> > >> > all
> > > > > > > >>> > >> > > >> > > together.
> > > > > > > >>> > >> > > >> > > >>
> > > > > > > >>> > >> > > >> > > >> Please share your thoughts on above
> > > approach.
> > > > > > > >>> > >> > > >> > > >>
> > > > > > > >>> > >> > > >> > > >> Also, please share what could be the man
> > > page
> > > > > > > content
> > > > > > > >>> for
> > > > > > > >>> > >> > "apex".
> > > > > > > >>> > >> > > >> > > >>
> > > > > > > >>> > >> > > >> > > >> Thanks,
> > > > > > > >>> > >> > > >> > > >> Chinmay.
> > > > > > > >>> > >> > > >> > > >>
> > > > > > > >>> > >> > > >> > > >>
> > > > > > > >>> > >> > > >> > > >>
> > > > > > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay
> > > > > > Kolhatkar <
> > > > > > > >>> > >> > > >> > chinmay@apache.org>
> > > > > > > >>> > >> > > >> > > >> wrote:
> > > > > > > >>> > >> > > >> > > >>
> > > > > > > >>> > >> > > >> > > >> > Yes.. It'll be build from source tar
> > > > > downloaded
> > > > > > > >>> from one
> > > > > > > >>> > >> of
> > > > > > > >>> > >> > the
> > > > > > > >>> > >> > > >> > apache
> > > > > > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> > > > > > > >>> > >> > > >> > > >> >
> > > > > > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's
> sourced
> > > > from
> > > > > > > dtcli.
> > > > > > > >>> > This
> > > > > > > >>> > >> > > leaves
> > > > > > > >>> > >> > > >> us
> > > > > > > >>> > >> > > >> > 3
> > > > > > > >>> > >> > > >> > > >> > options:
> > > > > > > >>> > >> > > >> > > >> > If we should not have any dt-env.sh,
> > dtcli
> > > > > would
> > > > > > > >>> need a
> > > > > > > >>> > >> > change
> > > > > > > >>> > >> > > >> while
> > > > > > > >>> > >> > > >> > > >> > packaging.
> > > > > > > >>> > >> > > >> > > >> >
> > > > > > > >>> > >> > > >> > > >> > But if we want to use dtcli as it is,
> > then
> > > > we
> > > > > > > would
> > > > > > > >>> need
> > > > > > > >>> > >> > > >> dt-env.sh
> > > > > > > >>> > >> > > >> > > >> >
> > > > > > > >>> > >> > > >> > > >> >
> > > > > > > >>> > >> > > >> > > >> >
> > > > > > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM,
> Thomas
> > > > Weise
> > > > > <
> > > > > > > >>> > >> > > >> > thomas@datatorrent.com
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > > >> > wrote:
> > > > > > > >>> > >> > > >> > > >> >
> > > > > > > >>> > >> > > >> > > >> >> Looks good overall, though there
> > > shouldn't
> > > > be
> > > > > > any
> > > > > > > >>> > >> dt-env.sh
> > > > > > > >>> > >> > > >> > > >> >>
> > > > > > > >>> > >> > > >> > > >> >> What will the bigtop package be built
> > > from,
> > > > > the
> > > > > > > >>> source
> > > > > > > >>> > >> tar?
> > > > > > > >>> > >> > > >> > > >> >>
> > > > > > > >>> > >> > > >> > > >> >>
> > > > > > > >>> > >> > > >> > > >> >>
> > > > > > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM,
> > Chinmay
> > > > > > > Kolhatkar
> > > > > > > >>> <
> > > > > > > >>> > >> > > >> > > chinmay@apache.org
> > > > > > > >>> > >> > > >> > > >> >
> > > > > > > >>> > >> > > >> > > >> >> wrote:
> > > > > > > >>> > >> > > >> > > >> >>
> > > > > > > >>> > >> > > >> > > >> >> > Hi Thomas,
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > First, we're not changing the name
> > > > > anywhere.
> > > > > > > >>> We'll
> > > > > > > >>> > >> follow
> > > > > > > >>> > >> > > what
> > > > > > > >>> > >> > > >> > > >> >> currently is
> > > > > > > >>> > >> > > >> > > >> >> > in source tarball for
> > 3.3.0-incubating
> > > > > > version.
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a
> directory
> > > > > > structure
> > > > > > > >>> below
> > > > > > > >>> > >> which
> > > > > > > >>> > >> > > is
> > > > > > > >>> > >> > > >> > > inline
> > > > > > > >>> > >> > > >> > > >> >> with
> > > > > > > >>> > >> > > >> > > >> >> > other existing integrations with
> > > bigtop.
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > The need for each file is as
> follows:
> > > > > > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked
> > > from
> > > > > > > >>> > >> > > >> engine/src/main/scripts/
> > > > > > > >>> > >> > > >> > > of
> > > > > > > >>> > >> > > >> > > >> >> source
> > > > > > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source
> > > from
> > > > > > dtcli.
> > > > > > > >>> This
> > > > > > > >>> > >> > > searches
> > > > > > > >>> > >> > > >> for
> > > > > > > >>> > >> > > >> > > >> hadoop
> > > > > > > >>> > >> > > >> > > >> >> > binary path and exports an env
> > variable
> > > > for
> > > > > > > >>> dtcli to
> > > > > > > >>> > >> use
> > > > > > > >>> > >> > it.
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This
> > list
> > > > is
> > > > > > > >>> extracted
> > > > > > > >>> > >> in a
> > > > > > > >>> > >> > > >> similar
> > > > > > > >>> > >> > > >> > > >> way to
> > > > > > > >>> > >> > > >> > > >> >> > how DT Community Edition finds
> > > dependency
> > > > > > jars.
> > > > > > > >>> > >> > > >> > > >> >> > Please note that this is the first
> > > > > iteration
> > > > > > > >>> list of
> > > > > > > >>> > >> jars.
> > > > > > > >>> > >> > > I'm
> > > > > > > >>> > >> > > >> > > >> trying to
> > > > > > > >>> > >> > > >> > > >> >> > narrow this down to only those
> which
> > > are
> > > > > > really
> > > > > > > >>> > >> required.
> > > > > > > >>> > >> > > >> > > >> >> > The test that I'm running to check
> if
> > > > dtcli
> > > > > > > runs
> > > > > > > >>> fine
> > > > > > > >>> > >> with
> > > > > > > >>> > >> > > >> given
> > > > > > > >>> > >> > > >> > > >> >> dependency
> > > > > > > >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status
> for
> > > pi
> > > > > > demo.
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > Above is the directory structure
> > which
> > > is
> > > > > > > >>> required
> > > > > > > >>> > for
> > > > > > > >>> > >> CLI
> > > > > > > >>> > >> > > to
> > > > > > > >>> > >> > > >> > work.
> > > > > > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has
> > > mentioned
> > > > > are
> > > > > > > some
> > > > > > > >>> > >> > additional
> > > > > > > >>> > >> > > >> > files
> > > > > > > >>> > >> > > >> > > >> which
> > > > > > > >>> > >> > > >> > > >> >> > makes the like of administrator
> > easier.
> > > > For
> > > > > > eg.
> > > > > > > >>> > >> /etc/skel
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > Please let me know if this is
> inline
> > > with
> > > > > > what
> > > > > > > >>> you're
> > > > > > > >>> > >> > > >> thinking.
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > Thanks,
> > > > > > > >>> > >> > > >> > > >> >> > Chinmay.
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > .
> > > > > > > >>> > >> > > >> > > >> >> > |-- bin
> > > > > > > >>> > >> > > >> > > >> >> > |   `-- dtcli
> > > > > > > >>> > >> > > >> > > >> >> > |-- conf
> > > > > > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > > > > > > >>> > >> > > >> > > >> >> > `-- lib
> > > > > > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> apex-api-3.3.0-incubating.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > > > > > apex-bufferserver-3.3.0-incubating.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > > apex-common-3.3.0-incubating.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> async-http-client-1.7.20.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > grizzly-http-servlet-2.1.2.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> hadoop-common-2.2.0-tests.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> jackson-mapper-asl-1.9.2.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> jersey-apache-client4-1.9.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > jetty-http-8.1.10.v20130312.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> jetty-io-8.1.10.v20130312.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > jetty-util-8.1.10.v20130312.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |--
> > validation-api-1.1.0.Final.jar
> > > > > > > >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > > > > > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM,
> > Thomas
> > > > > > Weise <
> > > > > > > >>> > >> > > >> > > >> thomas@datatorrent.com>
> > > > > > > >>> > >> > > >> > > >> >> > wrote:
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >> > > Chinmay,
> > > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > > >>> > >> > > >> > > >> >> > > Before discussing where to put
> the
> > > > files,
> > > > > > > let's
> > > > > > > >>> > make
> > > > > > > >>> > >> > sure
> > > > > > > >>> > >> > > >> they
> > > > > > > >>> > >> > > >> > > are
> > > > > > > >>> > >> > > >> > > >> >> really
> > > > > > > >>> > >> > > >> > > >> >> > > needed for the operation of the
> > CLI.
> > > As
> > > > > for
> > > > > > > >>> names,
> > > > > > > >>> > >> > > anything
> > > > > > > >>> > >> > > >> new
> > > > > > > >>> > >> > > >> > > >> needs
> > > > > > > >>> > >> > > >> > > >> >> to
> > > > > > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and
> should
> > > > > follow
> > > > > > > >>> common
> > > > > > > >>> > >> > > >> conventions,
> > > > > > > >>> > >> > > >> > > >> >> especially
> > > > > > > >>> > >> > > >> > > >> >> > > in Bigtop where there are many
> > > existing
> > > > > > > >>> > integrations
> > > > > > > >>> > >> to
> > > > > > > >>> > >> > > look
> > > > > > > >>> > >> > > >> > at.
> > > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > > >>> > >> > > >> > > >> >> > > Thanks,
> > > > > > > >>> > >> > > >> > > >> >> > > Thomas
> > > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM,
> > > > Chinmay
> > > > > > > >>> Kolhatkar
> > > > > > > >>> > <
> > > > > > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > > > > > > >>> > >> > > >> > > >> >> > > wrote:
> > > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep and
> > > > > Aniruddha.
> > > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to
> > > change
> > > > > the
> > > > > > > >>> dtcli
> > > > > > > >>> > >> > > >> considering it
> > > > > > > >>> > >> > > >> > > >> works
> > > > > > > >>> > >> > > >> > > >> >> > with
> > > > > > > >>> > >> > > >> > > >> >> > > DT
> > > > > > > >>> > >> > > >> > > >> >> > > > Community edition. We can keep
> > the
> > > > > > > directory
> > > > > > > >>> > >> structure
> > > > > > > >>> > >> > > >> > similar
> > > > > > > >>> > >> > > >> > > to
> > > > > > > >>> > >> > > >> > > >> >> that.
> > > > > > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which
> > > > contains
> > > > > > > >>> > information
> > > > > > > >>> > >> > > >> required
> > > > > > > >>> > >> > > >> > for
> > > > > > > >>> > >> > > >> > > >> >> dtcli
> > > > > > > >>> > >> > > >> > > >> >> > to
> > > > > > > >>> > >> > > >> > > >> >> > > > launch.
> > > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop
> > > community
> > > > > > that
> > > > > > > >>> > whether
> > > > > > > >>> > >> > they
> > > > > > > >>> > >> > > >> > > >> facilitate
> > > > > > > >>> > >> > > >> > > >> >> the
> > > > > > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs
> before
> > > user
> > > > > is
> > > > > > > >>> created.
> > > > > > > >>> > >> In
> > > > > > > >>> > >> > > >> either
> > > > > > > >>> > >> > > >> > > case,
> > > > > > > >>> > >> > > >> > > >> >> > current
> > > > > > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in
> > home
> > > > > > > directory.
> > > > > > > >>> > Also
> > > > > > > >>> > >> > > before
> > > > > > > >>> > >> > > >> > > putting
> > > > > > > >>> > >> > > >> > > >> >> > > anything
> > > > > > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define
> > what
> > > > are
> > > > > > the
> > > > > > > >>> > default
> > > > > > > >>> > >> > > >> contents
> > > > > > > >>> > >> > > >> > > that
> > > > > > > >>> > >> > > >> > > >> >> > should
> > > > > > > >>> > >> > > >> > > >> >> > > go
> > > > > > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is
> no
> > > > > > defaults,
> > > > > > > >>> > >> probably we
> > > > > > > >>> > >> > > >> should
> > > > > > > >>> > >> > > >> > > not
> > > > > > > >>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach
> looks
> > > > nice.
> > > > > > > This
> > > > > > > >>> way
> > > > > > > >>> > >> > > contents
> > > > > > > >>> > >> > > >> of
> > > > > > > >>> > >> > > >> > > >> >> dt-env.sh
> > > > > > > >>> > >> > > >> > > >> >> > > are
> > > > > > > >>> > >> > > >> > > >> >> > > > present as env variables. But I
> > > see a
> > > > > > catch
> > > > > > > >>> > there.
> > > > > > > >>> > >> > > Adding
> > > > > > > >>> > >> > > >> > > >> dt-env.sh
> > > > > > > >>> > >> > > >> > > >> >> to
> > > > > > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all
> the
> > > > > > variables
> > > > > > > >>> > >> available
> > > > > > > >>> > >> > at
> > > > > > > >>> > >> > > >> > > runtime
> > > > > > > >>> > >> > > >> > > >> all
> > > > > > > >>> > >> > > >> > > >> >> > the
> > > > > > > >>> > >> > > >> > > >> >> > > > time. I feel a little skeptical
> > > about
> > > > > > that.
> > > > > > > >>> > Maybe a
> > > > > > > >>> > >> > > >> possible
> > > > > > > >>> > >> > > >> > > >> >> collision
> > > > > > > >>> > >> > > >> > > >> >> > > can
> > > > > > > >>> > >> > > >> > > >> >> > > > occur with other application
> > vars.
> > > > > > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does
> > source
> > > > > > > >>> > >> > "../conf/dt-env.sh"
> > > > > > > >>> > >> > > >> and
> > > > > > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those
> > > variables
> > > > > are
> > > > > > > >>> anyway
> > > > > > > >>> > >> > > >> available.
> > > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR
> > effort
> > > is
> > > > > > > >>> happening
> > > > > > > >>> > at
> > > > > > > >>> > >> > > Bigtop
> > > > > > > >>> > >> > > >> > > >> >> > > Jira/repository.
> > > > > > > >>> > >> > > >> > > >> >> > > > Packaging related code usually
> > goes
> > > > > > there.
> > > > > > > >>> > (That's
> > > > > > > >>> > >> > what
> > > > > > > >>> > >> > > >> all
> > > > > > > >>> > >> > > >> > the
> > > > > > > >>> > >> > > >> > > >> >> > > components
> > > > > > > >>> > >> > > >> > > >> >> > > > in bigtop does).
> > > > > > > >>> > >> > > >> > > >> >> > > > Having said that, once a PR is
> > > > created,
> > > > > > > I'll
> > > > > > > >>> be
> > > > > > > >>> > >> > sharing
> > > > > > > >>> > >> > > >> the
> > > > > > > >>> > >> > > >> > > link
> > > > > > > >>> > >> > > >> > > >> of
> > > > > > > >>> > >> > > >> > > >> >> PR
> > > > > > > >>> > >> > > >> > > >> >> > > > here, so that apex community as
> > > well
> > > > > can
> > > > > > > >>> review
> > > > > > > >>> > it.
> > > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > Thanks,
> > > > > > > >>> > >> > > >> > > >> >> > > > Chinmay.
> > > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57
> PM,
> > > > > > Aniruddha
> > > > > > > >>> > >> Thombare <
> > > > > > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com>
> > wrote:
> > > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and
> approach.
> > > > > > > >>> > >> > > >> > > >> >> > > > > We may need to iron out
> details
> > > > about
> > > > > > > exact
> > > > > > > >>> > paths
> > > > > > > >>> > >> > etc.
> > > > > > > >>> > >> > > >> > > >> >> > > > > Which can be done on jira /
> PR
> > > > > > comments.
> > > > > > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm
> > > Pradeep
> > > > > A.
> > > > > > > >>> Dalvi <
> > > > > > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > > > > > > >>> > >> > > >> > > >> >> > > > > wrote:
> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at
> 2:51
> > > PM,
> > > > > > > Chinmay
> > > > > > > >>> > >> > Kolhatkar <
> > > > > > > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > wrote:
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar
> > > > present
> > > > > in
> > > > > > > DT
> > > > > > > >>> > >> community
> > > > > > > >>> > >> > > >> > edition
> > > > > > > >>> > >> > > >> > > >> will
> > > > > > > >>> > >> > > >> > > >> >> be
> > > > > > > >>> > >> > > >> > > >> >> > > > > included
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of jars
> I
> > > > found
> > > > > > from
> > > > > > > >>> > common
> > > > > > > >>> > >> > > >> > dependencies
> > > > > > > >>> > >> > > >> > > >> >> between
> > > > > > > >>> > >> > > >> > > >> >> > > of
> > > > > > > >>> > >> > > >> > > >> >> > > > > apex
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> jackson-core-asl-1.9.2.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > jackson-mapper-asl-1.9.2.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > async-http-client-1.7.20.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > validation-api-1.1.0.Final.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> commons-beanutils-1.8.3.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > jersey-apache-client4-1.9.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > hadoop-common-2.2.0-tests.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running
> > > some
> > > > > > tests
> > > > > > > do
> > > > > > > >>> > check
> > > > > > > >>> > >> > that
> > > > > > > >>> > >> > > >> > dtcli
> > > > > > > >>> > >> > > >> > > >> works
> > > > > > > >>> > >> > > >> > > >> >> > > > properly
> > > > > > > >>> > >> > > >> > > >> >> > > > > > for
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of
> > apps
> > > > with
> > > > > > > only
> > > > > > > >>> > these
> > > > > > > >>> > >> > > >> libraries
> > > > > > > >>> > >> > > >> > > >> >> present in
> > > > > > > >>> > >> > > >> > > >> >> > > > > > isolation
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > without dependency on
> local
> > > m2.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there
> are
> > > > > unwanted
> > > > > > > >>> jars
> > > > > > > >>> > >> which
> > > > > > > >>> > >> > are
> > > > > > > >>> > >> > > >> used
> > > > > > > >>> > >> > > >> > > for
> > > > > > > >>> > >> > > >> > > >> >> > compile
> > > > > > > >>> > >> > > >> > > >> >> > > > > time
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > dependency and not
> runtime
> > in
> > > > > above
> > > > > > > >>> list
> > > > > > > >>> > >> which I
> > > > > > > >>> > >> > > can
> > > > > > > >>> > >> > > >> > drop
> > > > > > > >>> > >> > > >> > > >> to
> > > > > > > >>> > >> > > >> > > >> >> keep
> > > > > > > >>> > >> > > >> > > >> >> > > > > package
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not
> > sure
> > > if
> > > > > all
> > > > > > > of
> > > > > > > >>> the
> > > > > > > >>> > >> > > >> dt-env.sh is
> > > > > > > >>> > >> > > >> > > >> >> required.
> > > > > > > >>> > >> > > >> > > >> >> > > > Only
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > required values I see in
> it
> > > are
> > > > > > > >>> classpath.
> > > > > > > >>> > >> and
> > > > > > > >>> > >> > > >> dtcli is
> > > > > > > >>> > >> > > >> > > >> >> capable
> > > > > > > >>> > >> > > >> > > >> >> > of
> > > > > > > >>> > >> > > >> > > >> >> > > > > > building
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes,
> > we'll
> > > > have
> > > > > > to
> > > > > > > >>> see
> > > > > > > >>> > how
> > > > > > > >>> > >> > dtcli
> > > > > > > >>> > >> > > >> can
> > > > > > > >>> > >> > > >> > > find
> > > > > > > >>> > >> > > >> > > >> >> the
> > > > > > > >>> > >> > > >> > > >> >> > jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > path,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such
> conf
> > > > > files.
> > > > > > If
> > > > > > > >>> we
> > > > > > > >>> > >> need a
> > > > > > > >>> > >> > > >> change
> > > > > > > >>> > >> > > >> > in
> > > > > > > >>> > >> > > >> > > >> >> dtcli,
> > > > > > > >>> > >> > > >> > > >> >> > > then
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > community's opinion is
> > > required
> > > > > for
> > > > > > > >>> whether
> > > > > > > >>> > >> > dtcli
> > > > > > > >>> > >> > > >> > should
> > > > > > > >>> > >> > > >> > > >> >> change
> > > > > > > >>> > >> > > >> > > >> >> > in
> > > > > > > >>> > >> > > >> > > >> >> > > > our
> > > > > > > >>> > >> > > >> > > >> >> > > > > > repo
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with
> > > required
> > > > > > > changes
> > > > > > > >>> > >> exist in
> > > > > > > >>> > >> > > >> bigtop
> > > > > > > >>> > >> > > >> > > >> repo
> > > > > > > >>> > >> > > >> > > >> >> > until
> > > > > > > >>> > >> > > >> > > >> >> > > we
> > > > > > > >>> > >> > > >> > > >> >> > > > > > make
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > Can we have these file
> paths
> > > set
> > > > > > using
> > > > > > > >>> > >> environment
> > > > > > > >>> > >> > > >> > > variables
> > > > > > > >>> > >> > > >> > > >> >> with
> > > > > > > >>> > >> > > >> > > >> >> > > some
> > > > > > > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> > > > > > > >>> > >> > > >> > > >> >> > > > > > Then we can set such params
> > in
> > > > > > > dt-env.sh.
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is
> > > this
> > > > > file
> > > > > > > >>> > different
> > > > > > > >>> > >> > from
> > > > > > > >>> > >> > > >> > > >> >> dt-site.xml?
> > > > > > > >>> > >> > > >> > > >> >> > Not
> > > > > > > >>> > >> > > >> > > >> >> > > > > sure
> > > > > > > >>> > >> > > >> > > >> >> > > > > > if
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would
> > > help.
> > > > > > > Files &
> > > > > > > >>> > Dirs
> > > > > > > >>> > >> > from
> > > > > > > >>> > >> > > >> > > /etc/skel
> > > > > > > >>> > >> > > >> > > >> >> are
> > > > > > > >>> > >> > > >> > > >> >> > > > copied
> > > > > > > >>> > >> > > >> > > >> >> > > > > to
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > home of new user when
> > useradd
> > > > > > program
> > > > > > > >>> is
> > > > > > > >>> > >> called.
> > > > > > > >>> > >> > > But
> > > > > > > >>> > >> > > >> > for
> > > > > > > >>> > >> > > >> > > >> >> existing
> > > > > > > >>> > >> > > >> > > >> >> > > > users
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any use.
> > > > > Moreover,
> > > > > > > >>> dtcli
> > > > > > > >>> > >> > creates
> > > > > > > >>> > >> > > >> > ~/.dt/
> > > > > > > >>> > >> > > >> > > on
> > > > > > > >>> > >> > > >> > > >> >> the
> > > > > > > >>> > >> > > >> > > >> >> > fly
> > > > > > > >>> > >> > > >> > > >> >> > > > if
> > > > > > > >>> > >> > > >> > > >> >> > > > > > not
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > encountered for the first
> > > time.
> > > > > > Again
> > > > > > > >>> > >> correct me
> > > > > > > >>> > >> > > if
> > > > > > > >>> > >> > > >> I'm
> > > > > > > >>> > >> > > >> > > >> wrong.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right. However
> > > this
> > > > > > would
> > > > > > > >>> > >> probably be
> > > > > > > >>> > >> > > >> > > necessary
> > > > > > > >>> > >> > > >> > > >> >> step
> > > > > > > >>> > >> > > >> > > >> >> > for
> > > > > > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also
> get
> > > > > > installed
> > > > > > > >>> > during
> > > > > > > >>> > >> OS
> > > > > > > >>> > >> > > >> install
> > > > > > > >>> > >> > > >> > > and
> > > > > > > >>> > >> > > >> > > >> >> > before
> > > > > > > >>> > >> > > >> > > >> >> > > > user
> > > > > > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc &
> > > > bash_profile
> > > > > > ->
> > > > > > > >>> I'm
> > > > > > > >>> > not
> > > > > > > >>> > >> > sure
> > > > > > > >>> > >> > > >> its
> > > > > > > >>> > >> > > >> > the
> > > > > > > >>> > >> > > >> > > >> best
> > > > > > > >>> > >> > > >> > > >> >> > > idea.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > We'll
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in
> > location
> > > > > which
> > > > > > is
> > > > > > > >>> by
> > > > > > > >>> > >> default
> > > > > > > >>> > >> > > >> > present
> > > > > > > >>> > >> > > >> > > in
> > > > > > > >>> > >> > > >> > > >> >> path
> > > > > > > >>> > >> > > >> > > >> >> > > i.e.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > For other env variables
> > > specific
> > > > to
> > > > > > > apex,
> > > > > > > >>> > I'll
> > > > > > > >>> > >> > > prefer
> > > > > > > >>> > >> > > >> to
> > > > > > > >>> > >> > > >> > > use
> > > > > > > >>> > >> > > >> > > >> >> > > dt-env.sh
> > > > > > > >>> > >> > > >> > > >> >> > > > > and
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather
> > > than
> > > > > > > changing
> > > > > > > >>> > >> bashrc
> > > > > > > >>> > >> > > >> etc...
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh
> to
> > > > > > > >>> /etc/profile.d/
> > > > > > > >>> > >> shall
> > > > > > > >>> > >> > > be
> > > > > > > >>> > >> > > >> the
> > > > > > > >>> > >> > > >> > > >> >> cleaner
> > > > > > > >>> > >> > > >> > > >> >> > way
> > > > > > > >>> > >> > > >> > > >> >> > > > to
> > > > > > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > Thanks,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at
> > 1:07
> > > > PM,
> > > > > > > >>> Aniruddha
> > > > > > > >>> > >> > > Thombare
> > > > > > > >>> > >> > > >> <
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> aniruddha@datatorrent.com>
> > > > > wrote:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > We need to consider
> > > > following:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > System wide default
> > config
> > > > > files
> > > > > > > can
> > > > > > > >>> be
> > > > > > > >>> > >> > located
> > > > > > > >>> > >> > > at
> > > > > > > >>> > >> > > >> > > >> following
> > > > > > > >>> > >> > > >> > > >> >> > > > > locations:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh
> (we
> > > may
> > > > > have
> > > > > > > to
> > > > > > > >>> > change
> > > > > > > >>> > >> > > dtcli
> > > > > > > >>> > >> > > >> > > >> behaviour
> > > > > > > >>> > >> > > >> > > >> >> on
> > > > > > > >>> > >> > > >> > > >> >> > > how
> > > > > > > >>> > >> > > >> > > >> >> > > > it
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > finds
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml
> (we
> > > may
> > > > > > have
> > > > > > > to
> > > > > > > >>> > >> change
> > > > > > > >>> > >> > > dtcli
> > > > > > > >>> > >> > > >> > > >> >> behaviour on
> > > > > > > >>> > >> > > >> > > >> >> > > how
> > > > > > > >>> > >> > > >> > > >> >> > > > > it
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > /etc/skel/.dt/dt-sited.xml
> > > > and
> > > > > > > other
> > > > > > > >>> > files
> > > > > > > >>> > >> > (for
> > > > > > > >>> > >> > > >> new
> > > > > > > >>> > >> > > >> > > users
> > > > > > > >>> > >> > > >> > > >> >> that
> > > > > > > >>> > >> > > >> > > >> >> > > will
> > > > > > > >>> > >> > > >> > > >> >> > > > > be
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > created in system in
> > > future.)
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > We may also have to
> > modify
> > > > > > bashrc /
> > > > > > > >>> > >> > bashprofile
> > > > > > > >>> > >> > > >> for
> > > > > > > >>> > >> > > >> > > >> >> population
> > > > > > > >>> > >> > > >> > > >> >> > > any
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > variables that are
> > > required.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in
> your
> > > > > > comments /
> > > > > > > >>> > >> > suggestion.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at
> > > 1:16
> > > > > AM,
> > > > > > > >>> Thomas
> > > > > > > >>> > >> Weise
> > > > > > > >>> > >> > <
> > > > > > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > -->
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016
> at
> > > 9:05
> > > > > AM,
> > > > > > > >>> Chinmay
> > > > > > > >>> > >> > > >> Kolhatkar <
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> chinmay@datatorrent.com
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with
> > > packaging
> > > > > > > effort
> > > > > > > >>> > >> (rpm+deb)
> > > > > > > >>> > >> > > of
> > > > > > > >>> > >> > > >> > apex,
> > > > > > > >>> > >> > > >> > > >> here
> > > > > > > >>> > >> > > >> > > >> >> > are
> > > > > > > >>> > >> > > >> > > >> >> > > > some
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > about package
> > structure
> > > > > etc..
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it
> on
> > > > > bbigtop
> > > > > > > >>> mailing
> > > > > > > >>> > >> > list, I
> > > > > > > >>> > >> > > >> have
> > > > > > > >>> > >> > > >> > > >> some
> > > > > > > >>> > >> > > >> > > >> >> > > question
> > > > > > > >>> > >> > > >> > > >> >> > > > > for
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > apex
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory
> > > > > structure
> > > > > > of
> > > > > > > >>> apex
> > > > > > > >>> > >> > package
> > > > > > > >>> > >> > > >> for
> > > > > > > >>> > >> > > >> > > both
> > > > > > > >>> > >> > > >> > > >> >> deb &
> > > > > > > >>> > >> > > >> > > >> >> > > > rpm:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > /usr/lib/apex/bin/dtcli
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >>
> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > /usr/lib/apex/lib/<other
> > > > > > > >>> dependent
> > > > > > > >>> > >> jars>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> > > > > > > >>> > /usr/lib/apex/bin/dtcli
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >>
> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > /usr/share/doc/apex/NOTICE
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
> > > > > 3.3.0-incubating.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent
> > jars>
> > > =
> > > > > All
> > > > > > > the
> > > > > > > >>> 3rd
> > > > > > > >>> > >> party
> > > > > > > >>> > >> > > >> jars
> > > > > > > >>> > >> > > >> > > which
> > > > > > > >>> > >> > > >> > > >> >> are
> > > > > > > >>> > >> > > >> > > >> >> > > > > required
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > for
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > apex
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the
> > > > > > > dependencies
> > > > > > > >>> are
> > > > > > > >>> > >> > > packaged
> > > > > > > >>> > >> > > >> as
> > > > > > > >>> > >> > > >> > > >> part of
> > > > > > > >>> > >> > > >> > > >> >> > > > rpm/deb
> > > > > > > >>> > >> > > >> > > >> >> > > > > by
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > any
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what
> > > those
> > > > > jars
> > > > > > > >>> are in
> > > > > > > >>> > >> > Bigtop
> > > > > > > >>> > >> > > >> > > context?
> > > > > > > >>> > >> > > >> > > >> >> Same
> > > > > > > >>> > >> > > >> > > >> >> > as
> > > > > > > >>> > >> > > >> > > >> >> > > > > > shipped
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > with
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > DT community addition
> > > under
> > > > > > lib/
> > > > > > > ?
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> =
> > > > > Licenses
> > > > > > of
> > > > > > > >>> > >> > > corresponding
> > > > > > > >>> > >> > > >> of
> > > > > > > >>> > >> > > >> > > 3rd
> > > > > > > >>> > >> > > >> > > >> >> party
> > > > > > > >>> > >> > > >> > > >> >> > > jars
> > > > > > > >>> > >> > > >> > > >> >> > > > > > which
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > needs
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to included while
> > > > > packaging.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions related
> to
> > > > this:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call
> the
> > > cli
> > > > > of
> > > > > > > >>> apex as
> > > > > > > >>> > >> > "apex"
> > > > > > > >>> > >> > > >> > instead
> > > > > > > >>> > >> > > >> > > >> of
> > > > > > > >>> > >> > > >> > > >> >> > > "dtcli"
> > > > > > > >>> > >> > > >> > > >> >> > > > in
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > I think we should
> keep
> > > the
> > > > > name
> > > > > > > >>> until
> > > > > > > >>> > we
> > > > > > > >>> > >> are
> > > > > > > >>> > >> > > >> able
> > > > > > > >>> > >> > > >> > to
> > > > > > > >>> > >> > > >> > > >> >> change
> > > > > > > >>> > >> > > >> > > >> >> > it
> > > > > > > >>> > >> > > >> > > >> >> > > in
> > > > > > > >>> > >> > > >> > > >> >> > > > > > Apex.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may
> get
> > > > > > confusing.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all
> > > > softwares
> > > > > > in
> > > > > > > >>> bigtop
> > > > > > > >>> > >> have
> > > > > > > >>> > >> > > man
> > > > > > > >>> > >> > > >> > page
> > > > > > > >>> > >> > > >> > > >> for
> > > > > > > >>> > >> > > >> > > >> >> > their
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we should
> > have
> > > it
> > > > > too
> > > > > > > for
> > > > > > > >>> > >> dtcli.
> > > > > > > >>> > >> > Is
> > > > > > > >>> > >> > > >> there
> > > > > > > >>> > >> > > >> > > any
> > > > > > > >>> > >> > > >> > > >> >> > > > > documentation
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > which I
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man
> > > page?
> > > > or
> > > > > > can
> > > > > > > >>> I use
> > > > > > > >>> > >> > output
> > > > > > > >>> > >> > > >> of
> > > > > > > >>> > >> > > >> > > >> "dtcli
> > > > > > > >>> > >> > > >> > > >> >> > > --help"?
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to
> call
> > > > > version
> > > > > > > of
> > > > > > > >>> apex
> > > > > > > >>> > >> in
> > > > > > > >>> > >> > > >> Bigtop
> > > > > > > >>> > >> > > >> > as
> > > > > > > >>> > >> > > >> > > >> >> 3.3.0 OR
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for
> apex
> > > > > package
> > > > > > of
> > > > > > > >>> > bigtop
> > > > > > > >>> > >> to
> > > > > > > >>> > >> > > >> depend
> > > > > > > >>> > >> > > >> > on
> > > > > > > >>> > >> > > >> > > >> >> 2.7.1
> > > > > > > >>> > >> > > >> > > >> >> > > > > version
> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any
> > > > problems
> > > > > > > that
> > > > > > > >>> we
> > > > > > > >>> > see
> > > > > > > >>> > >> > with
> > > > > > > >>> > >> > > >> this
> > > > > > > >>> > >> > > >> > > >> >> > dependency?
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Should work without
> > > > changed.
> > > > > I
> > > > > > > >>> thought
> > > > > > > >>> > we
> > > > > > > >>> > >> > > >> certified
> > > > > > > >>> > >> > > >> > > >> >> against
> > > > > > > >>> > >> > > >> > > >> >> > > > 2.7.0?
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the
> > > > apache
> > > > > > > mirror
> > > > > > > >>> > from
> > > > > > > >>> > >> > which
> > > > > > > >>> > >> > > >> > bigtop
> > > > > > > >>> > >> > > >> > > >> will
> > > > > > > >>> > >> > > >> > > >> >> > pick
> > > > > > > >>> > >> > > >> > > >> >> > > > the
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > apex
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > source for
> > compilation
> > > > and
> > > > > > > >>> packaging.
> > > > > > > >>> > >> > Please
> > > > > > > >>> > >> > > >> > > correct
> > > > > > > >>> > >> > > >> > > >> if
> > > > > > > >>> > >> > > >> > > >> >> > > > > incorrect:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > > >>> > >> > >
> > > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
> > > > > apache.osuosl.org
> > > > > > is
> > > > > > > >>> the
> > > > > > > >>> > >> mirror
> > > > > > > >>> > >> > > >> used
> > > > > > > >>> > >> > > >> > by
> > > > > > > >>> > >> > > >> > > >> all
> > > > > > > >>> > >> > > >> > > >> >> the
> > > > > > > >>> > >> > > >> > > >> >> > > > > > softwares
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > in
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share your
> > > > opinion.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016
> > at
> > > > > 11:19
> > > > > > > AM,
> > > > > > > >>> > >> Chinmay
> > > > > > > >>> > >> > > >> > Kolhatkar
> > > > > > > >>> > >> > > >> > > <
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > chinmay@datatorrent.com>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to
> > > work
> > > > on
> > > > > > the
> > > > > > > >>> > >> packaging.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already
> > started
> > > > > > > >>> discussion on
> > > > > > > >>> > >> > bigtop
> > > > > > > >>> > >> > > >> dev
> > > > > > > >>> > >> > > >> > > >> mailing
> > > > > > > >>> > >> > > >> > > >> >> > list
> > > > > > > >>> > >> > > >> > > >> >> > > > for
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also
> > > > > created a
> > > > > > > >>> Jira
> > > > > > > >>> > for
> > > > > > > >>> > >> the
> > > > > > > >>> > >> > > >> same.
> > > > > > > >>> > >> > > >> > > For
> > > > > > > >>> > >> > > >> > > >> >> this
> > > > > > > >>> > >> > > >> > > >> >> > > > > > communities
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here
> is
> > > the
> > > > > > bigtop
> > > > > > > >>> Jira:
> > > > > > > >>> > >> > > >> > BIGTOP-2313.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8,
> 2016
> > > at
> > > > > 1:13
> > > > > > > AM,
> > > > > > > >>> > >> Thomas
> > > > > > > >>> > >> > > >> Weise <
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see
> > anything
> > > > > under
> > > > > > > >>> > >> > prerequisites
> > > > > > > >>> > >> > > >> that
> > > > > > > >>> > >> > > >> > > >> would
> > > > > > > >>> > >> > > >> > > >> >> be a
> > > > > > > >>> > >> > > >> > > >> >> > > > > > problem.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > We
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF
> > licencing
> > > > > > > >>> compatibility
> > > > > > > >>> > >> as
> > > > > > > >>> > >> > > part
> > > > > > > >>> > >> > > >> of
> > > > > > > >>> > >> > > >> > > >> >> becoming
> > > > > > > >>> > >> > > >> > > >> >> > an
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > incubator
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on
> > the
> > > > > > > packaging
> > > > > > > >>> > during
> > > > > > > >>> > >> > the
> > > > > > > >>> > >> > > >> next
> > > > > > > >>> > >> > > >> > > >> weeks.
> > > > > > > >>> > >> > > >> > > >> >> > Since
> > > > > > > >>> > >> > > >> > > >> >> > > > the
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > work
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > will
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop,
> > > > related
> > > > > > > >>> > discussions
> > > > > > > >>> > >> and
> > > > > > > >>> > >> > > >> JIRAs
> > > > > > > >>> > >> > > >> > > >> should
> > > > > > > >>> > >> > > >> > > >> >> > also
> > > > > > > >>> > >> > > >> > > >> >> > > be
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to
> > > have
> > > > > the
> > > > > > > >>> > packaging
> > > > > > > >>> > >> in
> > > > > > > >>> > >> > > >> place
> > > > > > > >>> > >> > > >> > by
> > > > > > > >>> > >> > > >> > > >> end
> > > > > > > >>> > >> > > >> > > >> >> Feb.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4,
> > 2016
> > > at
> > > > > > 10:14
> > > > > > > >>> AM,
> > > > > > > >>> > >> > Chinmay
> > > > > > > >>> > >> > > >> > > >> Kolhatkar <
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > chinmay@datatorrent.com
> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're
> planning a
> > > > work
> > > > > on
> > > > > > > >>> adding
> > > > > > > >>> > >> > Apache
> > > > > > > >>> > >> > > >> Apex
> > > > > > > >>> > >> > > >> > > as a
> > > > > > > >>> > >> > > >> > > >> >> > > component
> > > > > > > >>> > >> > > >> > > >> >> > > > > to
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Apache
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the
> > > > > packaging
> > > > > > > >>> system
> > > > > > > >>> > for
> > > > > > > >>> > >> > the
> > > > > > > >>> > >> > > >> > Apache
> > > > > > > >>> > >> > > >> > > >> big
> > > > > > > >>> > >> > > >> > > >> >> data
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros
> > use
> > > > it,
> > > > > > > most
> > > > > > > >>> > >> recently
> > > > > > > >>> > >> > > EMR.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the
> > > tracking
> > > > > > Jira
> > > > > > > >>> task
> > > > > > > >>> > in
> > > > > > > >>> > >> > > >> APEXCORE
> > > > > > > >>> > >> > > >> > for
> > > > > > > >>> > >> > > >> > > >> the
> > > > > > > >>> > >> > > >> > > >> >> > same:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >>
> > > > > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan
> of
> > > > > > execution
> > > > > > > >>> is as
> > > > > > > >>> > >> > > follows:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1)
> Handle
> > > > > > > >>> prerequisites*
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop
> > has
> > > > some
> > > > > > > hard
> > > > > > > >>> and
> > > > > > > >>> > >> soft
> > > > > > > >>> > >> > > >> > > expectation
> > > > > > > >>> > >> > > >> > > >> >> for
> > > > > > > >>> > >> > > >> > > >> >> > new
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > components
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated
> into
> > > > > Bigtop.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the
> list
> > > of
> > > > > it:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >>
> > > > > > > >>> > >> > > >> > > >>
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> >
> > > > > > > >>> > >> > > >>
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> >
> > > > > > > >>> > >>
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them
> > seems
> > > > to
> > > > > be
> > > > > > > >>> > standard
> > > > > > > >>> > >> ASF
> > > > > > > >>> > >> > > >> based
> > > > > > > >>> > >> > > >> > > >> >> > > requirements,
> > > > > > > >>> > >> > > >> > > >> >> > > > > but
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > few
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > need
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked
> for:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software
> > > projects
> > > > > are
> > > > > > > >>> > expected
> > > > > > > >>> > >> to
> > > > > > > >>> > >> > be
> > > > > > > >>> > >> > > >> > > Licensed
> > > > > > > >>> > >> > > >> > > >> >> under
> > > > > > > >>> > >> > > >> > > >> >> > > > > Apache
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > License,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0
> (and
> > > > their
> > > > > > > >>> > >> dependencies
> > > > > > > >>> > >> > are
> > > > > > > >>> > >> > > >> > > expected
> > > > > > > >>> > >> > > >> > > >> >> to be
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > compatible
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > with
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is
> > > under
> > > > > ASL
> > > > > > > 2.0
> > > > > > > >>> but
> > > > > > > >>> > >> need
> > > > > > > >>> > >> > to
> > > > > > > >>> > >> > > >> > check
> > > > > > > >>> > >> > > >> > > if
> > > > > > > >>> > >> > > >> > > >> >> > > > > dependencies
> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible
> with
> > > ASL
> > > > > 2.0.
> > > > > > > >>> This I
> > > > > > > >>> > >> guess
> > > > > > > >>> > >> > > >> would
> > > > > > > >>> > >> > > >> > > be a
> > > > > > > >>> > >> > > >> > > >> >> > > > > verification
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > check.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software
> > > projects
> > > > > are
> > > > > > > >>> > expected
> > > > > > > >>> > >> to
> > > > > > > >>> > >> > be
> > > > > > > >>> > >> > > >> > > >> compatible
> > > > > > > >>> > >> > > >> > > >> >> > with
> > > > > > > >>> > >> > > >> > > >> >> > > > all
> > > > > > > >>> > >> > > >> > > >> >> > > > > of
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > the
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported
> > > platforms
> > > > > that
> > > > > > > >>> Bigtop
> > > > > > > >>> > >> > > >> distribution
> > > > > > > >>> > >> > > >> > > is
> > > > > > > >>> > >> > > >> > > >> >> > > targeting
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This
> needs
> > > > > > verifying
> > > > > > > >>> > whether
> > > > > > > >>> > >> > our
> > > > > > > >>> > >> > > >> > > software
> > > > > > > >>> > >> > > >> > > >> >> runs
> > > > > > > >>> > >> > > >> > > >> >> > > fine
> > > > > > > >>> > >> > > >> > > >> >> > > > in
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7
> > fedora-20
> > > > > > > >>> ubuntu-14.04
> > > > > > > >>> > >> > > debian-8
> > > > > > > >>> > >> > > >> > > >> >> > opensuse-13.2.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke
> > > tests
> > > > > that
> > > > > > > >>> should
> > > > > > > >>> > be
> > > > > > > >>> > >> > > added
> > > > > > > >>> > >> > > >> for
> > > > > > > >>> > >> > > >> > > >> >> > deployment.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying
> > the
> > > > > test
> > > > > > > >>> > artifacts
> > > > > > > >>> > >> > which
> > > > > > > >>> > >> > > >> goes
> > > > > > > >>> > >> > > >> > > >> beyond
> > > > > > > >>> > >> > > >> > > >> >> > > smoke
> > > > > > > >>> > >> > > >> > > >> >> > > > > test
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These
> are
> > > > > > basically
> > > > > > > >>> the
> > > > > > > >>> > >> > > integration
> > > > > > > >>> > >> > > >> > > tests
> > > > > > > >>> > >> > > >> > > >> for
> > > > > > > >>> > >> > > >> > > >> >> > > > > > verification
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > after
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment.
> This
> > > is
> > > > a
> > > > > > soft
> > > > > > > >>> > >> > requirement,
> > > > > > > >>> > >> > > >> but
> > > > > > > >>> > >> > > >> > > aim
> > > > > > > >>> > >> > > >> > > >> is
> > > > > > > >>> > >> > > >> > > >> >> to
> > > > > > > >>> > >> > > >> > > >> >> > > > > achieve
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > this
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > as
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least
> have
> > > > > > > >>> explanation why
> > > > > > > >>> > >> not
> > > > > > > >>> > >> > to
> > > > > > > >>> > >> > > >> > > include.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are
> any
> > > > from
> > > > > > the
> > > > > > > >>> link
> > > > > > > >>> > >> which
> > > > > > > >>> > >> > > >> > > explicitly
> > > > > > > >>> > >> > > >> > > >> >> needs
> > > > > > > >>> > >> > > >> > > >> >> > to
> > > > > > > >>> > >> > > >> > > >> >> > > > be
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > checked
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4,
> > > please
> > > > > let
> > > > > > > us
> > > > > > > >>> > know.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2)
> Adding
> > > Apex
> > > > > as
> > > > > > > >>> > component
> > > > > > > >>> > >> to
> > > > > > > >>> > >> > > >> Bigtop*
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of
> the
> > > mail
> > > > > > > >>> archive of
> > > > > > > >>> > >> > Bigtop,
> > > > > > > >>> > >> > > >> it
> > > > > > > >>> > >> > > >> > was
> > > > > > > >>> > >> > > >> > > >> >> learnt
> > > > > > > >>> > >> > > >> > > >> >> > > that
> > > > > > > >>> > >> > > >> > > >> >> > > > > the
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community want
> > to
> > > > see
> > > > > > the
> > > > > > > >>> > >> addition of
> > > > > > > >>> > >> > > new
> > > > > > > >>> > >> > > >> > > >> >> components
> > > > > > > >>> > >> > > >> > > >> >> > in
> > > > > > > >>> > >> > > >> > > >> >> > > > > > phases.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > Here
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > are
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This
> needs
> > > > > > creating
> > > > > > > of
> > > > > > > >>> > >> package
> > > > > > > >>> > >> > > i.e.
> > > > > > > >>> > >> > > >> > rpm
> > > > > > > >>> > >> > > >> > > &
> > > > > > > >>> > >> > > >> > > >> deb
> > > > > > > >>> > >> > > >> > > >> >> > > files.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > > > > > > >>> documentations/READMEs,
> > > > > > > >>> > >> > LICENSE,
> > > > > > > >>> > >> > > >> > > >> DISCLAMER,
> > > > > > > >>> > >> > > >> > > >> >> > NOTES
> > > > > > > >>> > >> > > >> > > >> >> > > > etc
> > > > > > > >>> > >> > > >> > > >> >> > > > > if
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > any
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> > > > > documentation
> > > > > > > that
> > > > > > > >>> > need
> > > > > > > >>> > >> to
> > > > > > > >>> > >> > be
> > > > > > > >>> > >> > > >> > added
> > > > > > > >>> > >> > > >> > > to
> > > > > > > >>> > >> > > >> > > >> >> > > > > distribution
> > > > > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > our
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> > license
> > > > > > > >>> information of
> > > > > > > >>> > >> > > >> > dependencies
> > > > > > > >>> > >> > > >> > > >> >> required
> > > > > > > >>> > >> > > >> > > >> >> > > to
> > > > > > > >>> > >> > > >> > > >> >> > > > be
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > added
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests
> > (at
> > > > > very
> > > > > > > >>> least)
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding
> > smoke
> > > > > test
> > > > > > > for
> > > > > > > >>> > >> > packaging.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet
> > recipes
> > > > for
> > > > > > > >>> automatic
> > > > > > > >>> > >> > > >> deployment
> > > > > > > >>> > >> > > >> > and
> > > > > > > >>> > >> > > >> > > >> >> > > > configuration
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add
> puppet
> > > > > recipes
> > > > > > > for
> > > > > > > >>> > >> > automatic
> > > > > > > >>> > >> > > >> > > >> deployment
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration
> > > tests
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For
> > > > verification
> > > > > > of
> > > > > > > >>> > >> > deployments.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license
> > > > clearance:
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run
> 'gradle
> > > rat'
> > > > > to
> > > > > > > make
> > > > > > > >>> > sure
> > > > > > > >>> > >> all
> > > > > > > >>> > >> > > new
> > > > > > > >>> > >> > > >> > > stuff
> > > > > > > >>> > >> > > >> > > >> is
> > > > > > > >>> > >> > > >> > > >> >> > > > compliant
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > with
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license
> > > > requirements.
> > > > > If
> > > > > > > >>> you add
> > > > > > > >>> > >> code
> > > > > > > >>> > >> > > >> > licenses
> > > > > > > >>> > >> > > >> > > >> >> under
> > > > > > > >>> > >> > > >> > > >> >> > > > > different
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would
> need
> > > to
> > > > be
> > > > > > > >>> listed in
> > > > > > > >>> > >> the
> > > > > > > >>> > >> > > >> NOTICE.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share
> > your
> > > > > > thoughts
> > > > > > > >>> on
> > > > > > > >>> > the
> > > > > > > >>> > >> > > >> approach.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
> > > > > > corresponding
> > > > > > > >>> > >> > communication
> > > > > > > >>> > >> > > >> on
> > > > > > > >>> > >> > > >> > > >> bigtop
> > > > > > > >>> > >> > > >> > > >> >> > > mailing
> > > > > > > >>> > >> > > >> > > >> >> > > > > list
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > as
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some
> > > > specific
> > > > > > > >>> > >> > > >> questions/suggestions
> > > > > > > >>> > >> > > >> > > >> >> related to
> > > > > > > >>> > >> > > >> > > >> >> > > > what
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > should
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > be
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the
> > > > package
> > > > > > and
> > > > > > > >>> what
> > > > > > > >>> > >> > should
> > > > > > > >>> > >> > > be
> > > > > > > >>> > >> > > >> > the
> > > > > > > >>> > >> > > >> > > >> smoke
> > > > > > > >>> > >> > > >> > > >> >> > > tests,
> > > > > > > >>> > >> > > >> > > >> >> > > > > but
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > in
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > the
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of
> not
> > > > having
> > > > > > too
> > > > > > > >>> much
> > > > > > > >>> > >> > content
> > > > > > > >>> > >> > > >> > here,
> > > > > > > >>> > >> > > >> > > >> we'll
> > > > > > > >>> > >> > > >> > > >> >> > put
> > > > > > > >>> > >> > > >> > > >> >> > > > the
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > questions
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail
> in
> > > > this
> > > > > > > >>> > mailthread.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > > >>> > >> > > >> > > >> >> >
> > > > > > > >>> > >> > > >> > > >> >>
> > > > > > > >>> > >> > > >> > > >> >
> > > > > > > >>> > >> > > >> > > >> >
> > > > > > > >>> > >> > > >> > > >>
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > > >
> > > > > > > >>> > >> > > >> > >
> > > > > > > >>> > >> > > >> >
> > > > > > > >>> > >> > > >>
> > > > > > > >>> > >> > > >
> > > > > > > >>> > >> > > >
> > > > > > > >>> > >> > >
> > > > > > > >>> > >> >
> > > > > > > >>> > >>
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

We should not package any Hadoop dependencies, they are provided by the
Bigtop stack.


On Thu, Feb 18, 2016 at 8:56 PM, Tushar Gosavi <tu...@datatorrent.com>
wrote:

> I just did a quick install of bigtop 1.1.0 on ubuntu and observed that
>  jersey-client-1.9.jar is already packaged in hadoop
> /usr/lib/hadoop/client/jersey-client.jar, so I guess it will be ok to
> include those in apex lib too. or If we are including hadoop jars
> in classpath while launching the apex application then it may not be
> needed.
>
> - Tushar.
>
>
>
> On Fri, Feb 19, 2016 at 8:44 AM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > No worries David. I've updated the manpage.
> >
> > Just an update: apex bigtop packaging for rpm+deb is done. Here is the
> > branch on which I'm working if someone wants to look at:
> > https://github.com/chinmaykolhatkar/bigtop/blob/BIGTOP-2314
> >
> > Only the 3rd questions below is the question that is left to be answered.
> > Once this is resolved, I'll create the pull request in bigtop.
> >
> > Following questions are related to licenses and inclusion of certain
> files
> > in apex rpm+deb:
> > 1. I'll be included LICENSE, DISCLAIMER, NOTICE from top folder of core
> > repo in our rpm and deb package. Those will be present in standard
> location
> > i.e. /usr/lib/apex/
> > Please let me know otherwise.
> >
> > 2. Most of our runtime dependencies has ASF license, but certain
> libraries
> > has MIT and BSD licenses, hence I've included those licenses too in rpm &
> > deb package in same location as above.
> > Please let me know otherwise.
> >
> > BSD - jline-2.11.jar, kryo-2.24.0.jar, minlog-1.2.jar
> > MIT - mbassador-1.1.9.jar
> >
> > 3. There are 2 jars (jersey-apache-client4-1.9.jar,
> jersey-client-1.9.jar)
> > which has duel license of CDDL + GPLv2 with classpath exception:
> > https://jersey.java.net/license.html
> > I'm not an expert here. Hence can someone let me know whether these
> > libraries is OK to be packaged in rpm+deb of apex?
> >
> > If yes, should we also include that license in our package?
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> > On Thu, Feb 18, 2016 at 11:35 PM, David Yan <da...@datatorrent.com>
> wrote:
> >
> > > I should've mentioned that I had a pull request that was merged
> yesterday
> > > that corrected the statement:
> > >
> > > https://github.com/DataTorrent/docs/pull/45
> > >
> > > On Thu, Feb 18, 2016 at 12:07 AM, Tushar Gosavi <
> tushar@datatorrent.com>
> > > wrote:
> > >
> > > > In following statement, can you replace dtGateway with
> > ApplicationMaster
> > > > and Yarn. or may be remove this statement entirely.
> > > > apex is a wrapper around the REST API provided by dtGatway,
> > > >
> > > > - Tushar.
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <
> > chinmay@apache.org>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I've attached man page for apex in Jira. Here is the link to it:
> > > > > https://issues.apache.org/jira/secure/attachment/12788403/apex.1
> > > > >
> > > > > Please view it by: man -l apex.1
> > > > >
> > > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <
> > > tushar@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Chinmay,
> > > > > >
> > > > > > I am not able to find the attachment, may be attachments are not
> > > > allowed
> > > > > in
> > > > > > the mailing list. Can you attach the man page to the Jira?
> > > > > >
> > > > > > - Tushar.
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <
> > > chinmay@apache.org
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I've attached man page file for the purpose of bigtop
> > integration.
> > > > > > > Please have a look and let me know if there any any feedback
> > about
> > > > it.
> > > > > > >
> > > > > > > You can see the man page content of this file with:
> > > > > > > man -l apex.1
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Chinmay.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> > > > > chinmay@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Thanks. Will follow bigtop recommendation. i.e. remove
> > > > "-incubating".
> > > > > > >>
> > > > > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
> > > > thomas@datatorrent.com
> > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Please follow the recommendation and continue discussion on
> the
> > > > > Bigtop
> > > > > > >>> list.
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> sent from mobile
> > > > > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <
> > chinmay@apache.org
> > > >
> > > > > > wrote:
> > > > > > >>>
> > > > > > >>> > I got a recommendation from bigtop community to drop
> > > > "-incubating"
> > > > > > from
> > > > > > >>> > version.
> > > > > > >>> >
> > > > > > >>> > Here is the mail thread:
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> > > > > > >>> >
> > > > > > >>> > Please let me know if removing "-incubating" is fine.
> > > > > > >>> >
> > > > > > >>> > Thanks,
> > > > > > >>> > Chinmay.
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> > > > > > chinmay@apache.org
> > > > > > >>> >
> > > > > > >>> > wrote:
> > > > > > >>> >
> > > > > > >>> > > I've already started communication there.
> > > > > > >>> > >
> > > > > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> > > > > > >>> thomas@datatorrent.com>
> > > > > > >>> > > wrote:
> > > > > > >>> > >
> > > > > > >>> > >> This is a question you should ask on the Bigtop mailing
> > > list.
> > > > > > >>> > >>
> > > > > > >>> > >> --
> > > > > > >>> > >> sent from mobile
> > > > > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <
> > > prad@apache.org>
> > > > > > >>> wrote:
> > > > > > >>> > >>
> > > > > > >>> > >> > Yes. In case of RPM, the only restriction on version
> is
> > > that
> > > > > it
> > > > > > >>> can
> > > > > > >>> > not
> > > > > > >>> > >> > contain '-'.
> > > > > > >>> > >> > However, as a workaround, one can use 'incubating' as
> a
> > > part
> > > > > of
> > > > > > >>> > >> %{release}.
> > > > > > >>> > >> >
> > > > > > >>> > >> > Version: 3.3.0
> > > > > > >>> > >> > Release: incubating
> > > > > > >>> > >> >
> > > > > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> > > > > > >>> > >> >
> > > > > > >>> > >> > It is the same practice followed for pre-release
> > packages
> > > > e.g.
> > > > > > >>> with
> > > > > > >>> > rcX,
> > > > > > >>> > >> > alphaX, betaX being used in %{release}.
> > > > > > >>> > >> >
> > > > > > >>> > >> > Regards,
> > > > > > >>> > >> > Pradeep A. Dalvi
> > > > > > >>> > >> >
> > > > > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> > > > > > >>> > chinmay@apache.org>
> > > > > > >>> > >> > wrote:
> > > > > > >>> > >> >
> > > > > > >>> > >> > > Hi Everyone,
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > In one of the previous mails in this thread, we
> > decided
> > > > that
> > > > > > >>> version
> > > > > > >>> > >> of
> > > > > > >>> > >> > the
> > > > > > >>> > >> > > package will be "3.3.0-incubating".
> > > > > > >>> > >> > > The problem that I'm facing is in RPM base version
> "-"
> > > is
> > > > > not
> > > > > > >>> > allowed.
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > Can we have package version to "3.3.0"?
> > > > > > >>> > >> > > i.e. the packages created will be as follows:
> > > > > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > > > > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > There is another integration for Zepplin in Bigtop
> > which
> > > > is
> > > > > > >>> > incubating
> > > > > > >>> > >> > and
> > > > > > >>> > >> > > is having base version as "0.5.6".
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > Please let me know your thoughts.
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > Thanks,
> > > > > > >>> > >> > > Chinmay.
> > > > > > >>> > >> > >
> > > > > > >>> > >> > >
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar
> <
> > > > > > >>> > >> chinmay@apache.org>
> > > > > > >>> > >> > > wrote:
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > > Thanks everyone for the feedback.
> > > > > > >>> > >> > > >
> > > > > > >>> > >> > > > Also as Ram and David mentioned, I'll start
> creating
> > > the
> > > > > man
> > > > > > >>> page
> > > > > > >>> > as
> > > > > > >>> > >> > > well.
> > > > > > >>> > >> > > >
> > > > > > >>> > >> > > > Thanks,
> > > > > > >>> > >> > > > Chinmay.
> > > > > > >>> > >> > > >
> > > > > > >>> > >> > > >
> > > > > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> > > > > > >>> > >> thomas@datatorrent.com>
> > > > > > >>> > >> > > > wrote:
> > > > > > >>> > >> > > >
> > > > > > >>> > >> > > >> I think we already agreed that we will retain the
> > > > current
> > > > > > >>> script
> > > > > > >>> > >> name
> > > > > > >>> > >> > in
> > > > > > >>> > >> > > >> this bigtop iteration. There is a ticket for
> > > renaming,
> > > > > add
> > > > > > >>> you
> > > > > > >>> > >> > > suggestions
> > > > > > >>> > >> > > >> there.
> > > > > > >>> > >> > > >>
> > > > > > >>> > >> > > >> --
> > > > > > >>> > >> > > >> sent from mobile
> > > > > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > > > > > >>> > sandesh@datatorrent.com>
> > > > > > >>> > >> > > wrote:
> > > > > > >>> > >> > > >>
> > > > > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start the
> > > > > transition
> > > > > > >>> > >> process by
> > > > > > >>> > >> > > >> making
> > > > > > >>> > >> > > >> > dtcli as the symlink to apex cli.
> > > > > > >>> > >> > > >> >
> > > > > > >>> > >> > > >> > Renaming is a breaking change should be done
> > during
> > > > > major
> > > > > > >>> > version
> > > > > > >>> > >> > > number
> > > > > > >>> > >> > > >> > change.
> > > > > > >>> > >> > > >> >
> > > > > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> > > > > > >>> > david@datatorrent.com
> > > > > > >>> > >> >
> > > > > > >>> > >> > > >> wrote:
> > > > > > >>> > >> > > >> >
> > > > > > >>> > >> > > >> > > Apologies for hitting the "Send" button too
> > > early.
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > What is the scope of the name change?  If we
> > > simply
> > > > > > have
> > > > > > >>> a
> > > > > > >>> > >> symlink
> > > > > > >>> > >> > > >> from
> > > > > > >>> > >> > > >> > > apex to dtcli, then it should have no
> backward
> > > > > > >>> compatibility
> > > > > > >>> > >> > problem
> > > > > > >>> > >> > > >> at
> > > > > > >>> > >> > > >> > > all.
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > But the proper name change would involve:
> > > > > > >>> > >> > > >> > > - Change the name of DTCli class to ApexCli
> > > > > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> > > > > > >>> > >> > > >> > > - Change the initial greeting when running
> the
> > > CLI
> > > > > from
> > > > > > >>> > >> something
> > > > > > >>> > >> > > like
> > > > > > >>> > >> > > >> > "DT
> > > > > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > The above changes may break existing scripts
> > that
> > > > > rely
> > > > > > >>> on the
> > > > > > >>> > >> the
> > > > > > >>> > >> > > >> string
> > > > > > >>> > >> > > >> > > "DT".
> > > > > > >>> > >> > > >> > > But if folks are okay with this potential
> > > problem,
> > > > I
> > > > > am
> > > > > > >>> okay
> > > > > > >>> > >> with
> > > > > > >>> > >> > > the
> > > > > > >>> > >> > > >> > name
> > > > > > >>> > >> > > >> > > change.
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > For the man page, we can start with the
> content
> > > of
> > > > > this
> > > > > > >>> page:
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > Plus the command line options when doing
> dtcli
> > > -h:
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > usage: DTCli
> > > > > > >>> > >> > > >> > >  -e <arg>    Commands are read from the
> > argument
> > > > > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at all
> > time
> > > > > > >>> > >> > > >> > >  -h          Print this help
> > > > > > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos
> > principal
> > > > > > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos
> keytab
> > > > > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> > > > > > >>> > >> > > >> > >  -r          JSON Raw mode
> > > > > > >>> > >> > > >> > >  -v          Verbose mode level 1
> > > > > > >>> > >> > > >> > >  -vv         Verbose mode level 2
> > > > > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> > > > > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > David
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> > > > > > >>> > >> david@datatorrent.com
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > >> > wrote:
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > > What is the scope of the name change?  If
> we
> > > > simply
> > > > > > >>> have a
> > > > > > >>> > >> > symlink
> > > > > > >>> > >> > > >> from
> > > > > > >>> > >> > > >> > > > apex to dtcli, then it should have no
> > backward
> > > > > > >>> > compatibility
> > > > > > >>> > >> > > >> problem at
> > > > > > >>> > >> > > >> > > all.
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > > > But the proper name change would involve:
> > > > > > >>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
> > > > > > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> > > > > > >>> > >> > > >> > > > - Change the initial greeting when running
> > the
> > > > CLI
> > > > > > from
> > > > > > >>> > >> > something
> > > > > > >>> > >> > > >> like
> > > > > > >>> > >> > > >> > > "DT
> > > > > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI
> ...".
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > > > The above changes may break existing
> scripts
> > > that
> > > > > > rely
> > > > > > >>> on
> > > > > > >>> > the
> > > > > > >>> > >> > the
> > > > > > >>> > >> > > >> > string
> > > > > > >>> > >> > > >> > > > "DT".
> > > > > > >>> > >> > > >> > > > But if folks are okay with this potential
> > > > problem,
> > > > > I
> > > > > > am
> > > > > > >>> > okay
> > > > > > >>> > >> > with
> > > > > > >>> > >> > > >> the
> > > > > > >>> > >> > > >> > > name
> > > > > > >>> > >> > > >> > > > change.
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > > > For the man page, we can start with the
> > content
> > > > of
> > > > > > this
> > > > > > >>> > page:
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay
> > > > Kolhatkar
> > > > > <
> > > > > > >>> > >> > > >> chinmay@apache.org
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> > > > wrote:
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > > >> Here is a suggestion I have related to
> name
> > > > > change:
> > > > > > >>> > >> > > >> > > >>
> > > > > > >>> > >> > > >> > > >> 1. While packaging we change the name of
> > > "dtcli"
> > > > > > >>> script to
> > > > > > >>> > >> > > "apex".
> > > > > > >>> > >> > > >> > This
> > > > > > >>> > >> > > >> > > is
> > > > > > >>> > >> > > >> > > >> done so that in future when we change the
> > name
> > > > to
> > > > > > >>> "apex",
> > > > > > >>> > >> users
> > > > > > >>> > >> > > of
> > > > > > >>> > >> > > >> > > bigtop
> > > > > > >>> > >> > > >> > > >> apex don't have to transition much.
> > > > > > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli
> which
> > > > points
> > > > > > to
> > > > > > >>> apex
> > > > > > >>> > >> > > script.
> > > > > > >>> > >> > > >> > This
> > > > > > >>> > >> > > >> > > is
> > > > > > >>> > >> > > >> > > >> for backward compatibility.
> > > > > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first
> > > > > > integration
> > > > > > >>> in
> > > > > > >>> > >> > bigtop.
> > > > > > >>> > >> > > >> This
> > > > > > >>> > >> > > >> > is
> > > > > > >>> > >> > > >> > > >> in
> > > > > > >>> > >> > > >> > > >> the interest of not changing the content
> of
> > > > dtcli
> > > > > or
> > > > > > >>> apex
> > > > > > >>> > >> > script.
> > > > > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all
> these
> > > > naming
> > > > > > >>> related
> > > > > > >>> > >> > > changes
> > > > > > >>> > >> > > >> and
> > > > > > >>> > >> > > >> > > >> update bigtop repository later to remove
> > > > > references
> > > > > > to
> > > > > > >>> > dtcli
> > > > > > >>> > >> > all
> > > > > > >>> > >> > > >> > > together.
> > > > > > >>> > >> > > >> > > >>
> > > > > > >>> > >> > > >> > > >> Please share your thoughts on above
> > approach.
> > > > > > >>> > >> > > >> > > >>
> > > > > > >>> > >> > > >> > > >> Also, please share what could be the man
> > page
> > > > > > content
> > > > > > >>> for
> > > > > > >>> > >> > "apex".
> > > > > > >>> > >> > > >> > > >>
> > > > > > >>> > >> > > >> > > >> Thanks,
> > > > > > >>> > >> > > >> > > >> Chinmay.
> > > > > > >>> > >> > > >> > > >>
> > > > > > >>> > >> > > >> > > >>
> > > > > > >>> > >> > > >> > > >>
> > > > > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay
> > > > > Kolhatkar <
> > > > > > >>> > >> > > >> > chinmay@apache.org>
> > > > > > >>> > >> > > >> > > >> wrote:
> > > > > > >>> > >> > > >> > > >>
> > > > > > >>> > >> > > >> > > >> > Yes.. It'll be build from source tar
> > > > downloaded
> > > > > > >>> from one
> > > > > > >>> > >> of
> > > > > > >>> > >> > the
> > > > > > >>> > >> > > >> > apache
> > > > > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> > > > > > >>> > >> > > >> > > >> >
> > > > > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced
> > > from
> > > > > > dtcli.
> > > > > > >>> > This
> > > > > > >>> > >> > > leaves
> > > > > > >>> > >> > > >> us
> > > > > > >>> > >> > > >> > 3
> > > > > > >>> > >> > > >> > > >> > options:
> > > > > > >>> > >> > > >> > > >> > If we should not have any dt-env.sh,
> dtcli
> > > > would
> > > > > > >>> need a
> > > > > > >>> > >> > change
> > > > > > >>> > >> > > >> while
> > > > > > >>> > >> > > >> > > >> > packaging.
> > > > > > >>> > >> > > >> > > >> >
> > > > > > >>> > >> > > >> > > >> > But if we want to use dtcli as it is,
> then
> > > we
> > > > > > would
> > > > > > >>> need
> > > > > > >>> > >> > > >> dt-env.sh
> > > > > > >>> > >> > > >> > > >> >
> > > > > > >>> > >> > > >> > > >> >
> > > > > > >>> > >> > > >> > > >> >
> > > > > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas
> > > Weise
> > > > <
> > > > > > >>> > >> > > >> > thomas@datatorrent.com
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > > >> > wrote:
> > > > > > >>> > >> > > >> > > >> >
> > > > > > >>> > >> > > >> > > >> >> Looks good overall, though there
> > shouldn't
> > > be
> > > > > any
> > > > > > >>> > >> dt-env.sh
> > > > > > >>> > >> > > >> > > >> >>
> > > > > > >>> > >> > > >> > > >> >> What will the bigtop package be built
> > from,
> > > > the
> > > > > > >>> source
> > > > > > >>> > >> tar?
> > > > > > >>> > >> > > >> > > >> >>
> > > > > > >>> > >> > > >> > > >> >>
> > > > > > >>> > >> > > >> > > >> >>
> > > > > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM,
> Chinmay
> > > > > > Kolhatkar
> > > > > > >>> <
> > > > > > >>> > >> > > >> > > chinmay@apache.org
> > > > > > >>> > >> > > >> > > >> >
> > > > > > >>> > >> > > >> > > >> >> wrote:
> > > > > > >>> > >> > > >> > > >> >>
> > > > > > >>> > >> > > >> > > >> >> > Hi Thomas,
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > First, we're not changing the name
> > > > anywhere.
> > > > > > >>> We'll
> > > > > > >>> > >> follow
> > > > > > >>> > >> > > what
> > > > > > >>> > >> > > >> > > >> >> currently is
> > > > > > >>> > >> > > >> > > >> >> > in source tarball for
> 3.3.0-incubating
> > > > > version.
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory
> > > > > structure
> > > > > > >>> below
> > > > > > >>> > >> which
> > > > > > >>> > >> > > is
> > > > > > >>> > >> > > >> > > inline
> > > > > > >>> > >> > > >> > > >> >> with
> > > > > > >>> > >> > > >> > > >> >> > other existing integrations with
> > bigtop.
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > The need for each file is as follows:
> > > > > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked
> > from
> > > > > > >>> > >> > > >> engine/src/main/scripts/
> > > > > > >>> > >> > > >> > > of
> > > > > > >>> > >> > > >> > > >> >> source
> > > > > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source
> > from
> > > > > dtcli.
> > > > > > >>> This
> > > > > > >>> > >> > > searches
> > > > > > >>> > >> > > >> for
> > > > > > >>> > >> > > >> > > >> hadoop
> > > > > > >>> > >> > > >> > > >> >> > binary path and exports an env
> variable
> > > for
> > > > > > >>> dtcli to
> > > > > > >>> > >> use
> > > > > > >>> > >> > it.
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This
> list
> > > is
> > > > > > >>> extracted
> > > > > > >>> > >> in a
> > > > > > >>> > >> > > >> similar
> > > > > > >>> > >> > > >> > > >> way to
> > > > > > >>> > >> > > >> > > >> >> > how DT Community Edition finds
> > dependency
> > > > > jars.
> > > > > > >>> > >> > > >> > > >> >> > Please note that this is the first
> > > > iteration
> > > > > > >>> list of
> > > > > > >>> > >> jars.
> > > > > > >>> > >> > > I'm
> > > > > > >>> > >> > > >> > > >> trying to
> > > > > > >>> > >> > > >> > > >> >> > narrow this down to only those which
> > are
> > > > > really
> > > > > > >>> > >> required.
> > > > > > >>> > >> > > >> > > >> >> > The test that I'm running to check if
> > > dtcli
> > > > > > runs
> > > > > > >>> fine
> > > > > > >>> > >> with
> > > > > > >>> > >> > > >> given
> > > > > > >>> > >> > > >> > > >> >> dependency
> > > > > > >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for
> > pi
> > > > > demo.
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > Above is the directory structure
> which
> > is
> > > > > > >>> required
> > > > > > >>> > for
> > > > > > >>> > >> CLI
> > > > > > >>> > >> > > to
> > > > > > >>> > >> > > >> > work.
> > > > > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has
> > mentioned
> > > > are
> > > > > > some
> > > > > > >>> > >> > additional
> > > > > > >>> > >> > > >> > files
> > > > > > >>> > >> > > >> > > >> which
> > > > > > >>> > >> > > >> > > >> >> > makes the like of administrator
> easier.
> > > For
> > > > > eg.
> > > > > > >>> > >> /etc/skel
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > Please let me know if this is inline
> > with
> > > > > what
> > > > > > >>> you're
> > > > > > >>> > >> > > >> thinking.
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > Thanks,
> > > > > > >>> > >> > > >> > > >> >> > Chinmay.
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > .
> > > > > > >>> > >> > > >> > > >> >> > |-- bin
> > > > > > >>> > >> > > >> > > >> >> > |   `-- dtcli
> > > > > > >>> > >> > > >> > > >> >> > |-- conf
> > > > > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > > > > > >>> > >> > > >> > > >> >> > `-- lib
> > > > > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > > > > > >>> > >> > > >> > > >> >> >     |--
> > > > > apex-bufferserver-3.3.0-incubating.jar
> > > > > > >>> > >> > > >> > > >> >> >     |--
> > apex-common-3.3.0-incubating.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > > > > > >>> > >> > > >> > > >> >> >     |--
> grizzly-http-servlet-2.1.2.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > > > > > >>> > >> > > >> > > >> >> >     |--
> jetty-http-8.1.10.v20130312.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > > > > >>> > >> > > >> > > >> >> >     |--
> jetty-util-8.1.10.v20130312.jar
> > > > > > >>> > >> > > >> > > >> >> >     |--
> > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > > > > > >>> > >> > > >> > > >> >> >     |--
> validation-api-1.1.0.Final.jar
> > > > > > >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > > > > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM,
> Thomas
> > > > > Weise <
> > > > > > >>> > >> > > >> > > >> thomas@datatorrent.com>
> > > > > > >>> > >> > > >> > > >> >> > wrote:
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >> > > Chinmay,
> > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > >>> > >> > > >> > > >> >> > > Before discussing where to put the
> > > files,
> > > > > > let's
> > > > > > >>> > make
> > > > > > >>> > >> > sure
> > > > > > >>> > >> > > >> they
> > > > > > >>> > >> > > >> > > are
> > > > > > >>> > >> > > >> > > >> >> really
> > > > > > >>> > >> > > >> > > >> >> > > needed for the operation of the
> CLI.
> > As
> > > > for
> > > > > > >>> names,
> > > > > > >>> > >> > > anything
> > > > > > >>> > >> > > >> new
> > > > > > >>> > >> > > >> > > >> needs
> > > > > > >>> > >> > > >> > > >> >> to
> > > > > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and should
> > > > follow
> > > > > > >>> common
> > > > > > >>> > >> > > >> conventions,
> > > > > > >>> > >> > > >> > > >> >> especially
> > > > > > >>> > >> > > >> > > >> >> > > in Bigtop where there are many
> > existing
> > > > > > >>> > integrations
> > > > > > >>> > >> to
> > > > > > >>> > >> > > look
> > > > > > >>> > >> > > >> > at.
> > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > >>> > >> > > >> > > >> >> > > Thanks,
> > > > > > >>> > >> > > >> > > >> >> > > Thomas
> > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM,
> > > Chinmay
> > > > > > >>> Kolhatkar
> > > > > > >>> > <
> > > > > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > > > > > >>> > >> > > >> > > >> >> > > wrote:
> > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep and
> > > > Aniruddha.
> > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to
> > change
> > > > the
> > > > > > >>> dtcli
> > > > > > >>> > >> > > >> considering it
> > > > > > >>> > >> > > >> > > >> works
> > > > > > >>> > >> > > >> > > >> >> > with
> > > > > > >>> > >> > > >> > > >> >> > > DT
> > > > > > >>> > >> > > >> > > >> >> > > > Community edition. We can keep
> the
> > > > > > directory
> > > > > > >>> > >> structure
> > > > > > >>> > >> > > >> > similar
> > > > > > >>> > >> > > >> > > to
> > > > > > >>> > >> > > >> > > >> >> that.
> > > > > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which
> > > contains
> > > > > > >>> > information
> > > > > > >>> > >> > > >> required
> > > > > > >>> > >> > > >> > for
> > > > > > >>> > >> > > >> > > >> >> dtcli
> > > > > > >>> > >> > > >> > > >> >> > to
> > > > > > >>> > >> > > >> > > >> >> > > > launch.
> > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop
> > community
> > > > > that
> > > > > > >>> > whether
> > > > > > >>> > >> > they
> > > > > > >>> > >> > > >> > > >> facilitate
> > > > > > >>> > >> > > >> > > >> >> the
> > > > > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs before
> > user
> > > > is
> > > > > > >>> created.
> > > > > > >>> > >> In
> > > > > > >>> > >> > > >> either
> > > > > > >>> > >> > > >> > > case,
> > > > > > >>> > >> > > >> > > >> >> > current
> > > > > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in
> home
> > > > > > directory.
> > > > > > >>> > Also
> > > > > > >>> > >> > > before
> > > > > > >>> > >> > > >> > > putting
> > > > > > >>> > >> > > >> > > >> >> > > anything
> > > > > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define
> what
> > > are
> > > > > the
> > > > > > >>> > default
> > > > > > >>> > >> > > >> contents
> > > > > > >>> > >> > > >> > > that
> > > > > > >>> > >> > > >> > > >> >> > should
> > > > > > >>> > >> > > >> > > >> >> > > go
> > > > > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no
> > > > > defaults,
> > > > > > >>> > >> probably we
> > > > > > >>> > >> > > >> should
> > > > > > >>> > >> > > >> > > not
> > > > > > >>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks
> > > nice.
> > > > > > This
> > > > > > >>> way
> > > > > > >>> > >> > > contents
> > > > > > >>> > >> > > >> of
> > > > > > >>> > >> > > >> > > >> >> dt-env.sh
> > > > > > >>> > >> > > >> > > >> >> > > are
> > > > > > >>> > >> > > >> > > >> >> > > > present as env variables. But I
> > see a
> > > > > catch
> > > > > > >>> > there.
> > > > > > >>> > >> > > Adding
> > > > > > >>> > >> > > >> > > >> dt-env.sh
> > > > > > >>> > >> > > >> > > >> >> to
> > > > > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the
> > > > > variables
> > > > > > >>> > >> available
> > > > > > >>> > >> > at
> > > > > > >>> > >> > > >> > > runtime
> > > > > > >>> > >> > > >> > > >> all
> > > > > > >>> > >> > > >> > > >> >> > the
> > > > > > >>> > >> > > >> > > >> >> > > > time. I feel a little skeptical
> > about
> > > > > that.
> > > > > > >>> > Maybe a
> > > > > > >>> > >> > > >> possible
> > > > > > >>> > >> > > >> > > >> >> collision
> > > > > > >>> > >> > > >> > > >> >> > > can
> > > > > > >>> > >> > > >> > > >> >> > > > occur with other application
> vars.
> > > > > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does
> source
> > > > > > >>> > >> > "../conf/dt-env.sh"
> > > > > > >>> > >> > > >> and
> > > > > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those
> > variables
> > > > are
> > > > > > >>> anyway
> > > > > > >>> > >> > > >> available.
> > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR
> effort
> > is
> > > > > > >>> happening
> > > > > > >>> > at
> > > > > > >>> > >> > > Bigtop
> > > > > > >>> > >> > > >> > > >> >> > > Jira/repository.
> > > > > > >>> > >> > > >> > > >> >> > > > Packaging related code usually
> goes
> > > > > there.
> > > > > > >>> > (That's
> > > > > > >>> > >> > what
> > > > > > >>> > >> > > >> all
> > > > > > >>> > >> > > >> > the
> > > > > > >>> > >> > > >> > > >> >> > > components
> > > > > > >>> > >> > > >> > > >> >> > > > in bigtop does).
> > > > > > >>> > >> > > >> > > >> >> > > > Having said that, once a PR is
> > > created,
> > > > > > I'll
> > > > > > >>> be
> > > > > > >>> > >> > sharing
> > > > > > >>> > >> > > >> the
> > > > > > >>> > >> > > >> > > link
> > > > > > >>> > >> > > >> > > >> of
> > > > > > >>> > >> > > >> > > >> >> PR
> > > > > > >>> > >> > > >> > > >> >> > > > here, so that apex community as
> > well
> > > > can
> > > > > > >>> review
> > > > > > >>> > it.
> > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > >>> > >> > > >> > > >> >> > > > Thanks,
> > > > > > >>> > >> > > >> > > >> >> > > > Chinmay.
> > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM,
> > > > > Aniruddha
> > > > > > >>> > >> Thombare <
> > > > > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com>
> wrote:
> > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> > > > > > >>> > >> > > >> > > >> >> > > > > We may need to iron out details
> > > about
> > > > > > exact
> > > > > > >>> > paths
> > > > > > >>> > >> > etc.
> > > > > > >>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR
> > > > > comments.
> > > > > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm
> > Pradeep
> > > > A.
> > > > > > >>> Dalvi <
> > > > > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > > > > > >>> > >> > > >> > > >> >> > > > > wrote:
> > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51
> > PM,
> > > > > > Chinmay
> > > > > > >>> > >> > Kolhatkar <
> > > > > > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > > > > > >>> > >> > > >> > > >> >> > > > > > wrote:
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar
> > > present
> > > > in
> > > > > > DT
> > > > > > >>> > >> community
> > > > > > >>> > >> > > >> > edition
> > > > > > >>> > >> > > >> > > >> will
> > > > > > >>> > >> > > >> > > >> >> be
> > > > > > >>> > >> > > >> > > >> >> > > > > included
> > > > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I
> > > found
> > > > > from
> > > > > > >>> > common
> > > > > > >>> > >> > > >> > dependencies
> > > > > > >>> > >> > > >> > > >> >> between
> > > > > > >>> > >> > > >> > > >> >> > > of
> > > > > > >>> > >> > > >> > > >> >> > > > > apex
> > > > > > >>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> jackson-mapper-asl-1.9.2.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> async-http-client-1.7.20.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > validation-api-1.1.0.Final.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> jersey-apache-client4-1.9.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> hadoop-common-2.2.0-tests.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running
> > some
> > > > > tests
> > > > > > do
> > > > > > >>> > check
> > > > > > >>> > >> > that
> > > > > > >>> > >> > > >> > dtcli
> > > > > > >>> > >> > > >> > > >> works
> > > > > > >>> > >> > > >> > > >> >> > > > properly
> > > > > > >>> > >> > > >> > > >> >> > > > > > for
> > > > > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of
> apps
> > > with
> > > > > > only
> > > > > > >>> > these
> > > > > > >>> > >> > > >> libraries
> > > > > > >>> > >> > > >> > > >> >> present in
> > > > > > >>> > >> > > >> > > >> >> > > > > > isolation
> > > > > > >>> > >> > > >> > > >> >> > > > > > > without dependency on local
> > m2.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there are
> > > > unwanted
> > > > > > >>> jars
> > > > > > >>> > >> which
> > > > > > >>> > >> > are
> > > > > > >>> > >> > > >> used
> > > > > > >>> > >> > > >> > > for
> > > > > > >>> > >> > > >> > > >> >> > compile
> > > > > > >>> > >> > > >> > > >> >> > > > > time
> > > > > > >>> > >> > > >> > > >> >> > > > > > > dependency and not runtime
> in
> > > > above
> > > > > > >>> list
> > > > > > >>> > >> which I
> > > > > > >>> > >> > > can
> > > > > > >>> > >> > > >> > drop
> > > > > > >>> > >> > > >> > > >> to
> > > > > > >>> > >> > > >> > > >> >> keep
> > > > > > >>> > >> > > >> > > >> >> > > > > package
> > > > > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not
> sure
> > if
> > > > all
> > > > > > of
> > > > > > >>> the
> > > > > > >>> > >> > > >> dt-env.sh is
> > > > > > >>> > >> > > >> > > >> >> required.
> > > > > > >>> > >> > > >> > > >> >> > > > Only
> > > > > > >>> > >> > > >> > > >> >> > > > > > > required values I see in it
> > are
> > > > > > >>> classpath.
> > > > > > >>> > >> and
> > > > > > >>> > >> > > >> dtcli is
> > > > > > >>> > >> > > >> > > >> >> capable
> > > > > > >>> > >> > > >> > > >> >> > of
> > > > > > >>> > >> > > >> > > >> >> > > > > > building
> > > > > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes,
> we'll
> > > have
> > > > > to
> > > > > > >>> see
> > > > > > >>> > how
> > > > > > >>> > >> > dtcli
> > > > > > >>> > >> > > >> can
> > > > > > >>> > >> > > >> > > find
> > > > > > >>> > >> > > >> > > >> >> the
> > > > > > >>> > >> > > >> > > >> >> > jar
> > > > > > >>> > >> > > >> > > >> >> > > > > path,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf
> > > > files.
> > > > > If
> > > > > > >>> we
> > > > > > >>> > >> need a
> > > > > > >>> > >> > > >> change
> > > > > > >>> > >> > > >> > in
> > > > > > >>> > >> > > >> > > >> >> dtcli,
> > > > > > >>> > >> > > >> > > >> >> > > then
> > > > > > >>> > >> > > >> > > >> >> > > > > > > community's opinion is
> > required
> > > > for
> > > > > > >>> whether
> > > > > > >>> > >> > dtcli
> > > > > > >>> > >> > > >> > should
> > > > > > >>> > >> > > >> > > >> >> change
> > > > > > >>> > >> > > >> > > >> >> > in
> > > > > > >>> > >> > > >> > > >> >> > > > our
> > > > > > >>> > >> > > >> > > >> >> > > > > > repo
> > > > > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with
> > required
> > > > > > changes
> > > > > > >>> > >> exist in
> > > > > > >>> > >> > > >> bigtop
> > > > > > >>> > >> > > >> > > >> repo
> > > > > > >>> > >> > > >> > > >> >> > until
> > > > > > >>> > >> > > >> > > >> >> > > we
> > > > > > >>> > >> > > >> > > >> >> > > > > > make
> > > > > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > Can we have these file paths
> > set
> > > > > using
> > > > > > >>> > >> environment
> > > > > > >>> > >> > > >> > > variables
> > > > > > >>> > >> > > >> > > >> >> with
> > > > > > >>> > >> > > >> > > >> >> > > some
> > > > > > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> > > > > > >>> > >> > > >> > > >> >> > > > > > Then we can set such params
> in
> > > > > > dt-env.sh.
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is
> > this
> > > > file
> > > > > > >>> > different
> > > > > > >>> > >> > from
> > > > > > >>> > >> > > >> > > >> >> dt-site.xml?
> > > > > > >>> > >> > > >> > > >> >> > Not
> > > > > > >>> > >> > > >> > > >> >> > > > > sure
> > > > > > >>> > >> > > >> > > >> >> > > > > > if
> > > > > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would
> > help.
> > > > > > Files &
> > > > > > >>> > Dirs
> > > > > > >>> > >> > from
> > > > > > >>> > >> > > >> > > /etc/skel
> > > > > > >>> > >> > > >> > > >> >> are
> > > > > > >>> > >> > > >> > > >> >> > > > copied
> > > > > > >>> > >> > > >> > > >> >> > > > > to
> > > > > > >>> > >> > > >> > > >> >> > > > > > > home of new user when
> useradd
> > > > > program
> > > > > > >>> is
> > > > > > >>> > >> called.
> > > > > > >>> > >> > > But
> > > > > > >>> > >> > > >> > for
> > > > > > >>> > >> > > >> > > >> >> existing
> > > > > > >>> > >> > > >> > > >> >> > > > users
> > > > > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any use.
> > > > Moreover,
> > > > > > >>> dtcli
> > > > > > >>> > >> > creates
> > > > > > >>> > >> > > >> > ~/.dt/
> > > > > > >>> > >> > > >> > > on
> > > > > > >>> > >> > > >> > > >> >> the
> > > > > > >>> > >> > > >> > > >> >> > fly
> > > > > > >>> > >> > > >> > > >> >> > > > if
> > > > > > >>> > >> > > >> > > >> >> > > > > > not
> > > > > > >>> > >> > > >> > > >> >> > > > > > > encountered for the first
> > time.
> > > > > Again
> > > > > > >>> > >> correct me
> > > > > > >>> > >> > > if
> > > > > > >>> > >> > > >> I'm
> > > > > > >>> > >> > > >> > > >> wrong.
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right. However
> > this
> > > > > would
> > > > > > >>> > >> probably be
> > > > > > >>> > >> > > >> > > necessary
> > > > > > >>> > >> > > >> > > >> >> step
> > > > > > >>> > >> > > >> > > >> >> > for
> > > > > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get
> > > > > installed
> > > > > > >>> > during
> > > > > > >>> > >> OS
> > > > > > >>> > >> > > >> install
> > > > > > >>> > >> > > >> > > and
> > > > > > >>> > >> > > >> > > >> >> > before
> > > > > > >>> > >> > > >> > > >> >> > > > user
> > > > > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc &
> > > bash_profile
> > > > > ->
> > > > > > >>> I'm
> > > > > > >>> > not
> > > > > > >>> > >> > sure
> > > > > > >>> > >> > > >> its
> > > > > > >>> > >> > > >> > the
> > > > > > >>> > >> > > >> > > >> best
> > > > > > >>> > >> > > >> > > >> >> > > idea.
> > > > > > >>> > >> > > >> > > >> >> > > > > > We'll
> > > > > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in
> location
> > > > which
> > > > > is
> > > > > > >>> by
> > > > > > >>> > >> default
> > > > > > >>> > >> > > >> > present
> > > > > > >>> > >> > > >> > > in
> > > > > > >>> > >> > > >> > > >> >> path
> > > > > > >>> > >> > > >> > > >> >> > > i.e.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > For other env variables
> > specific
> > > to
> > > > > > apex,
> > > > > > >>> > I'll
> > > > > > >>> > >> > > prefer
> > > > > > >>> > >> > > >> to
> > > > > > >>> > >> > > >> > > use
> > > > > > >>> > >> > > >> > > >> >> > > dt-env.sh
> > > > > > >>> > >> > > >> > > >> >> > > > > and
> > > > > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather
> > than
> > > > > > changing
> > > > > > >>> > >> bashrc
> > > > > > >>> > >> > > >> etc...
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
> > > > > > >>> /etc/profile.d/
> > > > > > >>> > >> shall
> > > > > > >>> > >> > > be
> > > > > > >>> > >> > > >> the
> > > > > > >>> > >> > > >> > > >> >> cleaner
> > > > > > >>> > >> > > >> > > >> >> > way
> > > > > > >>> > >> > > >> > > >> >> > > > to
> > > > > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > Thanks,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at
> 1:07
> > > PM,
> > > > > > >>> Aniruddha
> > > > > > >>> > >> > > Thombare
> > > > > > >>> > >> > > >> <
> > > > > > >>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com>
> > > > wrote:
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > We need to consider
> > > following:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > System wide default
> config
> > > > files
> > > > > > can
> > > > > > >>> be
> > > > > > >>> > >> > located
> > > > > > >>> > >> > > at
> > > > > > >>> > >> > > >> > > >> following
> > > > > > >>> > >> > > >> > > >> >> > > > > locations:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we
> > may
> > > > have
> > > > > > to
> > > > > > >>> > change
> > > > > > >>> > >> > > dtcli
> > > > > > >>> > >> > > >> > > >> behaviour
> > > > > > >>> > >> > > >> > > >> >> on
> > > > > > >>> > >> > > >> > > >> >> > > how
> > > > > > >>> > >> > > >> > > >> >> > > > it
> > > > > > >>> > >> > > >> > > >> >> > > > > > > finds
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we
> > may
> > > > > have
> > > > > > to
> > > > > > >>> > >> change
> > > > > > >>> > >> > > dtcli
> > > > > > >>> > >> > > >> > > >> >> behaviour on
> > > > > > >>> > >> > > >> > > >> >> > > how
> > > > > > >>> > >> > > >> > > >> >> > > > > it
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> /etc/skel/.dt/dt-sited.xml
> > > and
> > > > > > other
> > > > > > >>> > files
> > > > > > >>> > >> > (for
> > > > > > >>> > >> > > >> new
> > > > > > >>> > >> > > >> > > users
> > > > > > >>> > >> > > >> > > >> >> that
> > > > > > >>> > >> > > >> > > >> >> > > will
> > > > > > >>> > >> > > >> > > >> >> > > > > be
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > created in system in
> > future.)
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > We may also have to
> modify
> > > > > bashrc /
> > > > > > >>> > >> > bashprofile
> > > > > > >>> > >> > > >> for
> > > > > > >>> > >> > > >> > > >> >> population
> > > > > > >>> > >> > > >> > > >> >> > > any
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > variables that are
> > required.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your
> > > > > comments /
> > > > > > >>> > >> > suggestion.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at
> > 1:16
> > > > AM,
> > > > > > >>> Thomas
> > > > > > >>> > >> Weise
> > > > > > >>> > >> > <
> > > > > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > -->
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at
> > 9:05
> > > > AM,
> > > > > > >>> Chinmay
> > > > > > >>> > >> > > >> Kolhatkar <
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with
> > packaging
> > > > > > effort
> > > > > > >>> > >> (rpm+deb)
> > > > > > >>> > >> > > of
> > > > > > >>> > >> > > >> > apex,
> > > > > > >>> > >> > > >> > > >> here
> > > > > > >>> > >> > > >> > > >> >> > are
> > > > > > >>> > >> > > >> > > >> >> > > > some
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > about package
> structure
> > > > etc..
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on
> > > > bbigtop
> > > > > > >>> mailing
> > > > > > >>> > >> > list, I
> > > > > > >>> > >> > > >> have
> > > > > > >>> > >> > > >> > > >> some
> > > > > > >>> > >> > > >> > > >> >> > > question
> > > > > > >>> > >> > > >> > > >> >> > > > > for
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > apex
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory
> > > > structure
> > > > > of
> > > > > > >>> apex
> > > > > > >>> > >> > package
> > > > > > >>> > >> > > >> for
> > > > > > >>> > >> > > >> > > both
> > > > > > >>> > >> > > >> > > >> >> deb &
> > > > > > >>> > >> > > >> > > >> >> > > > rpm:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> /usr/lib/apex/bin/dtcli
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > /usr/lib/apex/lib/<other
> > > > > > >>> dependent
> > > > > > >>> > >> jars>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> > > > > > >>> > /usr/lib/apex/bin/dtcli
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > /usr/share/doc/apex/CHANGELOG
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > /usr/share/doc/apex/NOTICE
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
> > > > 3.3.0-incubating.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent
> jars>
> > =
> > > > All
> > > > > > the
> > > > > > >>> 3rd
> > > > > > >>> > >> party
> > > > > > >>> > >> > > >> jars
> > > > > > >>> > >> > > >> > > which
> > > > > > >>> > >> > > >> > > >> >> are
> > > > > > >>> > >> > > >> > > >> >> > > > > required
> > > > > > >>> > >> > > >> > > >> >> > > > > > > for
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > apex
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the
> > > > > > dependencies
> > > > > > >>> are
> > > > > > >>> > >> > > packaged
> > > > > > >>> > >> > > >> as
> > > > > > >>> > >> > > >> > > >> part of
> > > > > > >>> > >> > > >> > > >> >> > > > rpm/deb
> > > > > > >>> > >> > > >> > > >> >> > > > > by
> > > > > > >>> > >> > > >> > > >> >> > > > > > > any
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what
> > those
> > > > jars
> > > > > > >>> are in
> > > > > > >>> > >> > Bigtop
> > > > > > >>> > >> > > >> > > context?
> > > > > > >>> > >> > > >> > > >> >> Same
> > > > > > >>> > >> > > >> > > >> >> > as
> > > > > > >>> > >> > > >> > > >> >> > > > > > shipped
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > with
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > DT community addition
> > under
> > > > > lib/
> > > > > > ?
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> =
> > > > Licenses
> > > > > of
> > > > > > >>> > >> > > corresponding
> > > > > > >>> > >> > > >> of
> > > > > > >>> > >> > > >> > > 3rd
> > > > > > >>> > >> > > >> > > >> >> party
> > > > > > >>> > >> > > >> > > >> >> > > jars
> > > > > > >>> > >> > > >> > > >> >> > > > > > which
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > needs
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to included while
> > > > packaging.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions related to
> > > this:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the
> > cli
> > > > of
> > > > > > >>> apex as
> > > > > > >>> > >> > "apex"
> > > > > > >>> > >> > > >> > instead
> > > > > > >>> > >> > > >> > > >> of
> > > > > > >>> > >> > > >> > > >> >> > > "dtcli"
> > > > > > >>> > >> > > >> > > >> >> > > > in
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > I think we should keep
> > the
> > > > name
> > > > > > >>> until
> > > > > > >>> > we
> > > > > > >>> > >> are
> > > > > > >>> > >> > > >> able
> > > > > > >>> > >> > > >> > to
> > > > > > >>> > >> > > >> > > >> >> change
> > > > > > >>> > >> > > >> > > >> >> > it
> > > > > > >>> > >> > > >> > > >> >> > > in
> > > > > > >>> > >> > > >> > > >> >> > > > > > Apex.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get
> > > > > confusing.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all
> > > softwares
> > > > > in
> > > > > > >>> bigtop
> > > > > > >>> > >> have
> > > > > > >>> > >> > > man
> > > > > > >>> > >> > > >> > page
> > > > > > >>> > >> > > >> > > >> for
> > > > > > >>> > >> > > >> > > >> >> > their
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we should
> have
> > it
> > > > too
> > > > > > for
> > > > > > >>> > >> dtcli.
> > > > > > >>> > >> > Is
> > > > > > >>> > >> > > >> there
> > > > > > >>> > >> > > >> > > any
> > > > > > >>> > >> > > >> > > >> >> > > > > documentation
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > which I
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man
> > page?
> > > or
> > > > > can
> > > > > > >>> I use
> > > > > > >>> > >> > output
> > > > > > >>> > >> > > >> of
> > > > > > >>> > >> > > >> > > >> "dtcli
> > > > > > >>> > >> > > >> > > >> >> > > --help"?
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call
> > > > version
> > > > > > of
> > > > > > >>> apex
> > > > > > >>> > >> in
> > > > > > >>> > >> > > >> Bigtop
> > > > > > >>> > >> > > >> > as
> > > > > > >>> > >> > > >> > > >> >> 3.3.0 OR
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex
> > > > package
> > > > > of
> > > > > > >>> > bigtop
> > > > > > >>> > >> to
> > > > > > >>> > >> > > >> depend
> > > > > > >>> > >> > > >> > on
> > > > > > >>> > >> > > >> > > >> >> 2.7.1
> > > > > > >>> > >> > > >> > > >> >> > > > > version
> > > > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any
> > > problems
> > > > > > that
> > > > > > >>> we
> > > > > > >>> > see
> > > > > > >>> > >> > with
> > > > > > >>> > >> > > >> this
> > > > > > >>> > >> > > >> > > >> >> > dependency?
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Should work without
> > > changed.
> > > > I
> > > > > > >>> thought
> > > > > > >>> > we
> > > > > > >>> > >> > > >> certified
> > > > > > >>> > >> > > >> > > >> >> against
> > > > > > >>> > >> > > >> > > >> >> > > > 2.7.0?
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the
> > > apache
> > > > > > mirror
> > > > > > >>> > from
> > > > > > >>> > >> > which
> > > > > > >>> > >> > > >> > bigtop
> > > > > > >>> > >> > > >> > > >> will
> > > > > > >>> > >> > > >> > > >> >> > pick
> > > > > > >>> > >> > > >> > > >> >> > > > the
> > > > > > >>> > >> > > >> > > >> >> > > > > > > apex
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > source for
> compilation
> > > and
> > > > > > >>> packaging.
> > > > > > >>> > >> > Please
> > > > > > >>> > >> > > >> > > correct
> > > > > > >>> > >> > > >> > > >> if
> > > > > > >>> > >> > > >> > > >> >> > > > > incorrect:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > >>> > >> > >
> > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
> > > > apache.osuosl.org
> > > > > is
> > > > > > >>> the
> > > > > > >>> > >> mirror
> > > > > > >>> > >> > > >> used
> > > > > > >>> > >> > > >> > by
> > > > > > >>> > >> > > >> > > >> all
> > > > > > >>> > >> > > >> > > >> >> the
> > > > > > >>> > >> > > >> > > >> >> > > > > > softwares
> > > > > > >>> > >> > > >> > > >> >> > > > > > > in
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share your
> > > opinion.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016
> at
> > > > 11:19
> > > > > > AM,
> > > > > > >>> > >> Chinmay
> > > > > > >>> > >> > > >> > Kolhatkar
> > > > > > >>> > >> > > >> > > <
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > chinmay@datatorrent.com>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to
> > work
> > > on
> > > > > the
> > > > > > >>> > >> packaging.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already
> started
> > > > > > >>> discussion on
> > > > > > >>> > >> > bigtop
> > > > > > >>> > >> > > >> dev
> > > > > > >>> > >> > > >> > > >> mailing
> > > > > > >>> > >> > > >> > > >> >> > list
> > > > > > >>> > >> > > >> > > >> >> > > > for
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also
> > > > created a
> > > > > > >>> Jira
> > > > > > >>> > for
> > > > > > >>> > >> the
> > > > > > >>> > >> > > >> same.
> > > > > > >>> > >> > > >> > > For
> > > > > > >>> > >> > > >> > > >> >> this
> > > > > > >>> > >> > > >> > > >> >> > > > > > communities
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is
> > the
> > > > > bigtop
> > > > > > >>> Jira:
> > > > > > >>> > >> > > >> > BIGTOP-2313.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016
> > at
> > > > 1:13
> > > > > > AM,
> > > > > > >>> > >> Thomas
> > > > > > >>> > >> > > >> Weise <
> > > > > > >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see
> anything
> > > > under
> > > > > > >>> > >> > prerequisites
> > > > > > >>> > >> > > >> that
> > > > > > >>> > >> > > >> > > >> would
> > > > > > >>> > >> > > >> > > >> >> be a
> > > > > > >>> > >> > > >> > > >> >> > > > > > problem.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > We
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF
> licencing
> > > > > > >>> compatibility
> > > > > > >>> > >> as
> > > > > > >>> > >> > > part
> > > > > > >>> > >> > > >> of
> > > > > > >>> > >> > > >> > > >> >> becoming
> > > > > > >>> > >> > > >> > > >> >> > an
> > > > > > >>> > >> > > >> > > >> >> > > > > > > incubator
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on
> the
> > > > > > packaging
> > > > > > >>> > during
> > > > > > >>> > >> > the
> > > > > > >>> > >> > > >> next
> > > > > > >>> > >> > > >> > > >> weeks.
> > > > > > >>> > >> > > >> > > >> >> > Since
> > > > > > >>> > >> > > >> > > >> >> > > > the
> > > > > > >>> > >> > > >> > > >> >> > > > > > > work
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > will
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop,
> > > related
> > > > > > >>> > discussions
> > > > > > >>> > >> and
> > > > > > >>> > >> > > >> JIRAs
> > > > > > >>> > >> > > >> > > >> should
> > > > > > >>> > >> > > >> > > >> >> > also
> > > > > > >>> > >> > > >> > > >> >> > > be
> > > > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to
> > have
> > > > the
> > > > > > >>> > packaging
> > > > > > >>> > >> in
> > > > > > >>> > >> > > >> place
> > > > > > >>> > >> > > >> > by
> > > > > > >>> > >> > > >> > > >> end
> > > > > > >>> > >> > > >> > > >> >> Feb.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4,
> 2016
> > at
> > > > > 10:14
> > > > > > >>> AM,
> > > > > > >>> > >> > Chinmay
> > > > > > >>> > >> > > >> > > >> Kolhatkar <
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > chinmay@datatorrent.com
> > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a
> > > work
> > > > on
> > > > > > >>> adding
> > > > > > >>> > >> > Apache
> > > > > > >>> > >> > > >> Apex
> > > > > > >>> > >> > > >> > > as a
> > > > > > >>> > >> > > >> > > >> >> > > component
> > > > > > >>> > >> > > >> > > >> >> > > > > to
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > Apache
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the
> > > > packaging
> > > > > > >>> system
> > > > > > >>> > for
> > > > > > >>> > >> > the
> > > > > > >>> > >> > > >> > Apache
> > > > > > >>> > >> > > >> > > >> big
> > > > > > >>> > >> > > >> > > >> >> data
> > > > > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros
> use
> > > it,
> > > > > > most
> > > > > > >>> > >> recently
> > > > > > >>> > >> > > EMR.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the
> > tracking
> > > > > Jira
> > > > > > >>> task
> > > > > > >>> > in
> > > > > > >>> > >> > > >> APEXCORE
> > > > > > >>> > >> > > >> > for
> > > > > > >>> > >> > > >> > > >> the
> > > > > > >>> > >> > > >> > > >> >> > same:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >>
> > > > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of
> > > > > execution
> > > > > > >>> is as
> > > > > > >>> > >> > > follows:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
> > > > > > >>> prerequisites*
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop
> has
> > > some
> > > > > > hard
> > > > > > >>> and
> > > > > > >>> > >> soft
> > > > > > >>> > >> > > >> > > expectation
> > > > > > >>> > >> > > >> > > >> >> for
> > > > > > >>> > >> > > >> > > >> >> > new
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > components
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into
> > > > Bigtop.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list
> > of
> > > > it:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >>
> > > > > > >>> > >> > > >> > > >>
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> >
> > > > > > >>> > >> > > >>
> > > > > > >>> > >> > >
> > > > > > >>> > >> >
> > > > > > >>> > >>
> > > > > > >>> >
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them
> seems
> > > to
> > > > be
> > > > > > >>> > standard
> > > > > > >>> > >> ASF
> > > > > > >>> > >> > > >> based
> > > > > > >>> > >> > > >> > > >> >> > > requirements,
> > > > > > >>> > >> > > >> > > >> >> > > > > but
> > > > > > >>> > >> > > >> > > >> >> > > > > > > few
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > need
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software
> > projects
> > > > are
> > > > > > >>> > expected
> > > > > > >>> > >> to
> > > > > > >>> > >> > be
> > > > > > >>> > >> > > >> > > Licensed
> > > > > > >>> > >> > > >> > > >> >> under
> > > > > > >>> > >> > > >> > > >> >> > > > > Apache
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > License,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and
> > > their
> > > > > > >>> > >> dependencies
> > > > > > >>> > >> > are
> > > > > > >>> > >> > > >> > > expected
> > > > > > >>> > >> > > >> > > >> >> to be
> > > > > > >>> > >> > > >> > > >> >> > > > > > > compatible
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > with
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is
> > under
> > > > ASL
> > > > > > 2.0
> > > > > > >>> but
> > > > > > >>> > >> need
> > > > > > >>> > >> > to
> > > > > > >>> > >> > > >> > check
> > > > > > >>> > >> > > >> > > if
> > > > > > >>> > >> > > >> > > >> >> > > > > dependencies
> > > > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with
> > ASL
> > > > 2.0.
> > > > > > >>> This I
> > > > > > >>> > >> guess
> > > > > > >>> > >> > > >> would
> > > > > > >>> > >> > > >> > > be a
> > > > > > >>> > >> > > >> > > >> >> > > > > verification
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > check.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software
> > projects
> > > > are
> > > > > > >>> > expected
> > > > > > >>> > >> to
> > > > > > >>> > >> > be
> > > > > > >>> > >> > > >> > > >> compatible
> > > > > > >>> > >> > > >> > > >> >> > with
> > > > > > >>> > >> > > >> > > >> >> > > > all
> > > > > > >>> > >> > > >> > > >> >> > > > > of
> > > > > > >>> > >> > > >> > > >> >> > > > > > > the
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported
> > platforms
> > > > that
> > > > > > >>> Bigtop
> > > > > > >>> > >> > > >> distribution
> > > > > > >>> > >> > > >> > > is
> > > > > > >>> > >> > > >> > > >> >> > > targeting
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > > > > verifying
> > > > > > >>> > whether
> > > > > > >>> > >> > our
> > > > > > >>> > >> > > >> > > software
> > > > > > >>> > >> > > >> > > >> >> runs
> > > > > > >>> > >> > > >> > > >> >> > > fine
> > > > > > >>> > >> > > >> > > >> >> > > > in
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7
> fedora-20
> > > > > > >>> ubuntu-14.04
> > > > > > >>> > >> > > debian-8
> > > > > > >>> > >> > > >> > > >> >> > opensuse-13.2.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke
> > tests
> > > > that
> > > > > > >>> should
> > > > > > >>> > be
> > > > > > >>> > >> > > added
> > > > > > >>> > >> > > >> for
> > > > > > >>> > >> > > >> > > >> >> > deployment.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying
> the
> > > > test
> > > > > > >>> > artifacts
> > > > > > >>> > >> > which
> > > > > > >>> > >> > > >> goes
> > > > > > >>> > >> > > >> > > >> beyond
> > > > > > >>> > >> > > >> > > >> >> > > smoke
> > > > > > >>> > >> > > >> > > >> >> > > > > test
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are
> > > > > basically
> > > > > > >>> the
> > > > > > >>> > >> > > integration
> > > > > > >>> > >> > > >> > > tests
> > > > > > >>> > >> > > >> > > >> for
> > > > > > >>> > >> > > >> > > >> >> > > > > > verification
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > after
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This
> > is
> > > a
> > > > > soft
> > > > > > >>> > >> > requirement,
> > > > > > >>> > >> > > >> but
> > > > > > >>> > >> > > >> > > aim
> > > > > > >>> > >> > > >> > > >> is
> > > > > > >>> > >> > > >> > > >> >> to
> > > > > > >>> > >> > > >> > > >> >> > > > > achieve
> > > > > > >>> > >> > > >> > > >> >> > > > > > > this
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > as
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have
> > > > > > >>> explanation why
> > > > > > >>> > >> not
> > > > > > >>> > >> > to
> > > > > > >>> > >> > > >> > > include.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any
> > > from
> > > > > the
> > > > > > >>> link
> > > > > > >>> > >> which
> > > > > > >>> > >> > > >> > > explicitly
> > > > > > >>> > >> > > >> > > >> >> needs
> > > > > > >>> > >> > > >> > > >> >> > to
> > > > > > >>> > >> > > >> > > >> >> > > > be
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > checked
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4,
> > please
> > > > let
> > > > > > us
> > > > > > >>> > know.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding
> > Apex
> > > > as
> > > > > > >>> > component
> > > > > > >>> > >> to
> > > > > > >>> > >> > > >> Bigtop*
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the
> > mail
> > > > > > >>> archive of
> > > > > > >>> > >> > Bigtop,
> > > > > > >>> > >> > > >> it
> > > > > > >>> > >> > > >> > was
> > > > > > >>> > >> > > >> > > >> >> learnt
> > > > > > >>> > >> > > >> > > >> >> > > that
> > > > > > >>> > >> > > >> > > >> >> > > > > the
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community want
> to
> > > see
> > > > > the
> > > > > > >>> > >> addition of
> > > > > > >>> > >> > > new
> > > > > > >>> > >> > > >> > > >> >> components
> > > > > > >>> > >> > > >> > > >> >> > in
> > > > > > >>> > >> > > >> > > >> >> > > > > > phases.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > Here
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > are
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > > > > creating
> > > > > > of
> > > > > > >>> > >> package
> > > > > > >>> > >> > > i.e.
> > > > > > >>> > >> > > >> > rpm
> > > > > > >>> > >> > > >> > > &
> > > > > > >>> > >> > > >> > > >> deb
> > > > > > >>> > >> > > >> > > >> >> > > files.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > > > > > >>> documentations/READMEs,
> > > > > > >>> > >> > LICENSE,
> > > > > > >>> > >> > > >> > > >> DISCLAMER,
> > > > > > >>> > >> > > >> > > >> >> > NOTES
> > > > > > >>> > >> > > >> > > >> >> > > > etc
> > > > > > >>> > >> > > >> > > >> >> > > > > if
> > > > > > >>> > >> > > >> > > >> >> > > > > > > any
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> > > > documentation
> > > > > > that
> > > > > > >>> > need
> > > > > > >>> > >> to
> > > > > > >>> > >> > be
> > > > > > >>> > >> > > >> > added
> > > > > > >>> > >> > > >> > > to
> > > > > > >>> > >> > > >> > > >> >> > > > > distribution
> > > > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > our
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> license
> > > > > > >>> information of
> > > > > > >>> > >> > > >> > dependencies
> > > > > > >>> > >> > > >> > > >> >> required
> > > > > > >>> > >> > > >> > > >> >> > > to
> > > > > > >>> > >> > > >> > > >> >> > > > be
> > > > > > >>> > >> > > >> > > >> >> > > > > > > added
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests
> (at
> > > > very
> > > > > > >>> least)
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding
> smoke
> > > > test
> > > > > > for
> > > > > > >>> > >> > packaging.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet
> recipes
> > > for
> > > > > > >>> automatic
> > > > > > >>> > >> > > >> deployment
> > > > > > >>> > >> > > >> > and
> > > > > > >>> > >> > > >> > > >> >> > > > configuration
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet
> > > > recipes
> > > > > > for
> > > > > > >>> > >> > automatic
> > > > > > >>> > >> > > >> > > >> deployment
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration
> > tests
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For
> > > verification
> > > > > of
> > > > > > >>> > >> > deployments.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license
> > > clearance:
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle
> > rat'
> > > > to
> > > > > > make
> > > > > > >>> > sure
> > > > > > >>> > >> all
> > > > > > >>> > >> > > new
> > > > > > >>> > >> > > >> > > stuff
> > > > > > >>> > >> > > >> > > >> is
> > > > > > >>> > >> > > >> > > >> >> > > > compliant
> > > > > > >>> > >> > > >> > > >> >> > > > > > > with
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license
> > > requirements.
> > > > If
> > > > > > >>> you add
> > > > > > >>> > >> code
> > > > > > >>> > >> > > >> > licenses
> > > > > > >>> > >> > > >> > > >> >> under
> > > > > > >>> > >> > > >> > > >> >> > > > > different
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need
> > to
> > > be
> > > > > > >>> listed in
> > > > > > >>> > >> the
> > > > > > >>> > >> > > >> NOTICE.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share
> your
> > > > > thoughts
> > > > > > >>> on
> > > > > > >>> > the
> > > > > > >>> > >> > > >> approach.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
> > > > > corresponding
> > > > > > >>> > >> > communication
> > > > > > >>> > >> > > >> on
> > > > > > >>> > >> > > >> > > >> bigtop
> > > > > > >>> > >> > > >> > > >> >> > > mailing
> > > > > > >>> > >> > > >> > > >> >> > > > > list
> > > > > > >>> > >> > > >> > > >> >> > > > > > > as
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some
> > > specific
> > > > > > >>> > >> > > >> questions/suggestions
> > > > > > >>> > >> > > >> > > >> >> related to
> > > > > > >>> > >> > > >> > > >> >> > > > what
> > > > > > >>> > >> > > >> > > >> >> > > > > > > should
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > be
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the
> > > package
> > > > > and
> > > > > > >>> what
> > > > > > >>> > >> > should
> > > > > > >>> > >> > > be
> > > > > > >>> > >> > > >> > the
> > > > > > >>> > >> > > >> > > >> smoke
> > > > > > >>> > >> > > >> > > >> >> > > tests,
> > > > > > >>> > >> > > >> > > >> >> > > > > but
> > > > > > >>> > >> > > >> > > >> >> > > > > > > in
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > the
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not
> > > having
> > > > > too
> > > > > > >>> much
> > > > > > >>> > >> > content
> > > > > > >>> > >> > > >> > here,
> > > > > > >>> > >> > > >> > > >> we'll
> > > > > > >>> > >> > > >> > > >> >> > put
> > > > > > >>> > >> > > >> > > >> >> > > > the
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > questions
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in
> > > this
> > > > > > >>> > mailthread.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > > >>> > >> > > >> > > >> >> > > >
> > > > > > >>> > >> > > >> > > >> >> > >
> > > > > > >>> > >> > > >> > > >> >> >
> > > > > > >>> > >> > > >> > > >> >>
> > > > > > >>> > >> > > >> > > >> >
> > > > > > >>> > >> > > >> > > >> >
> > > > > > >>> > >> > > >> > > >>
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > > >
> > > > > > >>> > >> > > >> > >
> > > > > > >>> > >> > > >> >
> > > > > > >>> > >> > > >>
> > > > > > >>> > >> > > >
> > > > > > >>> > >> > > >
> > > > > > >>> > >> > >
> > > > > > >>> > >> >
> > > > > > >>> > >>
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Justin Mclean <ju...@me.com>]

Hi,

> I just did a quick install of bigtop 1.1.0 on ubuntu and observed that
> jersey-client-1.9.jar is already packaged in hadoop
> /usr/lib/hadoop/client/jersey-client.jar,

Jersey is dual CDDL/GPL licensed, which is a Category B license. [1]  Changes will need to be made to theLICENSE and NOTICE if this is added.

Thanks,
Justin

1. https://jersey.java.net/license.html <https://jersey.java.net/license.html>
2. http://www.apache.org/legal/resolved.html#category-b <http://www.apache.org/legal/resolved.html#category-b>

Re: Apache Apex as component in Apache bigtop

[Tushar Gosavi <tu...@datatorrent.com>]

I just did a quick install of bigtop 1.1.0 on ubuntu and observed that
 jersey-client-1.9.jar is already packaged in hadoop
/usr/lib/hadoop/client/jersey-client.jar, so I guess it will be ok to
include those in apex lib too. or If we are including hadoop jars
in classpath while launching the apex application then it may not be needed.

- Tushar.



On Fri, Feb 19, 2016 at 8:44 AM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> No worries David. I've updated the manpage.
>
> Just an update: apex bigtop packaging for rpm+deb is done. Here is the
> branch on which I'm working if someone wants to look at:
> https://github.com/chinmaykolhatkar/bigtop/blob/BIGTOP-2314
>
> Only the 3rd questions below is the question that is left to be answered.
> Once this is resolved, I'll create the pull request in bigtop.
>
> Following questions are related to licenses and inclusion of certain files
> in apex rpm+deb:
> 1. I'll be included LICENSE, DISCLAIMER, NOTICE from top folder of core
> repo in our rpm and deb package. Those will be present in standard location
> i.e. /usr/lib/apex/
> Please let me know otherwise.
>
> 2. Most of our runtime dependencies has ASF license, but certain libraries
> has MIT and BSD licenses, hence I've included those licenses too in rpm &
> deb package in same location as above.
> Please let me know otherwise.
>
> BSD - jline-2.11.jar, kryo-2.24.0.jar, minlog-1.2.jar
> MIT - mbassador-1.1.9.jar
>
> 3. There are 2 jars (jersey-apache-client4-1.9.jar, jersey-client-1.9.jar)
> which has duel license of CDDL + GPLv2 with classpath exception:
> https://jersey.java.net/license.html
> I'm not an expert here. Hence can someone let me know whether these
> libraries is OK to be packaged in rpm+deb of apex?
>
> If yes, should we also include that license in our package?
>
> Thanks,
> Chinmay.
>
>
>
> On Thu, Feb 18, 2016 at 11:35 PM, David Yan <da...@datatorrent.com> wrote:
>
> > I should've mentioned that I had a pull request that was merged yesterday
> > that corrected the statement:
> >
> > https://github.com/DataTorrent/docs/pull/45
> >
> > On Thu, Feb 18, 2016 at 12:07 AM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > In following statement, can you replace dtGateway with
> ApplicationMaster
> > > and Yarn. or may be remove this statement entirely.
> > > apex is a wrapper around the REST API provided by dtGatway,
> > >
> > > - Tushar.
> > >
> > >
> > >
> > >
> > > On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <
> chinmay@apache.org>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I've attached man page for apex in Jira. Here is the link to it:
> > > > https://issues.apache.org/jira/secure/attachment/12788403/apex.1
> > > >
> > > > Please view it by: man -l apex.1
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <
> > tushar@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Hi Chinmay,
> > > > >
> > > > > I am not able to find the attachment, may be attachments are not
> > > allowed
> > > > in
> > > > > the mailing list. Can you attach the man page to the Jira?
> > > > >
> > > > > - Tushar.
> > > > >
> > > > >
> > > > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <
> > chinmay@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I've attached man page file for the purpose of bigtop
> integration.
> > > > > > Please have a look and let me know if there any any feedback
> about
> > > it.
> > > > > >
> > > > > > You can see the man page content of this file with:
> > > > > > man -l apex.1
> > > > > >
> > > > > > Thanks,
> > > > > > Chinmay.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> > > > chinmay@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > >> Thanks. Will follow bigtop recommendation. i.e. remove
> > > "-incubating".
> > > > > >>
> > > > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
> > > thomas@datatorrent.com
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Please follow the recommendation and continue discussion on the
> > > > Bigtop
> > > > > >>> list.
> > > > > >>>
> > > > > >>> --
> > > > > >>> sent from mobile
> > > > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <
> chinmay@apache.org
> > >
> > > > > wrote:
> > > > > >>>
> > > > > >>> > I got a recommendation from bigtop community to drop
> > > "-incubating"
> > > > > from
> > > > > >>> > version.
> > > > > >>> >
> > > > > >>> > Here is the mail thread:
> > > > > >>> >
> > > > > >>> >
> > > > > >>>
> > > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> > > > > >>> >
> > > > > >>> > Please let me know if removing "-incubating" is fine.
> > > > > >>> >
> > > > > >>> > Thanks,
> > > > > >>> > Chinmay.
> > > > > >>> >
> > > > > >>> >
> > > > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> > > > > chinmay@apache.org
> > > > > >>> >
> > > > > >>> > wrote:
> > > > > >>> >
> > > > > >>> > > I've already started communication there.
> > > > > >>> > >
> > > > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> > > > > >>> thomas@datatorrent.com>
> > > > > >>> > > wrote:
> > > > > >>> > >
> > > > > >>> > >> This is a question you should ask on the Bigtop mailing
> > list.
> > > > > >>> > >>
> > > > > >>> > >> --
> > > > > >>> > >> sent from mobile
> > > > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <
> > prad@apache.org>
> > > > > >>> wrote:
> > > > > >>> > >>
> > > > > >>> > >> > Yes. In case of RPM, the only restriction on version is
> > that
> > > > it
> > > > > >>> can
> > > > > >>> > not
> > > > > >>> > >> > contain '-'.
> > > > > >>> > >> > However, as a workaround, one can use 'incubating' as a
> > part
> > > > of
> > > > > >>> > >> %{release}.
> > > > > >>> > >> >
> > > > > >>> > >> > Version: 3.3.0
> > > > > >>> > >> > Release: incubating
> > > > > >>> > >> >
> > > > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> > > > > >>> > >> >
> > > > > >>> > >> > It is the same practice followed for pre-release
> packages
> > > e.g.
> > > > > >>> with
> > > > > >>> > rcX,
> > > > > >>> > >> > alphaX, betaX being used in %{release}.
> > > > > >>> > >> >
> > > > > >>> > >> > Regards,
> > > > > >>> > >> > Pradeep A. Dalvi
> > > > > >>> > >> >
> > > > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> > > > > >>> > chinmay@apache.org>
> > > > > >>> > >> > wrote:
> > > > > >>> > >> >
> > > > > >>> > >> > > Hi Everyone,
> > > > > >>> > >> > >
> > > > > >>> > >> > > In one of the previous mails in this thread, we
> decided
> > > that
> > > > > >>> version
> > > > > >>> > >> of
> > > > > >>> > >> > the
> > > > > >>> > >> > > package will be "3.3.0-incubating".
> > > > > >>> > >> > > The problem that I'm facing is in RPM base version "-"
> > is
> > > > not
> > > > > >>> > allowed.
> > > > > >>> > >> > >
> > > > > >>> > >> > > Can we have package version to "3.3.0"?
> > > > > >>> > >> > > i.e. the packages created will be as follows:
> > > > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > > > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> > > > > >>> > >> > >
> > > > > >>> > >> > > There is another integration for Zepplin in Bigtop
> which
> > > is
> > > > > >>> > incubating
> > > > > >>> > >> > and
> > > > > >>> > >> > > is having base version as "0.5.6".
> > > > > >>> > >> > >
> > > > > >>> > >> > > Please let me know your thoughts.
> > > > > >>> > >> > >
> > > > > >>> > >> > > Thanks,
> > > > > >>> > >> > > Chinmay.
> > > > > >>> > >> > >
> > > > > >>> > >> > >
> > > > > >>> > >> > >
> > > > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> > > > > >>> > >> chinmay@apache.org>
> > > > > >>> > >> > > wrote:
> > > > > >>> > >> > >
> > > > > >>> > >> > > > Thanks everyone for the feedback.
> > > > > >>> > >> > > >
> > > > > >>> > >> > > > Also as Ram and David mentioned, I'll start creating
> > the
> > > > man
> > > > > >>> page
> > > > > >>> > as
> > > > > >>> > >> > > well.
> > > > > >>> > >> > > >
> > > > > >>> > >> > > > Thanks,
> > > > > >>> > >> > > > Chinmay.
> > > > > >>> > >> > > >
> > > > > >>> > >> > > >
> > > > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> > > > > >>> > >> thomas@datatorrent.com>
> > > > > >>> > >> > > > wrote:
> > > > > >>> > >> > > >
> > > > > >>> > >> > > >> I think we already agreed that we will retain the
> > > current
> > > > > >>> script
> > > > > >>> > >> name
> > > > > >>> > >> > in
> > > > > >>> > >> > > >> this bigtop iteration. There is a ticket for
> > renaming,
> > > > add
> > > > > >>> you
> > > > > >>> > >> > > suggestions
> > > > > >>> > >> > > >> there.
> > > > > >>> > >> > > >>
> > > > > >>> > >> > > >> --
> > > > > >>> > >> > > >> sent from mobile
> > > > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > > > > >>> > sandesh@datatorrent.com>
> > > > > >>> > >> > > wrote:
> > > > > >>> > >> > > >>
> > > > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start the
> > > > transition
> > > > > >>> > >> process by
> > > > > >>> > >> > > >> making
> > > > > >>> > >> > > >> > dtcli as the symlink to apex cli.
> > > > > >>> > >> > > >> >
> > > > > >>> > >> > > >> > Renaming is a breaking change should be done
> during
> > > > major
> > > > > >>> > version
> > > > > >>> > >> > > number
> > > > > >>> > >> > > >> > change.
> > > > > >>> > >> > > >> >
> > > > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> > > > > >>> > david@datatorrent.com
> > > > > >>> > >> >
> > > > > >>> > >> > > >> wrote:
> > > > > >>> > >> > > >> >
> > > > > >>> > >> > > >> > > Apologies for hitting the "Send" button too
> > early.
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > What is the scope of the name change?  If we
> > simply
> > > > > have
> > > > > >>> a
> > > > > >>> > >> symlink
> > > > > >>> > >> > > >> from
> > > > > >>> > >> > > >> > > apex to dtcli, then it should have no backward
> > > > > >>> compatibility
> > > > > >>> > >> > problem
> > > > > >>> > >> > > >> at
> > > > > >>> > >> > > >> > > all.
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > But the proper name change would involve:
> > > > > >>> > >> > > >> > > - Change the name of DTCli class to ApexCli
> > > > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> > > > > >>> > >> > > >> > > - Change the initial greeting when running the
> > CLI
> > > > from
> > > > > >>> > >> something
> > > > > >>> > >> > > like
> > > > > >>> > >> > > >> > "DT
> > > > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > The above changes may break existing scripts
> that
> > > > rely
> > > > > >>> on the
> > > > > >>> > >> the
> > > > > >>> > >> > > >> string
> > > > > >>> > >> > > >> > > "DT".
> > > > > >>> > >> > > >> > > But if folks are okay with this potential
> > problem,
> > > I
> > > > am
> > > > > >>> okay
> > > > > >>> > >> with
> > > > > >>> > >> > > the
> > > > > >>> > >> > > >> > name
> > > > > >>> > >> > > >> > > change.
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > For the man page, we can start with the content
> > of
> > > > this
> > > > > >>> page:
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > Plus the command line options when doing dtcli
> > -h:
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > usage: DTCli
> > > > > >>> > >> > > >> > >  -e <arg>    Commands are read from the
> argument
> > > > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at all
> time
> > > > > >>> > >> > > >> > >  -h          Print this help
> > > > > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos
> principal
> > > > > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> > > > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> > > > > >>> > >> > > >> > >  -r          JSON Raw mode
> > > > > >>> > >> > > >> > >  -v          Verbose mode level 1
> > > > > >>> > >> > > >> > >  -vv         Verbose mode level 2
> > > > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> > > > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > David
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> > > > > >>> > >> david@datatorrent.com
> > > > > >>> > >> > >
> > > > > >>> > >> > > >> > wrote:
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > > What is the scope of the name change?  If we
> > > simply
> > > > > >>> have a
> > > > > >>> > >> > symlink
> > > > > >>> > >> > > >> from
> > > > > >>> > >> > > >> > > > apex to dtcli, then it should have no
> backward
> > > > > >>> > compatibility
> > > > > >>> > >> > > >> problem at
> > > > > >>> > >> > > >> > > all.
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > > > But the proper name change would involve:
> > > > > >>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
> > > > > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> > > > > >>> > >> > > >> > > > - Change the initial greeting when running
> the
> > > CLI
> > > > > from
> > > > > >>> > >> > something
> > > > > >>> > >> > > >> like
> > > > > >>> > >> > > >> > > "DT
> > > > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > > > The above changes may break existing scripts
> > that
> > > > > rely
> > > > > >>> on
> > > > > >>> > the
> > > > > >>> > >> > the
> > > > > >>> > >> > > >> > string
> > > > > >>> > >> > > >> > > > "DT".
> > > > > >>> > >> > > >> > > > But if folks are okay with this potential
> > > problem,
> > > > I
> > > > > am
> > > > > >>> > okay
> > > > > >>> > >> > with
> > > > > >>> > >> > > >> the
> > > > > >>> > >> > > >> > > name
> > > > > >>> > >> > > >> > > > change.
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > > > For the man page, we can start with the
> content
> > > of
> > > > > this
> > > > > >>> > page:
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay
> > > Kolhatkar
> > > > <
> > > > > >>> > >> > > >> chinmay@apache.org
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> > > > wrote:
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > > >> Here is a suggestion I have related to name
> > > > change:
> > > > > >>> > >> > > >> > > >>
> > > > > >>> > >> > > >> > > >> 1. While packaging we change the name of
> > "dtcli"
> > > > > >>> script to
> > > > > >>> > >> > > "apex".
> > > > > >>> > >> > > >> > This
> > > > > >>> > >> > > >> > > is
> > > > > >>> > >> > > >> > > >> done so that in future when we change the
> name
> > > to
> > > > > >>> "apex",
> > > > > >>> > >> users
> > > > > >>> > >> > > of
> > > > > >>> > >> > > >> > > bigtop
> > > > > >>> > >> > > >> > > >> apex don't have to transition much.
> > > > > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli which
> > > points
> > > > > to
> > > > > >>> apex
> > > > > >>> > >> > > script.
> > > > > >>> > >> > > >> > This
> > > > > >>> > >> > > >> > > is
> > > > > >>> > >> > > >> > > >> for backward compatibility.
> > > > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first
> > > > > integration
> > > > > >>> in
> > > > > >>> > >> > bigtop.
> > > > > >>> > >> > > >> This
> > > > > >>> > >> > > >> > is
> > > > > >>> > >> > > >> > > >> in
> > > > > >>> > >> > > >> > > >> the interest of not changing the content of
> > > dtcli
> > > > or
> > > > > >>> apex
> > > > > >>> > >> > script.
> > > > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these
> > > naming
> > > > > >>> related
> > > > > >>> > >> > > changes
> > > > > >>> > >> > > >> and
> > > > > >>> > >> > > >> > > >> update bigtop repository later to remove
> > > > references
> > > > > to
> > > > > >>> > dtcli
> > > > > >>> > >> > all
> > > > > >>> > >> > > >> > > together.
> > > > > >>> > >> > > >> > > >>
> > > > > >>> > >> > > >> > > >> Please share your thoughts on above
> approach.
> > > > > >>> > >> > > >> > > >>
> > > > > >>> > >> > > >> > > >> Also, please share what could be the man
> page
> > > > > content
> > > > > >>> for
> > > > > >>> > >> > "apex".
> > > > > >>> > >> > > >> > > >>
> > > > > >>> > >> > > >> > > >> Thanks,
> > > > > >>> > >> > > >> > > >> Chinmay.
> > > > > >>> > >> > > >> > > >>
> > > > > >>> > >> > > >> > > >>
> > > > > >>> > >> > > >> > > >>
> > > > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay
> > > > Kolhatkar <
> > > > > >>> > >> > > >> > chinmay@apache.org>
> > > > > >>> > >> > > >> > > >> wrote:
> > > > > >>> > >> > > >> > > >>
> > > > > >>> > >> > > >> > > >> > Yes.. It'll be build from source tar
> > > downloaded
> > > > > >>> from one
> > > > > >>> > >> of
> > > > > >>> > >> > the
> > > > > >>> > >> > > >> > apache
> > > > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> > > > > >>> > >> > > >> > > >> >
> > > > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced
> > from
> > > > > dtcli.
> > > > > >>> > This
> > > > > >>> > >> > > leaves
> > > > > >>> > >> > > >> us
> > > > > >>> > >> > > >> > 3
> > > > > >>> > >> > > >> > > >> > options:
> > > > > >>> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli
> > > would
> > > > > >>> need a
> > > > > >>> > >> > change
> > > > > >>> > >> > > >> while
> > > > > >>> > >> > > >> > > >> > packaging.
> > > > > >>> > >> > > >> > > >> >
> > > > > >>> > >> > > >> > > >> > But if we want to use dtcli as it is, then
> > we
> > > > > would
> > > > > >>> need
> > > > > >>> > >> > > >> dt-env.sh
> > > > > >>> > >> > > >> > > >> >
> > > > > >>> > >> > > >> > > >> >
> > > > > >>> > >> > > >> > > >> >
> > > > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas
> > Weise
> > > <
> > > > > >>> > >> > > >> > thomas@datatorrent.com
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > > >> > wrote:
> > > > > >>> > >> > > >> > > >> >
> > > > > >>> > >> > > >> > > >> >> Looks good overall, though there
> shouldn't
> > be
> > > > any
> > > > > >>> > >> dt-env.sh
> > > > > >>> > >> > > >> > > >> >>
> > > > > >>> > >> > > >> > > >> >> What will the bigtop package be built
> from,
> > > the
> > > > > >>> source
> > > > > >>> > >> tar?
> > > > > >>> > >> > > >> > > >> >>
> > > > > >>> > >> > > >> > > >> >>
> > > > > >>> > >> > > >> > > >> >>
> > > > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay
> > > > > Kolhatkar
> > > > > >>> <
> > > > > >>> > >> > > >> > > chinmay@apache.org
> > > > > >>> > >> > > >> > > >> >
> > > > > >>> > >> > > >> > > >> >> wrote:
> > > > > >>> > >> > > >> > > >> >>
> > > > > >>> > >> > > >> > > >> >> > Hi Thomas,
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > First, we're not changing the name
> > > anywhere.
> > > > > >>> We'll
> > > > > >>> > >> follow
> > > > > >>> > >> > > what
> > > > > >>> > >> > > >> > > >> >> currently is
> > > > > >>> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating
> > > > version.
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory
> > > > structure
> > > > > >>> below
> > > > > >>> > >> which
> > > > > >>> > >> > > is
> > > > > >>> > >> > > >> > > inline
> > > > > >>> > >> > > >> > > >> >> with
> > > > > >>> > >> > > >> > > >> >> > other existing integrations with
> bigtop.
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > The need for each file is as follows:
> > > > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked
> from
> > > > > >>> > >> > > >> engine/src/main/scripts/
> > > > > >>> > >> > > >> > > of
> > > > > >>> > >> > > >> > > >> >> source
> > > > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source
> from
> > > > dtcli.
> > > > > >>> This
> > > > > >>> > >> > > searches
> > > > > >>> > >> > > >> for
> > > > > >>> > >> > > >> > > >> hadoop
> > > > > >>> > >> > > >> > > >> >> > binary path and exports an env variable
> > for
> > > > > >>> dtcli to
> > > > > >>> > >> use
> > > > > >>> > >> > it.
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list
> > is
> > > > > >>> extracted
> > > > > >>> > >> in a
> > > > > >>> > >> > > >> similar
> > > > > >>> > >> > > >> > > >> way to
> > > > > >>> > >> > > >> > > >> >> > how DT Community Edition finds
> dependency
> > > > jars.
> > > > > >>> > >> > > >> > > >> >> > Please note that this is the first
> > > iteration
> > > > > >>> list of
> > > > > >>> > >> jars.
> > > > > >>> > >> > > I'm
> > > > > >>> > >> > > >> > > >> trying to
> > > > > >>> > >> > > >> > > >> >> > narrow this down to only those which
> are
> > > > really
> > > > > >>> > >> required.
> > > > > >>> > >> > > >> > > >> >> > The test that I'm running to check if
> > dtcli
> > > > > runs
> > > > > >>> fine
> > > > > >>> > >> with
> > > > > >>> > >> > > >> given
> > > > > >>> > >> > > >> > > >> >> dependency
> > > > > >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for
> pi
> > > > demo.
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > Above is the directory structure which
> is
> > > > > >>> required
> > > > > >>> > for
> > > > > >>> > >> CLI
> > > > > >>> > >> > > to
> > > > > >>> > >> > > >> > work.
> > > > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has
> mentioned
> > > are
> > > > > some
> > > > > >>> > >> > additional
> > > > > >>> > >> > > >> > files
> > > > > >>> > >> > > >> > > >> which
> > > > > >>> > >> > > >> > > >> >> > makes the like of administrator easier.
> > For
> > > > eg.
> > > > > >>> > >> /etc/skel
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > Please let me know if this is inline
> with
> > > > what
> > > > > >>> you're
> > > > > >>> > >> > > >> thinking.
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > Thanks,
> > > > > >>> > >> > > >> > > >> >> > Chinmay.
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > .
> > > > > >>> > >> > > >> > > >> >> > |-- bin
> > > > > >>> > >> > > >> > > >> >> > |   `-- dtcli
> > > > > >>> > >> > > >> > > >> >> > |-- conf
> > > > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > > > > >>> > >> > > >> > > >> >> > `-- lib
> > > > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > > > > >>> > >> > > >> > > >> >> >     |--
> > > > apex-bufferserver-3.3.0-incubating.jar
> > > > > >>> > >> > > >> > > >> >> >     |--
> apex-common-3.3.0-incubating.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > > > >>> > >> > > >> > > >> >> >     |--
> > > jetty-websocket-8.1.10.v20130312.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > > > > >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > > > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas
> > > > Weise <
> > > > > >>> > >> > > >> > > >> thomas@datatorrent.com>
> > > > > >>> > >> > > >> > > >> >> > wrote:
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >> > > Chinmay,
> > > > > >>> > >> > > >> > > >> >> > >
> > > > > >>> > >> > > >> > > >> >> > > Before discussing where to put the
> > files,
> > > > > let's
> > > > > >>> > make
> > > > > >>> > >> > sure
> > > > > >>> > >> > > >> they
> > > > > >>> > >> > > >> > > are
> > > > > >>> > >> > > >> > > >> >> really
> > > > > >>> > >> > > >> > > >> >> > > needed for the operation of the CLI.
> As
> > > for
> > > > > >>> names,
> > > > > >>> > >> > > anything
> > > > > >>> > >> > > >> new
> > > > > >>> > >> > > >> > > >> needs
> > > > > >>> > >> > > >> > > >> >> to
> > > > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and should
> > > follow
> > > > > >>> common
> > > > > >>> > >> > > >> conventions,
> > > > > >>> > >> > > >> > > >> >> especially
> > > > > >>> > >> > > >> > > >> >> > > in Bigtop where there are many
> existing
> > > > > >>> > integrations
> > > > > >>> > >> to
> > > > > >>> > >> > > look
> > > > > >>> > >> > > >> > at.
> > > > > >>> > >> > > >> > > >> >> > >
> > > > > >>> > >> > > >> > > >> >> > > Thanks,
> > > > > >>> > >> > > >> > > >> >> > > Thomas
> > > > > >>> > >> > > >> > > >> >> > >
> > > > > >>> > >> > > >> > > >> >> > >
> > > > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM,
> > Chinmay
> > > > > >>> Kolhatkar
> > > > > >>> > <
> > > > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > > > > >>> > >> > > >> > > >> >> > > wrote:
> > > > > >>> > >> > > >> > > >> >> > >
> > > > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep and
> > > Aniruddha.
> > > > > >>> > >> > > >> > > >> >> > > >
> > > > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to
> change
> > > the
> > > > > >>> dtcli
> > > > > >>> > >> > > >> considering it
> > > > > >>> > >> > > >> > > >> works
> > > > > >>> > >> > > >> > > >> >> > with
> > > > > >>> > >> > > >> > > >> >> > > DT
> > > > > >>> > >> > > >> > > >> >> > > > Community edition. We can keep the
> > > > > directory
> > > > > >>> > >> structure
> > > > > >>> > >> > > >> > similar
> > > > > >>> > >> > > >> > > to
> > > > > >>> > >> > > >> > > >> >> that.
> > > > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which
> > contains
> > > > > >>> > information
> > > > > >>> > >> > > >> required
> > > > > >>> > >> > > >> > for
> > > > > >>> > >> > > >> > > >> >> dtcli
> > > > > >>> > >> > > >> > > >> >> > to
> > > > > >>> > >> > > >> > > >> >> > > > launch.
> > > > > >>> > >> > > >> > > >> >> > > >
> > > > > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop
> community
> > > > that
> > > > > >>> > whether
> > > > > >>> > >> > they
> > > > > >>> > >> > > >> > > >> facilitate
> > > > > >>> > >> > > >> > > >> >> the
> > > > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs before
> user
> > > is
> > > > > >>> created.
> > > > > >>> > >> In
> > > > > >>> > >> > > >> either
> > > > > >>> > >> > > >> > > case,
> > > > > >>> > >> > > >> > > >> >> > current
> > > > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home
> > > > > directory.
> > > > > >>> > Also
> > > > > >>> > >> > > before
> > > > > >>> > >> > > >> > > putting
> > > > > >>> > >> > > >> > > >> >> > > anything
> > > > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define what
> > are
> > > > the
> > > > > >>> > default
> > > > > >>> > >> > > >> contents
> > > > > >>> > >> > > >> > > that
> > > > > >>> > >> > > >> > > >> >> > should
> > > > > >>> > >> > > >> > > >> >> > > go
> > > > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no
> > > > defaults,
> > > > > >>> > >> probably we
> > > > > >>> > >> > > >> should
> > > > > >>> > >> > > >> > > not
> > > > > >>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > > > > >>> > >> > > >> > > >> >> > > >
> > > > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks
> > nice.
> > > > > This
> > > > > >>> way
> > > > > >>> > >> > > contents
> > > > > >>> > >> > > >> of
> > > > > >>> > >> > > >> > > >> >> dt-env.sh
> > > > > >>> > >> > > >> > > >> >> > > are
> > > > > >>> > >> > > >> > > >> >> > > > present as env variables. But I
> see a
> > > > catch
> > > > > >>> > there.
> > > > > >>> > >> > > Adding
> > > > > >>> > >> > > >> > > >> dt-env.sh
> > > > > >>> > >> > > >> > > >> >> to
> > > > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the
> > > > variables
> > > > > >>> > >> available
> > > > > >>> > >> > at
> > > > > >>> > >> > > >> > > runtime
> > > > > >>> > >> > > >> > > >> all
> > > > > >>> > >> > > >> > > >> >> > the
> > > > > >>> > >> > > >> > > >> >> > > > time. I feel a little skeptical
> about
> > > > that.
> > > > > >>> > Maybe a
> > > > > >>> > >> > > >> possible
> > > > > >>> > >> > > >> > > >> >> collision
> > > > > >>> > >> > > >> > > >> >> > > can
> > > > > >>> > >> > > >> > > >> >> > > > occur with other application vars.
> > > > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
> > > > > >>> > >> > "../conf/dt-env.sh"
> > > > > >>> > >> > > >> and
> > > > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those
> variables
> > > are
> > > > > >>> anyway
> > > > > >>> > >> > > >> available.
> > > > > >>> > >> > > >> > > >> >> > > >
> > > > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort
> is
> > > > > >>> happening
> > > > > >>> > at
> > > > > >>> > >> > > Bigtop
> > > > > >>> > >> > > >> > > >> >> > > Jira/repository.
> > > > > >>> > >> > > >> > > >> >> > > > Packaging related code usually goes
> > > > there.
> > > > > >>> > (That's
> > > > > >>> > >> > what
> > > > > >>> > >> > > >> all
> > > > > >>> > >> > > >> > the
> > > > > >>> > >> > > >> > > >> >> > > components
> > > > > >>> > >> > > >> > > >> >> > > > in bigtop does).
> > > > > >>> > >> > > >> > > >> >> > > > Having said that, once a PR is
> > created,
> > > > > I'll
> > > > > >>> be
> > > > > >>> > >> > sharing
> > > > > >>> > >> > > >> the
> > > > > >>> > >> > > >> > > link
> > > > > >>> > >> > > >> > > >> of
> > > > > >>> > >> > > >> > > >> >> PR
> > > > > >>> > >> > > >> > > >> >> > > > here, so that apex community as
> well
> > > can
> > > > > >>> review
> > > > > >>> > it.
> > > > > >>> > >> > > >> > > >> >> > > >
> > > > > >>> > >> > > >> > > >> >> > > > Thanks,
> > > > > >>> > >> > > >> > > >> >> > > > Chinmay.
> > > > > >>> > >> > > >> > > >> >> > > >
> > > > > >>> > >> > > >> > > >> >> > > >
> > > > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM,
> > > > Aniruddha
> > > > > >>> > >> Thombare <
> > > > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > > > > >>> > >> > > >> > > >> >> > > >
> > > > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> > > > > >>> > >> > > >> > > >> >> > > > > We may need to iron out details
> > about
> > > > > exact
> > > > > >>> > paths
> > > > > >>> > >> > etc.
> > > > > >>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR
> > > > comments.
> > > > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm
> Pradeep
> > > A.
> > > > > >>> Dalvi <
> > > > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > > > > >>> > >> > > >> > > >> >> > > > > wrote:
> > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51
> PM,
> > > > > Chinmay
> > > > > >>> > >> > Kolhatkar <
> > > > > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > > > > >>> > >> > > >> > > >> >> > > > > > wrote:
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar
> > present
> > > in
> > > > > DT
> > > > > >>> > >> community
> > > > > >>> > >> > > >> > edition
> > > > > >>> > >> > > >> > > >> will
> > > > > >>> > >> > > >> > > >> >> be
> > > > > >>> > >> > > >> > > >> >> > > > > included
> > > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I
> > found
> > > > from
> > > > > >>> > common
> > > > > >>> > >> > > >> > dependencies
> > > > > >>> > >> > > >> > > >> >> between
> > > > > >>> > >> > > >> > > >> >> > > of
> > > > > >>> > >> > > >> > > >> >> > > > > apex
> > > > > >>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> > > > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> validation-api-1.1.0.Final.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running
> some
> > > > tests
> > > > > do
> > > > > >>> > check
> > > > > >>> > >> > that
> > > > > >>> > >> > > >> > dtcli
> > > > > >>> > >> > > >> > > >> works
> > > > > >>> > >> > > >> > > >> >> > > > properly
> > > > > >>> > >> > > >> > > >> >> > > > > > for
> > > > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps
> > with
> > > > > only
> > > > > >>> > these
> > > > > >>> > >> > > >> libraries
> > > > > >>> > >> > > >> > > >> >> present in
> > > > > >>> > >> > > >> > > >> >> > > > > > isolation
> > > > > >>> > >> > > >> > > >> >> > > > > > > without dependency on local
> m2.
> > > > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there are
> > > unwanted
> > > > > >>> jars
> > > > > >>> > >> which
> > > > > >>> > >> > are
> > > > > >>> > >> > > >> used
> > > > > >>> > >> > > >> > > for
> > > > > >>> > >> > > >> > > >> >> > compile
> > > > > >>> > >> > > >> > > >> >> > > > > time
> > > > > >>> > >> > > >> > > >> >> > > > > > > dependency and not runtime in
> > > above
> > > > > >>> list
> > > > > >>> > >> which I
> > > > > >>> > >> > > can
> > > > > >>> > >> > > >> > drop
> > > > > >>> > >> > > >> > > >> to
> > > > > >>> > >> > > >> > > >> >> keep
> > > > > >>> > >> > > >> > > >> >> > > > > package
> > > > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > > > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure
> if
> > > all
> > > > > of
> > > > > >>> the
> > > > > >>> > >> > > >> dt-env.sh is
> > > > > >>> > >> > > >> > > >> >> required.
> > > > > >>> > >> > > >> > > >> >> > > > Only
> > > > > >>> > >> > > >> > > >> >> > > > > > > required values I see in it
> are
> > > > > >>> classpath.
> > > > > >>> > >> and
> > > > > >>> > >> > > >> dtcli is
> > > > > >>> > >> > > >> > > >> >> capable
> > > > > >>> > >> > > >> > > >> >> > of
> > > > > >>> > >> > > >> > > >> >> > > > > > building
> > > > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll
> > have
> > > > to
> > > > > >>> see
> > > > > >>> > how
> > > > > >>> > >> > dtcli
> > > > > >>> > >> > > >> can
> > > > > >>> > >> > > >> > > find
> > > > > >>> > >> > > >> > > >> >> the
> > > > > >>> > >> > > >> > > >> >> > jar
> > > > > >>> > >> > > >> > > >> >> > > > > path,
> > > > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf
> > > files.
> > > > If
> > > > > >>> we
> > > > > >>> > >> need a
> > > > > >>> > >> > > >> change
> > > > > >>> > >> > > >> > in
> > > > > >>> > >> > > >> > > >> >> dtcli,
> > > > > >>> > >> > > >> > > >> >> > > then
> > > > > >>> > >> > > >> > > >> >> > > > > > > community's opinion is
> required
> > > for
> > > > > >>> whether
> > > > > >>> > >> > dtcli
> > > > > >>> > >> > > >> > should
> > > > > >>> > >> > > >> > > >> >> change
> > > > > >>> > >> > > >> > > >> >> > in
> > > > > >>> > >> > > >> > > >> >> > > > our
> > > > > >>> > >> > > >> > > >> >> > > > > > repo
> > > > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with
> required
> > > > > changes
> > > > > >>> > >> exist in
> > > > > >>> > >> > > >> bigtop
> > > > > >>> > >> > > >> > > >> repo
> > > > > >>> > >> > > >> > > >> >> > until
> > > > > >>> > >> > > >> > > >> >> > > we
> > > > > >>> > >> > > >> > > >> >> > > > > > make
> > > > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > Can we have these file paths
> set
> > > > using
> > > > > >>> > >> environment
> > > > > >>> > >> > > >> > > variables
> > > > > >>> > >> > > >> > > >> >> with
> > > > > >>> > >> > > >> > > >> >> > > some
> > > > > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> > > > > >>> > >> > > >> > > >> >> > > > > > Then we can set such params in
> > > > > dt-env.sh.
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is
> this
> > > file
> > > > > >>> > different
> > > > > >>> > >> > from
> > > > > >>> > >> > > >> > > >> >> dt-site.xml?
> > > > > >>> > >> > > >> > > >> >> > Not
> > > > > >>> > >> > > >> > > >> >> > > > > sure
> > > > > >>> > >> > > >> > > >> >> > > > > > if
> > > > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would
> help.
> > > > > Files &
> > > > > >>> > Dirs
> > > > > >>> > >> > from
> > > > > >>> > >> > > >> > > /etc/skel
> > > > > >>> > >> > > >> > > >> >> are
> > > > > >>> > >> > > >> > > >> >> > > > copied
> > > > > >>> > >> > > >> > > >> >> > > > > to
> > > > > >>> > >> > > >> > > >> >> > > > > > > home of new user when useradd
> > > > program
> > > > > >>> is
> > > > > >>> > >> called.
> > > > > >>> > >> > > But
> > > > > >>> > >> > > >> > for
> > > > > >>> > >> > > >> > > >> >> existing
> > > > > >>> > >> > > >> > > >> >> > > > users
> > > > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any use.
> > > Moreover,
> > > > > >>> dtcli
> > > > > >>> > >> > creates
> > > > > >>> > >> > > >> > ~/.dt/
> > > > > >>> > >> > > >> > > on
> > > > > >>> > >> > > >> > > >> >> the
> > > > > >>> > >> > > >> > > >> >> > fly
> > > > > >>> > >> > > >> > > >> >> > > > if
> > > > > >>> > >> > > >> > > >> >> > > > > > not
> > > > > >>> > >> > > >> > > >> >> > > > > > > encountered for the first
> time.
> > > > Again
> > > > > >>> > >> correct me
> > > > > >>> > >> > > if
> > > > > >>> > >> > > >> I'm
> > > > > >>> > >> > > >> > > >> wrong.
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right. However
> this
> > > > would
> > > > > >>> > >> probably be
> > > > > >>> > >> > > >> > > necessary
> > > > > >>> > >> > > >> > > >> >> step
> > > > > >>> > >> > > >> > > >> >> > for
> > > > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get
> > > > installed
> > > > > >>> > during
> > > > > >>> > >> OS
> > > > > >>> > >> > > >> install
> > > > > >>> > >> > > >> > > and
> > > > > >>> > >> > > >> > > >> >> > before
> > > > > >>> > >> > > >> > > >> >> > > > user
> > > > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc &
> > bash_profile
> > > > ->
> > > > > >>> I'm
> > > > > >>> > not
> > > > > >>> > >> > sure
> > > > > >>> > >> > > >> its
> > > > > >>> > >> > > >> > the
> > > > > >>> > >> > > >> > > >> best
> > > > > >>> > >> > > >> > > >> >> > > idea.
> > > > > >>> > >> > > >> > > >> >> > > > > > We'll
> > > > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location
> > > which
> > > > is
> > > > > >>> by
> > > > > >>> > >> default
> > > > > >>> > >> > > >> > present
> > > > > >>> > >> > > >> > > in
> > > > > >>> > >> > > >> > > >> >> path
> > > > > >>> > >> > > >> > > >> >> > > i.e.
> > > > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > For other env variables
> specific
> > to
> > > > > apex,
> > > > > >>> > I'll
> > > > > >>> > >> > > prefer
> > > > > >>> > >> > > >> to
> > > > > >>> > >> > > >> > > use
> > > > > >>> > >> > > >> > > >> >> > > dt-env.sh
> > > > > >>> > >> > > >> > > >> >> > > > > and
> > > > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather
> than
> > > > > changing
> > > > > >>> > >> bashrc
> > > > > >>> > >> > > >> etc...
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
> > > > > >>> /etc/profile.d/
> > > > > >>> > >> shall
> > > > > >>> > >> > > be
> > > > > >>> > >> > > >> the
> > > > > >>> > >> > > >> > > >> >> cleaner
> > > > > >>> > >> > > >> > > >> >> > way
> > > > > >>> > >> > > >> > > >> >> > > > to
> > > > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > Thanks,
> > > > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07
> > PM,
> > > > > >>> Aniruddha
> > > > > >>> > >> > > Thombare
> > > > > >>> > >> > > >> <
> > > > > >>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com>
> > > wrote:
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > We need to consider
> > following:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > System wide default config
> > > files
> > > > > can
> > > > > >>> be
> > > > > >>> > >> > located
> > > > > >>> > >> > > at
> > > > > >>> > >> > > >> > > >> following
> > > > > >>> > >> > > >> > > >> >> > > > > locations:
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we
> may
> > > have
> > > > > to
> > > > > >>> > change
> > > > > >>> > >> > > dtcli
> > > > > >>> > >> > > >> > > >> behaviour
> > > > > >>> > >> > > >> > > >> >> on
> > > > > >>> > >> > > >> > > >> >> > > how
> > > > > >>> > >> > > >> > > >> >> > > > it
> > > > > >>> > >> > > >> > > >> >> > > > > > > finds
> > > > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> > > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we
> may
> > > > have
> > > > > to
> > > > > >>> > >> change
> > > > > >>> > >> > > dtcli
> > > > > >>> > >> > > >> > > >> >> behaviour on
> > > > > >>> > >> > > >> > > >> >> > > how
> > > > > >>> > >> > > >> > > >> >> > > > > it
> > > > > >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
> > > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml
> > and
> > > > > other
> > > > > >>> > files
> > > > > >>> > >> > (for
> > > > > >>> > >> > > >> new
> > > > > >>> > >> > > >> > > users
> > > > > >>> > >> > > >> > > >> >> that
> > > > > >>> > >> > > >> > > >> >> > > will
> > > > > >>> > >> > > >> > > >> >> > > > > be
> > > > > >>> > >> > > >> > > >> >> > > > > > > > created in system in
> future.)
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > We may also have to modify
> > > > bashrc /
> > > > > >>> > >> > bashprofile
> > > > > >>> > >> > > >> for
> > > > > >>> > >> > > >> > > >> >> population
> > > > > >>> > >> > > >> > > >> >> > > any
> > > > > >>> > >> > > >> > > >> >> > > > > > > > variables that are
> required.
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your
> > > > comments /
> > > > > >>> > >> > suggestion.
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at
> 1:16
> > > AM,
> > > > > >>> Thomas
> > > > > >>> > >> Weise
> > > > > >>> > >> > <
> > > > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > -->
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at
> 9:05
> > > AM,
> > > > > >>> Chinmay
> > > > > >>> > >> > > >> Kolhatkar <
> > > > > >>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with
> packaging
> > > > > effort
> > > > > >>> > >> (rpm+deb)
> > > > > >>> > >> > > of
> > > > > >>> > >> > > >> > apex,
> > > > > >>> > >> > > >> > > >> here
> > > > > >>> > >> > > >> > > >> >> > are
> > > > > >>> > >> > > >> > > >> >> > > > some
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > about package structure
> > > etc..
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on
> > > bbigtop
> > > > > >>> mailing
> > > > > >>> > >> > list, I
> > > > > >>> > >> > > >> have
> > > > > >>> > >> > > >> > > >> some
> > > > > >>> > >> > > >> > > >> >> > > question
> > > > > >>> > >> > > >> > > >> >> > > > > for
> > > > > >>> > >> > > >> > > >> >> > > > > > > > apex
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory
> > > structure
> > > > of
> > > > > >>> apex
> > > > > >>> > >> > package
> > > > > >>> > >> > > >> for
> > > > > >>> > >> > > >> > > both
> > > > > >>> > >> > > >> > > >> >> deb &
> > > > > >>> > >> > > >> > > >> >> > > > rpm:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> /usr/lib/apex/lib/<other
> > > > > >>> dependent
> > > > > >>> > >> jars>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> > > > > >>> > /usr/lib/apex/bin/dtcli
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> /usr/share/doc/man/man1/dtcli.1.gz
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > /usr/share/doc/apex/license/LICENSE.txt
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > /usr/share/doc/apex/CHANGELOG
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > /usr/share/doc/apex/NOTICE
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
> > > 3.3.0-incubating.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars>
> =
> > > All
> > > > > the
> > > > > >>> 3rd
> > > > > >>> > >> party
> > > > > >>> > >> > > >> jars
> > > > > >>> > >> > > >> > > which
> > > > > >>> > >> > > >> > > >> >> are
> > > > > >>> > >> > > >> > > >> >> > > > > required
> > > > > >>> > >> > > >> > > >> >> > > > > > > for
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > apex
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the
> > > > > dependencies
> > > > > >>> are
> > > > > >>> > >> > > packaged
> > > > > >>> > >> > > >> as
> > > > > >>> > >> > > >> > > >> part of
> > > > > >>> > >> > > >> > > >> >> > > > rpm/deb
> > > > > >>> > >> > > >> > > >> >> > > > > by
> > > > > >>> > >> > > >> > > >> >> > > > > > > any
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what
> those
> > > jars
> > > > > >>> are in
> > > > > >>> > >> > Bigtop
> > > > > >>> > >> > > >> > > context?
> > > > > >>> > >> > > >> > > >> >> Same
> > > > > >>> > >> > > >> > > >> >> > as
> > > > > >>> > >> > > >> > > >> >> > > > > > shipped
> > > > > >>> > >> > > >> > > >> >> > > > > > > > with
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > DT community addition
> under
> > > > lib/
> > > > > ?
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> =
> > > Licenses
> > > > of
> > > > > >>> > >> > > corresponding
> > > > > >>> > >> > > >> of
> > > > > >>> > >> > > >> > > 3rd
> > > > > >>> > >> > > >> > > >> >> party
> > > > > >>> > >> > > >> > > >> >> > > jars
> > > > > >>> > >> > > >> > > >> >> > > > > > which
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > needs
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > to included while
> > > packaging.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions related to
> > this:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the
> cli
> > > of
> > > > > >>> apex as
> > > > > >>> > >> > "apex"
> > > > > >>> > >> > > >> > instead
> > > > > >>> > >> > > >> > > >> of
> > > > > >>> > >> > > >> > > >> >> > > "dtcli"
> > > > > >>> > >> > > >> > > >> >> > > > in
> > > > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > I think we should keep
> the
> > > name
> > > > > >>> until
> > > > > >>> > we
> > > > > >>> > >> are
> > > > > >>> > >> > > >> able
> > > > > >>> > >> > > >> > to
> > > > > >>> > >> > > >> > > >> >> change
> > > > > >>> > >> > > >> > > >> >> > it
> > > > > >>> > >> > > >> > > >> >> > > in
> > > > > >>> > >> > > >> > > >> >> > > > > > Apex.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get
> > > > confusing.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all
> > softwares
> > > > in
> > > > > >>> bigtop
> > > > > >>> > >> have
> > > > > >>> > >> > > man
> > > > > >>> > >> > > >> > page
> > > > > >>> > >> > > >> > > >> for
> > > > > >>> > >> > > >> > > >> >> > their
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we should have
> it
> > > too
> > > > > for
> > > > > >>> > >> dtcli.
> > > > > >>> > >> > Is
> > > > > >>> > >> > > >> there
> > > > > >>> > >> > > >> > > any
> > > > > >>> > >> > > >> > > >> >> > > > > documentation
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > which I
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man
> page?
> > or
> > > > can
> > > > > >>> I use
> > > > > >>> > >> > output
> > > > > >>> > >> > > >> of
> > > > > >>> > >> > > >> > > >> "dtcli
> > > > > >>> > >> > > >> > > >> >> > > --help"?
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call
> > > version
> > > > > of
> > > > > >>> apex
> > > > > >>> > >> in
> > > > > >>> > >> > > >> Bigtop
> > > > > >>> > >> > > >> > as
> > > > > >>> > >> > > >> > > >> >> 3.3.0 OR
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex
> > > package
> > > > of
> > > > > >>> > bigtop
> > > > > >>> > >> to
> > > > > >>> > >> > > >> depend
> > > > > >>> > >> > > >> > on
> > > > > >>> > >> > > >> > > >> >> 2.7.1
> > > > > >>> > >> > > >> > > >> >> > > > > version
> > > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any
> > problems
> > > > > that
> > > > > >>> we
> > > > > >>> > see
> > > > > >>> > >> > with
> > > > > >>> > >> > > >> this
> > > > > >>> > >> > > >> > > >> >> > dependency?
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > Should work without
> > changed.
> > > I
> > > > > >>> thought
> > > > > >>> > we
> > > > > >>> > >> > > >> certified
> > > > > >>> > >> > > >> > > >> >> against
> > > > > >>> > >> > > >> > > >> >> > > > 2.7.0?
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the
> > apache
> > > > > mirror
> > > > > >>> > from
> > > > > >>> > >> > which
> > > > > >>> > >> > > >> > bigtop
> > > > > >>> > >> > > >> > > >> will
> > > > > >>> > >> > > >> > > >> >> > pick
> > > > > >>> > >> > > >> > > >> >> > > > the
> > > > > >>> > >> > > >> > > >> >> > > > > > > apex
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > source for compilation
> > and
> > > > > >>> packaging.
> > > > > >>> > >> > Please
> > > > > >>> > >> > > >> > > correct
> > > > > >>> > >> > > >> > > >> if
> > > > > >>> > >> > > >> > > >> >> > > > > incorrect:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > >
> > > > > >>> > >> > >
> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
> > > apache.osuosl.org
> > > > is
> > > > > >>> the
> > > > > >>> > >> mirror
> > > > > >>> > >> > > >> used
> > > > > >>> > >> > > >> > by
> > > > > >>> > >> > > >> > > >> all
> > > > > >>> > >> > > >> > > >> >> the
> > > > > >>> > >> > > >> > > >> >> > > > > > softwares
> > > > > >>> > >> > > >> > > >> >> > > > > > > in
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share your
> > opinion.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at
> > > 11:19
> > > > > AM,
> > > > > >>> > >> Chinmay
> > > > > >>> > >> > > >> > Kolhatkar
> > > > > >>> > >> > > >> > > <
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> chinmay@datatorrent.com>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to
> work
> > on
> > > > the
> > > > > >>> > >> packaging.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already started
> > > > > >>> discussion on
> > > > > >>> > >> > bigtop
> > > > > >>> > >> > > >> dev
> > > > > >>> > >> > > >> > > >> mailing
> > > > > >>> > >> > > >> > > >> >> > list
> > > > > >>> > >> > > >> > > >> >> > > > for
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also
> > > created a
> > > > > >>> Jira
> > > > > >>> > for
> > > > > >>> > >> the
> > > > > >>> > >> > > >> same.
> > > > > >>> > >> > > >> > > For
> > > > > >>> > >> > > >> > > >> >> this
> > > > > >>> > >> > > >> > > >> >> > > > > > communities
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is
> the
> > > > bigtop
> > > > > >>> Jira:
> > > > > >>> > >> > > >> > BIGTOP-2313.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016
> at
> > > 1:13
> > > > > AM,
> > > > > >>> > >> Thomas
> > > > > >>> > >> > > >> Weise <
> > > > > >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything
> > > under
> > > > > >>> > >> > prerequisites
> > > > > >>> > >> > > >> that
> > > > > >>> > >> > > >> > > >> would
> > > > > >>> > >> > > >> > > >> >> be a
> > > > > >>> > >> > > >> > > >> >> > > > > > problem.
> > > > > >>> > >> > > >> > > >> >> > > > > > > We
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
> > > > > >>> compatibility
> > > > > >>> > >> as
> > > > > >>> > >> > > part
> > > > > >>> > >> > > >> of
> > > > > >>> > >> > > >> > > >> >> becoming
> > > > > >>> > >> > > >> > > >> >> > an
> > > > > >>> > >> > > >> > > >> >> > > > > > > incubator
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the
> > > > > packaging
> > > > > >>> > during
> > > > > >>> > >> > the
> > > > > >>> > >> > > >> next
> > > > > >>> > >> > > >> > > >> weeks.
> > > > > >>> > >> > > >> > > >> >> > Since
> > > > > >>> > >> > > >> > > >> >> > > > the
> > > > > >>> > >> > > >> > > >> >> > > > > > > work
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > will
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop,
> > related
> > > > > >>> > discussions
> > > > > >>> > >> and
> > > > > >>> > >> > > >> JIRAs
> > > > > >>> > >> > > >> > > >> should
> > > > > >>> > >> > > >> > > >> >> > also
> > > > > >>> > >> > > >> > > >> >> > > be
> > > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to
> have
> > > the
> > > > > >>> > packaging
> > > > > >>> > >> in
> > > > > >>> > >> > > >> place
> > > > > >>> > >> > > >> > by
> > > > > >>> > >> > > >> > > >> end
> > > > > >>> > >> > > >> > > >> >> Feb.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016
> at
> > > > 10:14
> > > > > >>> AM,
> > > > > >>> > >> > Chinmay
> > > > > >>> > >> > > >> > > >> Kolhatkar <
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > chinmay@datatorrent.com
> > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a
> > work
> > > on
> > > > > >>> adding
> > > > > >>> > >> > Apache
> > > > > >>> > >> > > >> Apex
> > > > > >>> > >> > > >> > > as a
> > > > > >>> > >> > > >> > > >> >> > > component
> > > > > >>> > >> > > >> > > >> >> > > > > to
> > > > > >>> > >> > > >> > > >> >> > > > > > > > Apache
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the
> > > packaging
> > > > > >>> system
> > > > > >>> > for
> > > > > >>> > >> > the
> > > > > >>> > >> > > >> > Apache
> > > > > >>> > >> > > >> > > >> big
> > > > > >>> > >> > > >> > > >> >> data
> > > > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use
> > it,
> > > > > most
> > > > > >>> > >> recently
> > > > > >>> > >> > > EMR.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the
> tracking
> > > > Jira
> > > > > >>> task
> > > > > >>> > in
> > > > > >>> > >> > > >> APEXCORE
> > > > > >>> > >> > > >> > for
> > > > > >>> > >> > > >> > > >> the
> > > > > >>> > >> > > >> > > >> >> > same:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >>
> > > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of
> > > > execution
> > > > > >>> is as
> > > > > >>> > >> > > follows:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
> > > > > >>> prerequisites*
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has
> > some
> > > > > hard
> > > > > >>> and
> > > > > >>> > >> soft
> > > > > >>> > >> > > >> > > expectation
> > > > > >>> > >> > > >> > > >> >> for
> > > > > >>> > >> > > >> > > >> >> > new
> > > > > >>> > >> > > >> > > >> >> > > > > > > > components
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into
> > > Bigtop.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list
> of
> > > it:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > >>> > >> > > >> > > >> >> > > >
> > > > > >>> > >> > > >> > > >> >> > >
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >>
> > > > > >>> > >> > > >> > > >>
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> >
> > > > > >>> > >> > > >>
> > > > > >>> > >> > >
> > > > > >>> > >> >
> > > > > >>> > >>
> > > > > >>> >
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems
> > to
> > > be
> > > > > >>> > standard
> > > > > >>> > >> ASF
> > > > > >>> > >> > > >> based
> > > > > >>> > >> > > >> > > >> >> > > requirements,
> > > > > >>> > >> > > >> > > >> >> > > > > but
> > > > > >>> > >> > > >> > > >> >> > > > > > > few
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > need
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software
> projects
> > > are
> > > > > >>> > expected
> > > > > >>> > >> to
> > > > > >>> > >> > be
> > > > > >>> > >> > > >> > > Licensed
> > > > > >>> > >> > > >> > > >> >> under
> > > > > >>> > >> > > >> > > >> >> > > > > Apache
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > License,
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and
> > their
> > > > > >>> > >> dependencies
> > > > > >>> > >> > are
> > > > > >>> > >> > > >> > > expected
> > > > > >>> > >> > > >> > > >> >> to be
> > > > > >>> > >> > > >> > > >> >> > > > > > > compatible
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > with
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is
> under
> > > ASL
> > > > > 2.0
> > > > > >>> but
> > > > > >>> > >> need
> > > > > >>> > >> > to
> > > > > >>> > >> > > >> > check
> > > > > >>> > >> > > >> > > if
> > > > > >>> > >> > > >> > > >> >> > > > > dependencies
> > > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with
> ASL
> > > 2.0.
> > > > > >>> This I
> > > > > >>> > >> guess
> > > > > >>> > >> > > >> would
> > > > > >>> > >> > > >> > > be a
> > > > > >>> > >> > > >> > > >> >> > > > > verification
> > > > > >>> > >> > > >> > > >> >> > > > > > > > check.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software
> projects
> > > are
> > > > > >>> > expected
> > > > > >>> > >> to
> > > > > >>> > >> > be
> > > > > >>> > >> > > >> > > >> compatible
> > > > > >>> > >> > > >> > > >> >> > with
> > > > > >>> > >> > > >> > > >> >> > > > all
> > > > > >>> > >> > > >> > > >> >> > > > > of
> > > > > >>> > >> > > >> > > >> >> > > > > > > the
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported
> platforms
> > > that
> > > > > >>> Bigtop
> > > > > >>> > >> > > >> distribution
> > > > > >>> > >> > > >> > > is
> > > > > >>> > >> > > >> > > >> >> > > targeting
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > > > verifying
> > > > > >>> > whether
> > > > > >>> > >> > our
> > > > > >>> > >> > > >> > > software
> > > > > >>> > >> > > >> > > >> >> runs
> > > > > >>> > >> > > >> > > >> >> > > fine
> > > > > >>> > >> > > >> > > >> >> > > > in
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
> > > > > >>> ubuntu-14.04
> > > > > >>> > >> > > debian-8
> > > > > >>> > >> > > >> > > >> >> > opensuse-13.2.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke
> tests
> > > that
> > > > > >>> should
> > > > > >>> > be
> > > > > >>> > >> > > added
> > > > > >>> > >> > > >> for
> > > > > >>> > >> > > >> > > >> >> > deployment.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the
> > > test
> > > > > >>> > artifacts
> > > > > >>> > >> > which
> > > > > >>> > >> > > >> goes
> > > > > >>> > >> > > >> > > >> beyond
> > > > > >>> > >> > > >> > > >> >> > > smoke
> > > > > >>> > >> > > >> > > >> >> > > > > test
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are
> > > > basically
> > > > > >>> the
> > > > > >>> > >> > > integration
> > > > > >>> > >> > > >> > > tests
> > > > > >>> > >> > > >> > > >> for
> > > > > >>> > >> > > >> > > >> >> > > > > > verification
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > after
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This
> is
> > a
> > > > soft
> > > > > >>> > >> > requirement,
> > > > > >>> > >> > > >> but
> > > > > >>> > >> > > >> > > aim
> > > > > >>> > >> > > >> > > >> is
> > > > > >>> > >> > > >> > > >> >> to
> > > > > >>> > >> > > >> > > >> >> > > > > achieve
> > > > > >>> > >> > > >> > > >> >> > > > > > > this
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > as
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have
> > > > > >>> explanation why
> > > > > >>> > >> not
> > > > > >>> > >> > to
> > > > > >>> > >> > > >> > > include.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any
> > from
> > > > the
> > > > > >>> link
> > > > > >>> > >> which
> > > > > >>> > >> > > >> > > explicitly
> > > > > >>> > >> > > >> > > >> >> needs
> > > > > >>> > >> > > >> > > >> >> > to
> > > > > >>> > >> > > >> > > >> >> > > > be
> > > > > >>> > >> > > >> > > >> >> > > > > > > > checked
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4,
> please
> > > let
> > > > > us
> > > > > >>> > know.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding
> Apex
> > > as
> > > > > >>> > component
> > > > > >>> > >> to
> > > > > >>> > >> > > >> Bigtop*
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the
> mail
> > > > > >>> archive of
> > > > > >>> > >> > Bigtop,
> > > > > >>> > >> > > >> it
> > > > > >>> > >> > > >> > was
> > > > > >>> > >> > > >> > > >> >> learnt
> > > > > >>> > >> > > >> > > >> >> > > that
> > > > > >>> > >> > > >> > > >> >> > > > > the
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community want to
> > see
> > > > the
> > > > > >>> > >> addition of
> > > > > >>> > >> > > new
> > > > > >>> > >> > > >> > > >> >> components
> > > > > >>> > >> > > >> > > >> >> > in
> > > > > >>> > >> > > >> > > >> >> > > > > > phases.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > Here
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > are
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > > > creating
> > > > > of
> > > > > >>> > >> package
> > > > > >>> > >> > > i.e.
> > > > > >>> > >> > > >> > rpm
> > > > > >>> > >> > > >> > > &
> > > > > >>> > >> > > >> > > >> deb
> > > > > >>> > >> > > >> > > >> >> > > files.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > > > > >>> documentations/READMEs,
> > > > > >>> > >> > LICENSE,
> > > > > >>> > >> > > >> > > >> DISCLAMER,
> > > > > >>> > >> > > >> > > >> >> > NOTES
> > > > > >>> > >> > > >> > > >> >> > > > etc
> > > > > >>> > >> > > >> > > >> >> > > > > if
> > > > > >>> > >> > > >> > > >> >> > > > > > > any
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> > > documentation
> > > > > that
> > > > > >>> > need
> > > > > >>> > >> to
> > > > > >>> > >> > be
> > > > > >>> > >> > > >> > added
> > > > > >>> > >> > > >> > > to
> > > > > >>> > >> > > >> > > >> >> > > > > distribution
> > > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > > >>> > >> > > >> > > >> >> > > > > > > > our
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license
> > > > > >>> information of
> > > > > >>> > >> > > >> > dependencies
> > > > > >>> > >> > > >> > > >> >> required
> > > > > >>> > >> > > >> > > >> >> > > to
> > > > > >>> > >> > > >> > > >> >> > > > be
> > > > > >>> > >> > > >> > > >> >> > > > > > > added
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at
> > > very
> > > > > >>> least)
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke
> > > test
> > > > > for
> > > > > >>> > >> > packaging.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes
> > for
> > > > > >>> automatic
> > > > > >>> > >> > > >> deployment
> > > > > >>> > >> > > >> > and
> > > > > >>> > >> > > >> > > >> >> > > > configuration
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet
> > > recipes
> > > > > for
> > > > > >>> > >> > automatic
> > > > > >>> > >> > > >> > > >> deployment
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration
> tests
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For
> > verification
> > > > of
> > > > > >>> > >> > deployments.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license
> > clearance:
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle
> rat'
> > > to
> > > > > make
> > > > > >>> > sure
> > > > > >>> > >> all
> > > > > >>> > >> > > new
> > > > > >>> > >> > > >> > > stuff
> > > > > >>> > >> > > >> > > >> is
> > > > > >>> > >> > > >> > > >> >> > > > compliant
> > > > > >>> > >> > > >> > > >> >> > > > > > > with
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license
> > requirements.
> > > If
> > > > > >>> you add
> > > > > >>> > >> code
> > > > > >>> > >> > > >> > licenses
> > > > > >>> > >> > > >> > > >> >> under
> > > > > >>> > >> > > >> > > >> >> > > > > different
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need
> to
> > be
> > > > > >>> listed in
> > > > > >>> > >> the
> > > > > >>> > >> > > >> NOTICE.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your
> > > > thoughts
> > > > > >>> on
> > > > > >>> > the
> > > > > >>> > >> > > >> approach.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
> > > > corresponding
> > > > > >>> > >> > communication
> > > > > >>> > >> > > >> on
> > > > > >>> > >> > > >> > > >> bigtop
> > > > > >>> > >> > > >> > > >> >> > > mailing
> > > > > >>> > >> > > >> > > >> >> > > > > list
> > > > > >>> > >> > > >> > > >> >> > > > > > > as
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some
> > specific
> > > > > >>> > >> > > >> questions/suggestions
> > > > > >>> > >> > > >> > > >> >> related to
> > > > > >>> > >> > > >> > > >> >> > > > what
> > > > > >>> > >> > > >> > > >> >> > > > > > > should
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > be
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the
> > package
> > > > and
> > > > > >>> what
> > > > > >>> > >> > should
> > > > > >>> > >> > > be
> > > > > >>> > >> > > >> > the
> > > > > >>> > >> > > >> > > >> smoke
> > > > > >>> > >> > > >> > > >> >> > > tests,
> > > > > >>> > >> > > >> > > >> >> > > > > but
> > > > > >>> > >> > > >> > > >> >> > > > > > > in
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > the
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not
> > having
> > > > too
> > > > > >>> much
> > > > > >>> > >> > content
> > > > > >>> > >> > > >> > here,
> > > > > >>> > >> > > >> > > >> we'll
> > > > > >>> > >> > > >> > > >> >> > put
> > > > > >>> > >> > > >> > > >> >> > > > the
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > questions
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in
> > this
> > > > > >>> > mailthread.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > > >
> > > > > >>> > >> > > >> > > >> >> > > > >
> > > > > >>> > >> > > >> > > >> >> > > >
> > > > > >>> > >> > > >> > > >> >> > >
> > > > > >>> > >> > > >> > > >> >> >
> > > > > >>> > >> > > >> > > >> >>
> > > > > >>> > >> > > >> > > >> >
> > > > > >>> > >> > > >> > > >> >
> > > > > >>> > >> > > >> > > >>
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > > >
> > > > > >>> > >> > > >> > >
> > > > > >>> > >> > > >> >
> > > > > >>> > >> > > >>
> > > > > >>> > >> > > >
> > > > > >>> > >> > > >
> > > > > >>> > >> > >
> > > > > >>> > >> >
> > > > > >>> > >>
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Justin Mclean <ju...@me.com>]

Hi,

> BSD - jline-2.11.jar, kryo-2.24.0.jar, minlog-1.2.jar
> MIT - mbassador-1.1.9.jar

You shod also look inside there jars and see what they bindle there may be other differently licensed software contained inside.

Depending on the content LICENSE and NOTICE may have to change as well. [1][2]

Thanks,
Justin

1. http://www.apache.org/dev/licensing-howto.html#guiding-principle
2.http://www.apache.org/dev/licensing-howto.html#binary

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

No worries David. I've updated the manpage.

Just an update: apex bigtop packaging for rpm+deb is done. Here is the
branch on which I'm working if someone wants to look at:
https://github.com/chinmaykolhatkar/bigtop/blob/BIGTOP-2314

Only the 3rd questions below is the question that is left to be answered.
Once this is resolved, I'll create the pull request in bigtop.

Following questions are related to licenses and inclusion of certain files
in apex rpm+deb:
1. I'll be included LICENSE, DISCLAIMER, NOTICE from top folder of core
repo in our rpm and deb package. Those will be present in standard location
i.e. /usr/lib/apex/
Please let me know otherwise.

2. Most of our runtime dependencies has ASF license, but certain libraries
has MIT and BSD licenses, hence I've included those licenses too in rpm &
deb package in same location as above.
Please let me know otherwise.

BSD - jline-2.11.jar, kryo-2.24.0.jar, minlog-1.2.jar
MIT - mbassador-1.1.9.jar

3. There are 2 jars (jersey-apache-client4-1.9.jar, jersey-client-1.9.jar)
which has duel license of CDDL + GPLv2 with classpath exception:
https://jersey.java.net/license.html
I'm not an expert here. Hence can someone let me know whether these
libraries is OK to be packaged in rpm+deb of apex?

If yes, should we also include that license in our package?

Thanks,
Chinmay.



On Thu, Feb 18, 2016 at 11:35 PM, David Yan <da...@datatorrent.com> wrote:

> I should've mentioned that I had a pull request that was merged yesterday
> that corrected the statement:
>
> https://github.com/DataTorrent/docs/pull/45
>
> On Thu, Feb 18, 2016 at 12:07 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > In following statement, can you replace dtGateway with ApplicationMaster
> > and Yarn. or may be remove this statement entirely.
> > apex is a wrapper around the REST API provided by dtGatway,
> >
> > - Tushar.
> >
> >
> >
> >
> > On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > I've attached man page for apex in Jira. Here is the link to it:
> > > https://issues.apache.org/jira/secure/attachment/12788403/apex.1
> > >
> > > Please view it by: man -l apex.1
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <
> tushar@datatorrent.com>
> > > wrote:
> > >
> > > > Hi Chinmay,
> > > >
> > > > I am not able to find the attachment, may be attachments are not
> > allowed
> > > in
> > > > the mailing list. Can you attach the man page to the Jira?
> > > >
> > > > - Tushar.
> > > >
> > > >
> > > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <
> chinmay@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I've attached man page file for the purpose of bigtop integration.
> > > > > Please have a look and let me know if there any any feedback about
> > it.
> > > > >
> > > > > You can see the man page content of this file with:
> > > > > man -l apex.1
> > > > >
> > > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> > > chinmay@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Thanks. Will follow bigtop recommendation. i.e. remove
> > "-incubating".
> > > > >>
> > > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
> > thomas@datatorrent.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >>> Please follow the recommendation and continue discussion on the
> > > Bigtop
> > > > >>> list.
> > > > >>>
> > > > >>> --
> > > > >>> sent from mobile
> > > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <chinmay@apache.org
> >
> > > > wrote:
> > > > >>>
> > > > >>> > I got a recommendation from bigtop community to drop
> > "-incubating"
> > > > from
> > > > >>> > version.
> > > > >>> >
> > > > >>> > Here is the mail thread:
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> > > > >>> >
> > > > >>> > Please let me know if removing "-incubating" is fine.
> > > > >>> >
> > > > >>> > Thanks,
> > > > >>> > Chinmay.
> > > > >>> >
> > > > >>> >
> > > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> > > > chinmay@apache.org
> > > > >>> >
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> > > I've already started communication there.
> > > > >>> > >
> > > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> > > > >>> thomas@datatorrent.com>
> > > > >>> > > wrote:
> > > > >>> > >
> > > > >>> > >> This is a question you should ask on the Bigtop mailing
> list.
> > > > >>> > >>
> > > > >>> > >> --
> > > > >>> > >> sent from mobile
> > > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <
> prad@apache.org>
> > > > >>> wrote:
> > > > >>> > >>
> > > > >>> > >> > Yes. In case of RPM, the only restriction on version is
> that
> > > it
> > > > >>> can
> > > > >>> > not
> > > > >>> > >> > contain '-'.
> > > > >>> > >> > However, as a workaround, one can use 'incubating' as a
> part
> > > of
> > > > >>> > >> %{release}.
> > > > >>> > >> >
> > > > >>> > >> > Version: 3.3.0
> > > > >>> > >> > Release: incubating
> > > > >>> > >> >
> > > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> > > > >>> > >> >
> > > > >>> > >> > It is the same practice followed for pre-release packages
> > e.g.
> > > > >>> with
> > > > >>> > rcX,
> > > > >>> > >> > alphaX, betaX being used in %{release}.
> > > > >>> > >> >
> > > > >>> > >> > Regards,
> > > > >>> > >> > Pradeep A. Dalvi
> > > > >>> > >> >
> > > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> > > > >>> > chinmay@apache.org>
> > > > >>> > >> > wrote:
> > > > >>> > >> >
> > > > >>> > >> > > Hi Everyone,
> > > > >>> > >> > >
> > > > >>> > >> > > In one of the previous mails in this thread, we decided
> > that
> > > > >>> version
> > > > >>> > >> of
> > > > >>> > >> > the
> > > > >>> > >> > > package will be "3.3.0-incubating".
> > > > >>> > >> > > The problem that I'm facing is in RPM base version "-"
> is
> > > not
> > > > >>> > allowed.
> > > > >>> > >> > >
> > > > >>> > >> > > Can we have package version to "3.3.0"?
> > > > >>> > >> > > i.e. the packages created will be as follows:
> > > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> > > > >>> > >> > >
> > > > >>> > >> > > There is another integration for Zepplin in Bigtop which
> > is
> > > > >>> > incubating
> > > > >>> > >> > and
> > > > >>> > >> > > is having base version as "0.5.6".
> > > > >>> > >> > >
> > > > >>> > >> > > Please let me know your thoughts.
> > > > >>> > >> > >
> > > > >>> > >> > > Thanks,
> > > > >>> > >> > > Chinmay.
> > > > >>> > >> > >
> > > > >>> > >> > >
> > > > >>> > >> > >
> > > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> > > > >>> > >> chinmay@apache.org>
> > > > >>> > >> > > wrote:
> > > > >>> > >> > >
> > > > >>> > >> > > > Thanks everyone for the feedback.
> > > > >>> > >> > > >
> > > > >>> > >> > > > Also as Ram and David mentioned, I'll start creating
> the
> > > man
> > > > >>> page
> > > > >>> > as
> > > > >>> > >> > > well.
> > > > >>> > >> > > >
> > > > >>> > >> > > > Thanks,
> > > > >>> > >> > > > Chinmay.
> > > > >>> > >> > > >
> > > > >>> > >> > > >
> > > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> > > > >>> > >> thomas@datatorrent.com>
> > > > >>> > >> > > > wrote:
> > > > >>> > >> > > >
> > > > >>> > >> > > >> I think we already agreed that we will retain the
> > current
> > > > >>> script
> > > > >>> > >> name
> > > > >>> > >> > in
> > > > >>> > >> > > >> this bigtop iteration. There is a ticket for
> renaming,
> > > add
> > > > >>> you
> > > > >>> > >> > > suggestions
> > > > >>> > >> > > >> there.
> > > > >>> > >> > > >>
> > > > >>> > >> > > >> --
> > > > >>> > >> > > >> sent from mobile
> > > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > > > >>> > sandesh@datatorrent.com>
> > > > >>> > >> > > wrote:
> > > > >>> > >> > > >>
> > > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start the
> > > transition
> > > > >>> > >> process by
> > > > >>> > >> > > >> making
> > > > >>> > >> > > >> > dtcli as the symlink to apex cli.
> > > > >>> > >> > > >> >
> > > > >>> > >> > > >> > Renaming is a breaking change should be done during
> > > major
> > > > >>> > version
> > > > >>> > >> > > number
> > > > >>> > >> > > >> > change.
> > > > >>> > >> > > >> >
> > > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> > > > >>> > david@datatorrent.com
> > > > >>> > >> >
> > > > >>> > >> > > >> wrote:
> > > > >>> > >> > > >> >
> > > > >>> > >> > > >> > > Apologies for hitting the "Send" button too
> early.
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > What is the scope of the name change?  If we
> simply
> > > > have
> > > > >>> a
> > > > >>> > >> symlink
> > > > >>> > >> > > >> from
> > > > >>> > >> > > >> > > apex to dtcli, then it should have no backward
> > > > >>> compatibility
> > > > >>> > >> > problem
> > > > >>> > >> > > >> at
> > > > >>> > >> > > >> > > all.
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > But the proper name change would involve:
> > > > >>> > >> > > >> > > - Change the name of DTCli class to ApexCli
> > > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> > > > >>> > >> > > >> > > - Change the initial greeting when running the
> CLI
> > > from
> > > > >>> > >> something
> > > > >>> > >> > > like
> > > > >>> > >> > > >> > "DT
> > > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > The above changes may break existing scripts that
> > > rely
> > > > >>> on the
> > > > >>> > >> the
> > > > >>> > >> > > >> string
> > > > >>> > >> > > >> > > "DT".
> > > > >>> > >> > > >> > > But if folks are okay with this potential
> problem,
> > I
> > > am
> > > > >>> okay
> > > > >>> > >> with
> > > > >>> > >> > > the
> > > > >>> > >> > > >> > name
> > > > >>> > >> > > >> > > change.
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > For the man page, we can start with the content
> of
> > > this
> > > > >>> page:
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > Plus the command line options when doing dtcli
> -h:
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > usage: DTCli
> > > > >>> > >> > > >> > >  -e <arg>    Commands are read from the argument
> > > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at all time
> > > > >>> > >> > > >> > >  -h          Print this help
> > > > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos principal
> > > > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> > > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> > > > >>> > >> > > >> > >  -r          JSON Raw mode
> > > > >>> > >> > > >> > >  -v          Verbose mode level 1
> > > > >>> > >> > > >> > >  -vv         Verbose mode level 2
> > > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> > > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > David
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> > > > >>> > >> david@datatorrent.com
> > > > >>> > >> > >
> > > > >>> > >> > > >> > wrote:
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > > What is the scope of the name change?  If we
> > simply
> > > > >>> have a
> > > > >>> > >> > symlink
> > > > >>> > >> > > >> from
> > > > >>> > >> > > >> > > > apex to dtcli, then it should have no backward
> > > > >>> > compatibility
> > > > >>> > >> > > >> problem at
> > > > >>> > >> > > >> > > all.
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > > But the proper name change would involve:
> > > > >>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
> > > > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> > > > >>> > >> > > >> > > > - Change the initial greeting when running the
> > CLI
> > > > from
> > > > >>> > >> > something
> > > > >>> > >> > > >> like
> > > > >>> > >> > > >> > > "DT
> > > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > > The above changes may break existing scripts
> that
> > > > rely
> > > > >>> on
> > > > >>> > the
> > > > >>> > >> > the
> > > > >>> > >> > > >> > string
> > > > >>> > >> > > >> > > > "DT".
> > > > >>> > >> > > >> > > > But if folks are okay with this potential
> > problem,
> > > I
> > > > am
> > > > >>> > okay
> > > > >>> > >> > with
> > > > >>> > >> > > >> the
> > > > >>> > >> > > >> > > name
> > > > >>> > >> > > >> > > > change.
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > > For the man page, we can start with the content
> > of
> > > > this
> > > > >>> > page:
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay
> > Kolhatkar
> > > <
> > > > >>> > >> > > >> chinmay@apache.org
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> > > > wrote:
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >> Here is a suggestion I have related to name
> > > change:
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> 1. While packaging we change the name of
> "dtcli"
> > > > >>> script to
> > > > >>> > >> > > "apex".
> > > > >>> > >> > > >> > This
> > > > >>> > >> > > >> > > is
> > > > >>> > >> > > >> > > >> done so that in future when we change the name
> > to
> > > > >>> "apex",
> > > > >>> > >> users
> > > > >>> > >> > > of
> > > > >>> > >> > > >> > > bigtop
> > > > >>> > >> > > >> > > >> apex don't have to transition much.
> > > > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli which
> > points
> > > > to
> > > > >>> apex
> > > > >>> > >> > > script.
> > > > >>> > >> > > >> > This
> > > > >>> > >> > > >> > > is
> > > > >>> > >> > > >> > > >> for backward compatibility.
> > > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first
> > > > integration
> > > > >>> in
> > > > >>> > >> > bigtop.
> > > > >>> > >> > > >> This
> > > > >>> > >> > > >> > is
> > > > >>> > >> > > >> > > >> in
> > > > >>> > >> > > >> > > >> the interest of not changing the content of
> > dtcli
> > > or
> > > > >>> apex
> > > > >>> > >> > script.
> > > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these
> > naming
> > > > >>> related
> > > > >>> > >> > > changes
> > > > >>> > >> > > >> and
> > > > >>> > >> > > >> > > >> update bigtop repository later to remove
> > > references
> > > > to
> > > > >>> > dtcli
> > > > >>> > >> > all
> > > > >>> > >> > > >> > > together.
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> Please share your thoughts on above approach.
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> Also, please share what could be the man page
> > > > content
> > > > >>> for
> > > > >>> > >> > "apex".
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> Thanks,
> > > > >>> > >> > > >> > > >> Chinmay.
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay
> > > Kolhatkar <
> > > > >>> > >> > > >> > chinmay@apache.org>
> > > > >>> > >> > > >> > > >> wrote:
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >> > Yes.. It'll be build from source tar
> > downloaded
> > > > >>> from one
> > > > >>> > >> of
> > > > >>> > >> > the
> > > > >>> > >> > > >> > apache
> > > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced
> from
> > > > dtcli.
> > > > >>> > This
> > > > >>> > >> > > leaves
> > > > >>> > >> > > >> us
> > > > >>> > >> > > >> > 3
> > > > >>> > >> > > >> > > >> > options:
> > > > >>> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli
> > would
> > > > >>> need a
> > > > >>> > >> > change
> > > > >>> > >> > > >> while
> > > > >>> > >> > > >> > > >> > packaging.
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> > But if we want to use dtcli as it is, then
> we
> > > > would
> > > > >>> need
> > > > >>> > >> > > >> dt-env.sh
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas
> Weise
> > <
> > > > >>> > >> > > >> > thomas@datatorrent.com
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >> > wrote:
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> >> Looks good overall, though there shouldn't
> be
> > > any
> > > > >>> > >> dt-env.sh
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >> What will the bigtop package be built from,
> > the
> > > > >>> source
> > > > >>> > >> tar?
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay
> > > > Kolhatkar
> > > > >>> <
> > > > >>> > >> > > >> > > chinmay@apache.org
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> >> wrote:
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >> > Hi Thomas,
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > First, we're not changing the name
> > anywhere.
> > > > >>> We'll
> > > > >>> > >> follow
> > > > >>> > >> > > what
> > > > >>> > >> > > >> > > >> >> currently is
> > > > >>> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating
> > > version.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory
> > > structure
> > > > >>> below
> > > > >>> > >> which
> > > > >>> > >> > > is
> > > > >>> > >> > > >> > > inline
> > > > >>> > >> > > >> > > >> >> with
> > > > >>> > >> > > >> > > >> >> > other existing integrations with bigtop.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > The need for each file is as follows:
> > > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> > > > >>> > >> > > >> engine/src/main/scripts/
> > > > >>> > >> > > >> > > of
> > > > >>> > >> > > >> > > >> >> source
> > > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from
> > > dtcli.
> > > > >>> This
> > > > >>> > >> > > searches
> > > > >>> > >> > > >> for
> > > > >>> > >> > > >> > > >> hadoop
> > > > >>> > >> > > >> > > >> >> > binary path and exports an env variable
> for
> > > > >>> dtcli to
> > > > >>> > >> use
> > > > >>> > >> > it.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list
> is
> > > > >>> extracted
> > > > >>> > >> in a
> > > > >>> > >> > > >> similar
> > > > >>> > >> > > >> > > >> way to
> > > > >>> > >> > > >> > > >> >> > how DT Community Edition finds dependency
> > > jars.
> > > > >>> > >> > > >> > > >> >> > Please note that this is the first
> > iteration
> > > > >>> list of
> > > > >>> > >> jars.
> > > > >>> > >> > > I'm
> > > > >>> > >> > > >> > > >> trying to
> > > > >>> > >> > > >> > > >> >> > narrow this down to only those which are
> > > really
> > > > >>> > >> required.
> > > > >>> > >> > > >> > > >> >> > The test that I'm running to check if
> dtcli
> > > > runs
> > > > >>> fine
> > > > >>> > >> with
> > > > >>> > >> > > >> given
> > > > >>> > >> > > >> > > >> >> dependency
> > > > >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi
> > > demo.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > Above is the directory structure which is
> > > > >>> required
> > > > >>> > for
> > > > >>> > >> CLI
> > > > >>> > >> > > to
> > > > >>> > >> > > >> > work.
> > > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned
> > are
> > > > some
> > > > >>> > >> > additional
> > > > >>> > >> > > >> > files
> > > > >>> > >> > > >> > > >> which
> > > > >>> > >> > > >> > > >> >> > makes the like of administrator easier.
> For
> > > eg.
> > > > >>> > >> /etc/skel
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > Please let me know if this is inline with
> > > what
> > > > >>> you're
> > > > >>> > >> > > >> thinking.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > Thanks,
> > > > >>> > >> > > >> > > >> >> > Chinmay.
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > .
> > > > >>> > >> > > >> > > >> >> > |-- bin
> > > > >>> > >> > > >> > > >> >> > |   `-- dtcli
> > > > >>> > >> > > >> > > >> >> > |-- conf
> > > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > > > >>> > >> > > >> > > >> >> > `-- lib
> > > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > > > >>> > >> > > >> > > >> >> >     |--
> > > apex-bufferserver-3.3.0-incubating.jar
> > > > >>> > >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > > > >>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > > > >>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > > > >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > > > >>> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > > >>> > >> > > >> > > >> >> >     |--
> > jetty-websocket-8.1.10.v20130312.jar
> > > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > > > >>> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > > > >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas
> > > Weise <
> > > > >>> > >> > > >> > > >> thomas@datatorrent.com>
> > > > >>> > >> > > >> > > >> >> > wrote:
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >> > > Chinmay,
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> > > Before discussing where to put the
> files,
> > > > let's
> > > > >>> > make
> > > > >>> > >> > sure
> > > > >>> > >> > > >> they
> > > > >>> > >> > > >> > > are
> > > > >>> > >> > > >> > > >> >> really
> > > > >>> > >> > > >> > > >> >> > > needed for the operation of the CLI. As
> > for
> > > > >>> names,
> > > > >>> > >> > > anything
> > > > >>> > >> > > >> new
> > > > >>> > >> > > >> > > >> needs
> > > > >>> > >> > > >> > > >> >> to
> > > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and should
> > follow
> > > > >>> common
> > > > >>> > >> > > >> conventions,
> > > > >>> > >> > > >> > > >> >> especially
> > > > >>> > >> > > >> > > >> >> > > in Bigtop where there are many existing
> > > > >>> > integrations
> > > > >>> > >> to
> > > > >>> > >> > > look
> > > > >>> > >> > > >> > at.
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> > > Thanks,
> > > > >>> > >> > > >> > > >> >> > > Thomas
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM,
> Chinmay
> > > > >>> Kolhatkar
> > > > >>> > <
> > > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > > > >>> > >> > > >> > > >> >> > > wrote:
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep and
> > Aniruddha.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to change
> > the
> > > > >>> dtcli
> > > > >>> > >> > > >> considering it
> > > > >>> > >> > > >> > > >> works
> > > > >>> > >> > > >> > > >> >> > with
> > > > >>> > >> > > >> > > >> >> > > DT
> > > > >>> > >> > > >> > > >> >> > > > Community edition. We can keep the
> > > > directory
> > > > >>> > >> structure
> > > > >>> > >> > > >> > similar
> > > > >>> > >> > > >> > > to
> > > > >>> > >> > > >> > > >> >> that.
> > > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which
> contains
> > > > >>> > information
> > > > >>> > >> > > >> required
> > > > >>> > >> > > >> > for
> > > > >>> > >> > > >> > > >> >> dtcli
> > > > >>> > >> > > >> > > >> >> > to
> > > > >>> > >> > > >> > > >> >> > > > launch.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop community
> > > that
> > > > >>> > whether
> > > > >>> > >> > they
> > > > >>> > >> > > >> > > >> facilitate
> > > > >>> > >> > > >> > > >> >> the
> > > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs before user
> > is
> > > > >>> created.
> > > > >>> > >> In
> > > > >>> > >> > > >> either
> > > > >>> > >> > > >> > > case,
> > > > >>> > >> > > >> > > >> >> > current
> > > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home
> > > > directory.
> > > > >>> > Also
> > > > >>> > >> > > before
> > > > >>> > >> > > >> > > putting
> > > > >>> > >> > > >> > > >> >> > > anything
> > > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define what
> are
> > > the
> > > > >>> > default
> > > > >>> > >> > > >> contents
> > > > >>> > >> > > >> > > that
> > > > >>> > >> > > >> > > >> >> > should
> > > > >>> > >> > > >> > > >> >> > > go
> > > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no
> > > defaults,
> > > > >>> > >> probably we
> > > > >>> > >> > > >> should
> > > > >>> > >> > > >> > > not
> > > > >>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks
> nice.
> > > > This
> > > > >>> way
> > > > >>> > >> > > contents
> > > > >>> > >> > > >> of
> > > > >>> > >> > > >> > > >> >> dt-env.sh
> > > > >>> > >> > > >> > > >> >> > > are
> > > > >>> > >> > > >> > > >> >> > > > present as env variables. But I see a
> > > catch
> > > > >>> > there.
> > > > >>> > >> > > Adding
> > > > >>> > >> > > >> > > >> dt-env.sh
> > > > >>> > >> > > >> > > >> >> to
> > > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the
> > > variables
> > > > >>> > >> available
> > > > >>> > >> > at
> > > > >>> > >> > > >> > > runtime
> > > > >>> > >> > > >> > > >> all
> > > > >>> > >> > > >> > > >> >> > the
> > > > >>> > >> > > >> > > >> >> > > > time. I feel a little skeptical about
> > > that.
> > > > >>> > Maybe a
> > > > >>> > >> > > >> possible
> > > > >>> > >> > > >> > > >> >> collision
> > > > >>> > >> > > >> > > >> >> > > can
> > > > >>> > >> > > >> > > >> >> > > > occur with other application vars.
> > > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
> > > > >>> > >> > "../conf/dt-env.sh"
> > > > >>> > >> > > >> and
> > > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables
> > are
> > > > >>> anyway
> > > > >>> > >> > > >> available.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is
> > > > >>> happening
> > > > >>> > at
> > > > >>> > >> > > Bigtop
> > > > >>> > >> > > >> > > >> >> > > Jira/repository.
> > > > >>> > >> > > >> > > >> >> > > > Packaging related code usually goes
> > > there.
> > > > >>> > (That's
> > > > >>> > >> > what
> > > > >>> > >> > > >> all
> > > > >>> > >> > > >> > the
> > > > >>> > >> > > >> > > >> >> > > components
> > > > >>> > >> > > >> > > >> >> > > > in bigtop does).
> > > > >>> > >> > > >> > > >> >> > > > Having said that, once a PR is
> created,
> > > > I'll
> > > > >>> be
> > > > >>> > >> > sharing
> > > > >>> > >> > > >> the
> > > > >>> > >> > > >> > > link
> > > > >>> > >> > > >> > > >> of
> > > > >>> > >> > > >> > > >> >> PR
> > > > >>> > >> > > >> > > >> >> > > > here, so that apex community as well
> > can
> > > > >>> review
> > > > >>> > it.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > Thanks,
> > > > >>> > >> > > >> > > >> >> > > > Chinmay.
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM,
> > > Aniruddha
> > > > >>> > >> Thombare <
> > > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> > > > >>> > >> > > >> > > >> >> > > > > We may need to iron out details
> about
> > > > exact
> > > > >>> > paths
> > > > >>> > >> > etc.
> > > > >>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR
> > > comments.
> > > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep
> > A.
> > > > >>> Dalvi <
> > > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > > > >>> > >> > > >> > > >> >> > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM,
> > > > Chinmay
> > > > >>> > >> > Kolhatkar <
> > > > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > > > >>> > >> > > >> > > >> >> > > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar
> present
> > in
> > > > DT
> > > > >>> > >> community
> > > > >>> > >> > > >> > edition
> > > > >>> > >> > > >> > > >> will
> > > > >>> > >> > > >> > > >> >> be
> > > > >>> > >> > > >> > > >> >> > > > > included
> > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I
> found
> > > from
> > > > >>> > common
> > > > >>> > >> > > >> > dependencies
> > > > >>> > >> > > >> > > >> >> between
> > > > >>> > >> > > >> > > >> >> > > of
> > > > >>> > >> > > >> > > >> >> > > > > apex
> > > > >>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> > > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > jetty-websocket-8.1.10.v20130312.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some
> > > tests
> > > > do
> > > > >>> > check
> > > > >>> > >> > that
> > > > >>> > >> > > >> > dtcli
> > > > >>> > >> > > >> > > >> works
> > > > >>> > >> > > >> > > >> >> > > > properly
> > > > >>> > >> > > >> > > >> >> > > > > > for
> > > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps
> with
> > > > only
> > > > >>> > these
> > > > >>> > >> > > >> libraries
> > > > >>> > >> > > >> > > >> >> present in
> > > > >>> > >> > > >> > > >> >> > > > > > isolation
> > > > >>> > >> > > >> > > >> >> > > > > > > without dependency on local m2.
> > > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there are
> > unwanted
> > > > >>> jars
> > > > >>> > >> which
> > > > >>> > >> > are
> > > > >>> > >> > > >> used
> > > > >>> > >> > > >> > > for
> > > > >>> > >> > > >> > > >> >> > compile
> > > > >>> > >> > > >> > > >> >> > > > > time
> > > > >>> > >> > > >> > > >> >> > > > > > > dependency and not runtime in
> > above
> > > > >>> list
> > > > >>> > >> which I
> > > > >>> > >> > > can
> > > > >>> > >> > > >> > drop
> > > > >>> > >> > > >> > > >> to
> > > > >>> > >> > > >> > > >> >> keep
> > > > >>> > >> > > >> > > >> >> > > > > package
> > > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if
> > all
> > > > of
> > > > >>> the
> > > > >>> > >> > > >> dt-env.sh is
> > > > >>> > >> > > >> > > >> >> required.
> > > > >>> > >> > > >> > > >> >> > > > Only
> > > > >>> > >> > > >> > > >> >> > > > > > > required values I see in it are
> > > > >>> classpath.
> > > > >>> > >> and
> > > > >>> > >> > > >> dtcli is
> > > > >>> > >> > > >> > > >> >> capable
> > > > >>> > >> > > >> > > >> >> > of
> > > > >>> > >> > > >> > > >> >> > > > > > building
> > > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll
> have
> > > to
> > > > >>> see
> > > > >>> > how
> > > > >>> > >> > dtcli
> > > > >>> > >> > > >> can
> > > > >>> > >> > > >> > > find
> > > > >>> > >> > > >> > > >> >> the
> > > > >>> > >> > > >> > > >> >> > jar
> > > > >>> > >> > > >> > > >> >> > > > > path,
> > > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf
> > files.
> > > If
> > > > >>> we
> > > > >>> > >> need a
> > > > >>> > >> > > >> change
> > > > >>> > >> > > >> > in
> > > > >>> > >> > > >> > > >> >> dtcli,
> > > > >>> > >> > > >> > > >> >> > > then
> > > > >>> > >> > > >> > > >> >> > > > > > > community's opinion is required
> > for
> > > > >>> whether
> > > > >>> > >> > dtcli
> > > > >>> > >> > > >> > should
> > > > >>> > >> > > >> > > >> >> change
> > > > >>> > >> > > >> > > >> >> > in
> > > > >>> > >> > > >> > > >> >> > > > our
> > > > >>> > >> > > >> > > >> >> > > > > > repo
> > > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with required
> > > > changes
> > > > >>> > >> exist in
> > > > >>> > >> > > >> bigtop
> > > > >>> > >> > > >> > > >> repo
> > > > >>> > >> > > >> > > >> >> > until
> > > > >>> > >> > > >> > > >> >> > > we
> > > > >>> > >> > > >> > > >> >> > > > > > make
> > > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > Can we have these file paths set
> > > using
> > > > >>> > >> environment
> > > > >>> > >> > > >> > > variables
> > > > >>> > >> > > >> > > >> >> with
> > > > >>> > >> > > >> > > >> >> > > some
> > > > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> > > > >>> > >> > > >> > > >> >> > > > > > Then we can set such params in
> > > > dt-env.sh.
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this
> > file
> > > > >>> > different
> > > > >>> > >> > from
> > > > >>> > >> > > >> > > >> >> dt-site.xml?
> > > > >>> > >> > > >> > > >> >> > Not
> > > > >>> > >> > > >> > > >> >> > > > > sure
> > > > >>> > >> > > >> > > >> >> > > > > > if
> > > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help.
> > > > Files &
> > > > >>> > Dirs
> > > > >>> > >> > from
> > > > >>> > >> > > >> > > /etc/skel
> > > > >>> > >> > > >> > > >> >> are
> > > > >>> > >> > > >> > > >> >> > > > copied
> > > > >>> > >> > > >> > > >> >> > > > > to
> > > > >>> > >> > > >> > > >> >> > > > > > > home of new user when useradd
> > > program
> > > > >>> is
> > > > >>> > >> called.
> > > > >>> > >> > > But
> > > > >>> > >> > > >> > for
> > > > >>> > >> > > >> > > >> >> existing
> > > > >>> > >> > > >> > > >> >> > > > users
> > > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any use.
> > Moreover,
> > > > >>> dtcli
> > > > >>> > >> > creates
> > > > >>> > >> > > >> > ~/.dt/
> > > > >>> > >> > > >> > > on
> > > > >>> > >> > > >> > > >> >> the
> > > > >>> > >> > > >> > > >> >> > fly
> > > > >>> > >> > > >> > > >> >> > > > if
> > > > >>> > >> > > >> > > >> >> > > > > > not
> > > > >>> > >> > > >> > > >> >> > > > > > > encountered for the first time.
> > > Again
> > > > >>> > >> correct me
> > > > >>> > >> > > if
> > > > >>> > >> > > >> I'm
> > > > >>> > >> > > >> > > >> wrong.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right. However this
> > > would
> > > > >>> > >> probably be
> > > > >>> > >> > > >> > > necessary
> > > > >>> > >> > > >> > > >> >> step
> > > > >>> > >> > > >> > > >> >> > for
> > > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get
> > > installed
> > > > >>> > during
> > > > >>> > >> OS
> > > > >>> > >> > > >> install
> > > > >>> > >> > > >> > > and
> > > > >>> > >> > > >> > > >> >> > before
> > > > >>> > >> > > >> > > >> >> > > > user
> > > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc &
> bash_profile
> > > ->
> > > > >>> I'm
> > > > >>> > not
> > > > >>> > >> > sure
> > > > >>> > >> > > >> its
> > > > >>> > >> > > >> > the
> > > > >>> > >> > > >> > > >> best
> > > > >>> > >> > > >> > > >> >> > > idea.
> > > > >>> > >> > > >> > > >> >> > > > > > We'll
> > > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location
> > which
> > > is
> > > > >>> by
> > > > >>> > >> default
> > > > >>> > >> > > >> > present
> > > > >>> > >> > > >> > > in
> > > > >>> > >> > > >> > > >> >> path
> > > > >>> > >> > > >> > > >> >> > > i.e.
> > > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > For other env variables specific
> to
> > > > apex,
> > > > >>> > I'll
> > > > >>> > >> > > prefer
> > > > >>> > >> > > >> to
> > > > >>> > >> > > >> > > use
> > > > >>> > >> > > >> > > >> >> > > dt-env.sh
> > > > >>> > >> > > >> > > >> >> > > > > and
> > > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather than
> > > > changing
> > > > >>> > >> bashrc
> > > > >>> > >> > > >> etc...
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
> > > > >>> /etc/profile.d/
> > > > >>> > >> shall
> > > > >>> > >> > > be
> > > > >>> > >> > > >> the
> > > > >>> > >> > > >> > > >> >> cleaner
> > > > >>> > >> > > >> > > >> >> > way
> > > > >>> > >> > > >> > > >> >> > > > to
> > > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > Thanks,
> > > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07
> PM,
> > > > >>> Aniruddha
> > > > >>> > >> > > Thombare
> > > > >>> > >> > > >> <
> > > > >>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com>
> > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > We need to consider
> following:
> > > > >>> > >> > > >> > > >> >> > > > > > > > System wide default config
> > files
> > > > can
> > > > >>> be
> > > > >>> > >> > located
> > > > >>> > >> > > at
> > > > >>> > >> > > >> > > >> following
> > > > >>> > >> > > >> > > >> >> > > > > locations:
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may
> > have
> > > > to
> > > > >>> > change
> > > > >>> > >> > > dtcli
> > > > >>> > >> > > >> > > >> behaviour
> > > > >>> > >> > > >> > > >> >> on
> > > > >>> > >> > > >> > > >> >> > > how
> > > > >>> > >> > > >> > > >> >> > > > it
> > > > >>> > >> > > >> > > >> >> > > > > > > finds
> > > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may
> > > have
> > > > to
> > > > >>> > >> change
> > > > >>> > >> > > dtcli
> > > > >>> > >> > > >> > > >> >> behaviour on
> > > > >>> > >> > > >> > > >> >> > > how
> > > > >>> > >> > > >> > > >> >> > > > > it
> > > > >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
> > > > >>> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml
> and
> > > > other
> > > > >>> > files
> > > > >>> > >> > (for
> > > > >>> > >> > > >> new
> > > > >>> > >> > > >> > > users
> > > > >>> > >> > > >> > > >> >> that
> > > > >>> > >> > > >> > > >> >> > > will
> > > > >>> > >> > > >> > > >> >> > > > > be
> > > > >>> > >> > > >> > > >> >> > > > > > > > created in system in future.)
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > We may also have to modify
> > > bashrc /
> > > > >>> > >> > bashprofile
> > > > >>> > >> > > >> for
> > > > >>> > >> > > >> > > >> >> population
> > > > >>> > >> > > >> > > >> >> > > any
> > > > >>> > >> > > >> > > >> >> > > > > > > > variables that are required.
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your
> > > comments /
> > > > >>> > >> > suggestion.
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16
> > AM,
> > > > >>> Thomas
> > > > >>> > >> Weise
> > > > >>> > >> > <
> > > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > -->
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05
> > AM,
> > > > >>> Chinmay
> > > > >>> > >> > > >> Kolhatkar <
> > > > >>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging
> > > > effort
> > > > >>> > >> (rpm+deb)
> > > > >>> > >> > > of
> > > > >>> > >> > > >> > apex,
> > > > >>> > >> > > >> > > >> here
> > > > >>> > >> > > >> > > >> >> > are
> > > > >>> > >> > > >> > > >> >> > > > some
> > > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > about package structure
> > etc..
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on
> > bbigtop
> > > > >>> mailing
> > > > >>> > >> > list, I
> > > > >>> > >> > > >> have
> > > > >>> > >> > > >> > > >> some
> > > > >>> > >> > > >> > > >> >> > > question
> > > > >>> > >> > > >> > > >> >> > > > > for
> > > > >>> > >> > > >> > > >> >> > > > > > > > apex
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory
> > structure
> > > of
> > > > >>> apex
> > > > >>> > >> > package
> > > > >>> > >> > > >> for
> > > > >>> > >> > > >> > > both
> > > > >>> > >> > > >> > > >> >> deb &
> > > > >>> > >> > > >> > > >> >> > > > rpm:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other
> > > > >>> dependent
> > > > >>> > >> jars>
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> > > > >>> > /usr/lib/apex/bin/dtcli
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> /usr/share/doc/man/man1/dtcli.1.gz
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > /usr/share/doc/apex/license/LICENSE.txt
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > /usr/share/doc/apex/CHANGELOG
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> /usr/share/doc/apex/NOTICE
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
> > 3.3.0-incubating.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> =
> > All
> > > > the
> > > > >>> 3rd
> > > > >>> > >> party
> > > > >>> > >> > > >> jars
> > > > >>> > >> > > >> > > which
> > > > >>> > >> > > >> > > >> >> are
> > > > >>> > >> > > >> > > >> >> > > > > required
> > > > >>> > >> > > >> > > >> >> > > > > > > for
> > > > >>> > >> > > >> > > >> >> > > > > > > > > apex
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the
> > > > dependencies
> > > > >>> are
> > > > >>> > >> > > packaged
> > > > >>> > >> > > >> as
> > > > >>> > >> > > >> > > >> part of
> > > > >>> > >> > > >> > > >> >> > > > rpm/deb
> > > > >>> > >> > > >> > > >> >> > > > > by
> > > > >>> > >> > > >> > > >> >> > > > > > > any
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what those
> > jars
> > > > >>> are in
> > > > >>> > >> > Bigtop
> > > > >>> > >> > > >> > > context?
> > > > >>> > >> > > >> > > >> >> Same
> > > > >>> > >> > > >> > > >> >> > as
> > > > >>> > >> > > >> > > >> >> > > > > > shipped
> > > > >>> > >> > > >> > > >> >> > > > > > > > with
> > > > >>> > >> > > >> > > >> >> > > > > > > > > DT community addition under
> > > lib/
> > > > ?
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> =
> > Licenses
> > > of
> > > > >>> > >> > > corresponding
> > > > >>> > >> > > >> of
> > > > >>> > >> > > >> > > 3rd
> > > > >>> > >> > > >> > > >> >> party
> > > > >>> > >> > > >> > > >> >> > > jars
> > > > >>> > >> > > >> > > >> >> > > > > > which
> > > > >>> > >> > > >> > > >> >> > > > > > > > > needs
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > to included while
> > packaging.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions related to
> this:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli
> > of
> > > > >>> apex as
> > > > >>> > >> > "apex"
> > > > >>> > >> > > >> > instead
> > > > >>> > >> > > >> > > >> of
> > > > >>> > >> > > >> > > >> >> > > "dtcli"
> > > > >>> > >> > > >> > > >> >> > > > in
> > > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > I think we should keep the
> > name
> > > > >>> until
> > > > >>> > we
> > > > >>> > >> are
> > > > >>> > >> > > >> able
> > > > >>> > >> > > >> > to
> > > > >>> > >> > > >> > > >> >> change
> > > > >>> > >> > > >> > > >> >> > it
> > > > >>> > >> > > >> > > >> >> > > in
> > > > >>> > >> > > >> > > >> >> > > > > > Apex.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get
> > > confusing.
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all
> softwares
> > > in
> > > > >>> bigtop
> > > > >>> > >> have
> > > > >>> > >> > > man
> > > > >>> > >> > > >> > page
> > > > >>> > >> > > >> > > >> for
> > > > >>> > >> > > >> > > >> >> > their
> > > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we should have it
> > too
> > > > for
> > > > >>> > >> dtcli.
> > > > >>> > >> > Is
> > > > >>> > >> > > >> there
> > > > >>> > >> > > >> > > any
> > > > >>> > >> > > >> > > >> >> > > > > documentation
> > > > >>> > >> > > >> > > >> >> > > > > > > > > which I
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man page?
> or
> > > can
> > > > >>> I use
> > > > >>> > >> > output
> > > > >>> > >> > > >> of
> > > > >>> > >> > > >> > > >> "dtcli
> > > > >>> > >> > > >> > > >> >> > > --help"?
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call
> > version
> > > > of
> > > > >>> apex
> > > > >>> > >> in
> > > > >>> > >> > > >> Bigtop
> > > > >>> > >> > > >> > as
> > > > >>> > >> > > >> > > >> >> 3.3.0 OR
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex
> > package
> > > of
> > > > >>> > bigtop
> > > > >>> > >> to
> > > > >>> > >> > > >> depend
> > > > >>> > >> > > >> > on
> > > > >>> > >> > > >> > > >> >> 2.7.1
> > > > >>> > >> > > >> > > >> >> > > > > version
> > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any
> problems
> > > > that
> > > > >>> we
> > > > >>> > see
> > > > >>> > >> > with
> > > > >>> > >> > > >> this
> > > > >>> > >> > > >> > > >> >> > dependency?
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > Should work without
> changed.
> > I
> > > > >>> thought
> > > > >>> > we
> > > > >>> > >> > > >> certified
> > > > >>> > >> > > >> > > >> >> against
> > > > >>> > >> > > >> > > >> >> > > > 2.7.0?
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the
> apache
> > > > mirror
> > > > >>> > from
> > > > >>> > >> > which
> > > > >>> > >> > > >> > bigtop
> > > > >>> > >> > > >> > > >> will
> > > > >>> > >> > > >> > > >> >> > pick
> > > > >>> > >> > > >> > > >> >> > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > apex
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > source for compilation
> and
> > > > >>> packaging.
> > > > >>> > >> > Please
> > > > >>> > >> > > >> > > correct
> > > > >>> > >> > > >> > > >> if
> > > > >>> > >> > > >> > > >> >> > > > > incorrect:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > >
> > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
> > apache.osuosl.org
> > > is
> > > > >>> the
> > > > >>> > >> mirror
> > > > >>> > >> > > >> used
> > > > >>> > >> > > >> > by
> > > > >>> > >> > > >> > > >> all
> > > > >>> > >> > > >> > > >> >> the
> > > > >>> > >> > > >> > > >> >> > > > > > softwares
> > > > >>> > >> > > >> > > >> >> > > > > > > in
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share your
> opinion.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at
> > 11:19
> > > > AM,
> > > > >>> > >> Chinmay
> > > > >>> > >> > > >> > Kolhatkar
> > > > >>> > >> > > >> > > <
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work
> on
> > > the
> > > > >>> > >> packaging.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already started
> > > > >>> discussion on
> > > > >>> > >> > bigtop
> > > > >>> > >> > > >> dev
> > > > >>> > >> > > >> > > >> mailing
> > > > >>> > >> > > >> > > >> >> > list
> > > > >>> > >> > > >> > > >> >> > > > for
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also
> > created a
> > > > >>> Jira
> > > > >>> > for
> > > > >>> > >> the
> > > > >>> > >> > > >> same.
> > > > >>> > >> > > >> > > For
> > > > >>> > >> > > >> > > >> >> this
> > > > >>> > >> > > >> > > >> >> > > > > > communities
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is the
> > > bigtop
> > > > >>> Jira:
> > > > >>> > >> > > >> > BIGTOP-2313.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at
> > 1:13
> > > > AM,
> > > > >>> > >> Thomas
> > > > >>> > >> > > >> Weise <
> > > > >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything
> > under
> > > > >>> > >> > prerequisites
> > > > >>> > >> > > >> that
> > > > >>> > >> > > >> > > >> would
> > > > >>> > >> > > >> > > >> >> be a
> > > > >>> > >> > > >> > > >> >> > > > > > problem.
> > > > >>> > >> > > >> > > >> >> > > > > > > We
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
> > > > >>> compatibility
> > > > >>> > >> as
> > > > >>> > >> > > part
> > > > >>> > >> > > >> of
> > > > >>> > >> > > >> > > >> >> becoming
> > > > >>> > >> > > >> > > >> >> > an
> > > > >>> > >> > > >> > > >> >> > > > > > > incubator
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the
> > > > packaging
> > > > >>> > during
> > > > >>> > >> > the
> > > > >>> > >> > > >> next
> > > > >>> > >> > > >> > > >> weeks.
> > > > >>> > >> > > >> > > >> >> > Since
> > > > >>> > >> > > >> > > >> >> > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > work
> > > > >>> > >> > > >> > > >> >> > > > > > > > > will
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop,
> related
> > > > >>> > discussions
> > > > >>> > >> and
> > > > >>> > >> > > >> JIRAs
> > > > >>> > >> > > >> > > >> should
> > > > >>> > >> > > >> > > >> >> > also
> > > > >>> > >> > > >> > > >> >> > > be
> > > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have
> > the
> > > > >>> > packaging
> > > > >>> > >> in
> > > > >>> > >> > > >> place
> > > > >>> > >> > > >> > by
> > > > >>> > >> > > >> > > >> end
> > > > >>> > >> > > >> > > >> >> Feb.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at
> > > 10:14
> > > > >>> AM,
> > > > >>> > >> > Chinmay
> > > > >>> > >> > > >> > > >> Kolhatkar <
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> chinmay@datatorrent.com
> > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a
> work
> > on
> > > > >>> adding
> > > > >>> > >> > Apache
> > > > >>> > >> > > >> Apex
> > > > >>> > >> > > >> > > as a
> > > > >>> > >> > > >> > > >> >> > > component
> > > > >>> > >> > > >> > > >> >> > > > > to
> > > > >>> > >> > > >> > > >> >> > > > > > > > Apache
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the
> > packaging
> > > > >>> system
> > > > >>> > for
> > > > >>> > >> > the
> > > > >>> > >> > > >> > Apache
> > > > >>> > >> > > >> > > >> big
> > > > >>> > >> > > >> > > >> >> data
> > > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use
> it,
> > > > most
> > > > >>> > >> recently
> > > > >>> > >> > > EMR.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking
> > > Jira
> > > > >>> task
> > > > >>> > in
> > > > >>> > >> > > >> APEXCORE
> > > > >>> > >> > > >> > for
> > > > >>> > >> > > >> > > >> the
> > > > >>> > >> > > >> > > >> >> > same:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >>
> > > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of
> > > execution
> > > > >>> is as
> > > > >>> > >> > > follows:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
> > > > >>> prerequisites*
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has
> some
> > > > hard
> > > > >>> and
> > > > >>> > >> soft
> > > > >>> > >> > > >> > > expectation
> > > > >>> > >> > > >> > > >> >> for
> > > > >>> > >> > > >> > > >> >> > new
> > > > >>> > >> > > >> > > >> >> > > > > > > > components
> > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into
> > Bigtop.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of
> > it:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> >
> > > > >>> > >> > > >>
> > > > >>> > >> > >
> > > > >>> > >> >
> > > > >>> > >>
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems
> to
> > be
> > > > >>> > standard
> > > > >>> > >> ASF
> > > > >>> > >> > > >> based
> > > > >>> > >> > > >> > > >> >> > > requirements,
> > > > >>> > >> > > >> > > >> >> > > > > but
> > > > >>> > >> > > >> > > >> >> > > > > > > few
> > > > >>> > >> > > >> > > >> >> > > > > > > > > need
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects
> > are
> > > > >>> > expected
> > > > >>> > >> to
> > > > >>> > >> > be
> > > > >>> > >> > > >> > > Licensed
> > > > >>> > >> > > >> > > >> >> under
> > > > >>> > >> > > >> > > >> >> > > > > Apache
> > > > >>> > >> > > >> > > >> >> > > > > > > > > License,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and
> their
> > > > >>> > >> dependencies
> > > > >>> > >> > are
> > > > >>> > >> > > >> > > expected
> > > > >>> > >> > > >> > > >> >> to be
> > > > >>> > >> > > >> > > >> >> > > > > > > compatible
> > > > >>> > >> > > >> > > >> >> > > > > > > > > with
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under
> > ASL
> > > > 2.0
> > > > >>> but
> > > > >>> > >> need
> > > > >>> > >> > to
> > > > >>> > >> > > >> > check
> > > > >>> > >> > > >> > > if
> > > > >>> > >> > > >> > > >> >> > > > > dependencies
> > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL
> > 2.0.
> > > > >>> This I
> > > > >>> > >> guess
> > > > >>> > >> > > >> would
> > > > >>> > >> > > >> > > be a
> > > > >>> > >> > > >> > > >> >> > > > > verification
> > > > >>> > >> > > >> > > >> >> > > > > > > > check.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects
> > are
> > > > >>> > expected
> > > > >>> > >> to
> > > > >>> > >> > be
> > > > >>> > >> > > >> > > >> compatible
> > > > >>> > >> > > >> > > >> >> > with
> > > > >>> > >> > > >> > > >> >> > > > all
> > > > >>> > >> > > >> > > >> >> > > > > of
> > > > >>> > >> > > >> > > >> >> > > > > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms
> > that
> > > > >>> Bigtop
> > > > >>> > >> > > >> distribution
> > > > >>> > >> > > >> > > is
> > > > >>> > >> > > >> > > >> >> > > targeting
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > > verifying
> > > > >>> > whether
> > > > >>> > >> > our
> > > > >>> > >> > > >> > > software
> > > > >>> > >> > > >> > > >> >> runs
> > > > >>> > >> > > >> > > >> >> > > fine
> > > > >>> > >> > > >> > > >> >> > > > in
> > > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
> > > > >>> ubuntu-14.04
> > > > >>> > >> > > debian-8
> > > > >>> > >> > > >> > > >> >> > opensuse-13.2.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests
> > that
> > > > >>> should
> > > > >>> > be
> > > > >>> > >> > > added
> > > > >>> > >> > > >> for
> > > > >>> > >> > > >> > > >> >> > deployment.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the
> > test
> > > > >>> > artifacts
> > > > >>> > >> > which
> > > > >>> > >> > > >> goes
> > > > >>> > >> > > >> > > >> beyond
> > > > >>> > >> > > >> > > >> >> > > smoke
> > > > >>> > >> > > >> > > >> >> > > > > test
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are
> > > basically
> > > > >>> the
> > > > >>> > >> > > integration
> > > > >>> > >> > > >> > > tests
> > > > >>> > >> > > >> > > >> for
> > > > >>> > >> > > >> > > >> >> > > > > > verification
> > > > >>> > >> > > >> > > >> >> > > > > > > > > after
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is
> a
> > > soft
> > > > >>> > >> > requirement,
> > > > >>> > >> > > >> but
> > > > >>> > >> > > >> > > aim
> > > > >>> > >> > > >> > > >> is
> > > > >>> > >> > > >> > > >> >> to
> > > > >>> > >> > > >> > > >> >> > > > > achieve
> > > > >>> > >> > > >> > > >> >> > > > > > > this
> > > > >>> > >> > > >> > > >> >> > > > > > > > > as
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have
> > > > >>> explanation why
> > > > >>> > >> not
> > > > >>> > >> > to
> > > > >>> > >> > > >> > > include.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any
> from
> > > the
> > > > >>> link
> > > > >>> > >> which
> > > > >>> > >> > > >> > > explicitly
> > > > >>> > >> > > >> > > >> >> needs
> > > > >>> > >> > > >> > > >> >> > to
> > > > >>> > >> > > >> > > >> >> > > > be
> > > > >>> > >> > > >> > > >> >> > > > > > > > checked
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please
> > let
> > > > us
> > > > >>> > know.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex
> > as
> > > > >>> > component
> > > > >>> > >> to
> > > > >>> > >> > > >> Bigtop*
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail
> > > > >>> archive of
> > > > >>> > >> > Bigtop,
> > > > >>> > >> > > >> it
> > > > >>> > >> > > >> > was
> > > > >>> > >> > > >> > > >> >> learnt
> > > > >>> > >> > > >> > > >> >> > > that
> > > > >>> > >> > > >> > > >> >> > > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community want to
> see
> > > the
> > > > >>> > >> addition of
> > > > >>> > >> > > new
> > > > >>> > >> > > >> > > >> >> components
> > > > >>> > >> > > >> > > >> >> > in
> > > > >>> > >> > > >> > > >> >> > > > > > phases.
> > > > >>> > >> > > >> > > >> >> > > > > > > > Here
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > are
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > > creating
> > > > of
> > > > >>> > >> package
> > > > >>> > >> > > i.e.
> > > > >>> > >> > > >> > rpm
> > > > >>> > >> > > >> > > &
> > > > >>> > >> > > >> > > >> deb
> > > > >>> > >> > > >> > > >> >> > > files.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > > > >>> documentations/READMEs,
> > > > >>> > >> > LICENSE,
> > > > >>> > >> > > >> > > >> DISCLAMER,
> > > > >>> > >> > > >> > > >> >> > NOTES
> > > > >>> > >> > > >> > > >> >> > > > etc
> > > > >>> > >> > > >> > > >> >> > > > > if
> > > > >>> > >> > > >> > > >> >> > > > > > > any
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> > documentation
> > > > that
> > > > >>> > need
> > > > >>> > >> to
> > > > >>> > >> > be
> > > > >>> > >> > > >> > added
> > > > >>> > >> > > >> > > to
> > > > >>> > >> > > >> > > >> >> > > > > distribution
> > > > >>> > >> > > >> > > >> >> > > > > > of
> > > > >>> > >> > > >> > > >> >> > > > > > > > our
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license
> > > > >>> information of
> > > > >>> > >> > > >> > dependencies
> > > > >>> > >> > > >> > > >> >> required
> > > > >>> > >> > > >> > > >> >> > > to
> > > > >>> > >> > > >> > > >> >> > > > be
> > > > >>> > >> > > >> > > >> >> > > > > > > added
> > > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at
> > very
> > > > >>> least)
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke
> > test
> > > > for
> > > > >>> > >> > packaging.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes
> for
> > > > >>> automatic
> > > > >>> > >> > > >> deployment
> > > > >>> > >> > > >> > and
> > > > >>> > >> > > >> > > >> >> > > > configuration
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet
> > recipes
> > > > for
> > > > >>> > >> > automatic
> > > > >>> > >> > > >> > > >> deployment
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For
> verification
> > > of
> > > > >>> > >> > deployments.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license
> clearance:
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat'
> > to
> > > > make
> > > > >>> > sure
> > > > >>> > >> all
> > > > >>> > >> > > new
> > > > >>> > >> > > >> > > stuff
> > > > >>> > >> > > >> > > >> is
> > > > >>> > >> > > >> > > >> >> > > > compliant
> > > > >>> > >> > > >> > > >> >> > > > > > > with
> > > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license
> requirements.
> > If
> > > > >>> you add
> > > > >>> > >> code
> > > > >>> > >> > > >> > licenses
> > > > >>> > >> > > >> > > >> >> under
> > > > >>> > >> > > >> > > >> >> > > > > different
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need to
> be
> > > > >>> listed in
> > > > >>> > >> the
> > > > >>> > >> > > >> NOTICE.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your
> > > thoughts
> > > > >>> on
> > > > >>> > the
> > > > >>> > >> > > >> approach.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
> > > corresponding
> > > > >>> > >> > communication
> > > > >>> > >> > > >> on
> > > > >>> > >> > > >> > > >> bigtop
> > > > >>> > >> > > >> > > >> >> > > mailing
> > > > >>> > >> > > >> > > >> >> > > > > list
> > > > >>> > >> > > >> > > >> >> > > > > > > as
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some
> specific
> > > > >>> > >> > > >> questions/suggestions
> > > > >>> > >> > > >> > > >> >> related to
> > > > >>> > >> > > >> > > >> >> > > > what
> > > > >>> > >> > > >> > > >> >> > > > > > > should
> > > > >>> > >> > > >> > > >> >> > > > > > > > > be
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the
> package
> > > and
> > > > >>> what
> > > > >>> > >> > should
> > > > >>> > >> > > be
> > > > >>> > >> > > >> > the
> > > > >>> > >> > > >> > > >> smoke
> > > > >>> > >> > > >> > > >> >> > > tests,
> > > > >>> > >> > > >> > > >> >> > > > > but
> > > > >>> > >> > > >> > > >> >> > > > > > > in
> > > > >>> > >> > > >> > > >> >> > > > > > > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not
> having
> > > too
> > > > >>> much
> > > > >>> > >> > content
> > > > >>> > >> > > >> > here,
> > > > >>> > >> > > >> > > >> we'll
> > > > >>> > >> > > >> > > >> >> > put
> > > > >>> > >> > > >> > > >> >> > > > the
> > > > >>> > >> > > >> > > >> >> > > > > > > > > questions
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in
> this
> > > > >>> > mailthread.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > > >
> > > > >>> > >> > > >> > > >> >> > > > > >
> > > > >>> > >> > > >> > > >> >> > > > >
> > > > >>> > >> > > >> > > >> >> > > >
> > > > >>> > >> > > >> > > >> >> > >
> > > > >>> > >> > > >> > > >> >> >
> > > > >>> > >> > > >> > > >> >>
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >> >
> > > > >>> > >> > > >> > > >>
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > > >
> > > > >>> > >> > > >> > >
> > > > >>> > >> > > >> >
> > > > >>> > >> > > >>
> > > > >>> > >> > > >
> > > > >>> > >> > > >
> > > > >>> > >> > >
> > > > >>> > >> >
> > > > >>> > >>
> > > > >>> > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[David Yan <da...@datatorrent.com>]

I should've mentioned that I had a pull request that was merged yesterday
that corrected the statement:

https://github.com/DataTorrent/docs/pull/45

On Thu, Feb 18, 2016 at 12:07 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:

> In following statement, can you replace dtGateway with ApplicationMaster
> and Yarn. or may be remove this statement entirely.
> apex is a wrapper around the REST API provided by dtGatway,
>
> - Tushar.
>
>
>
>
> On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Hi,
> >
> > I've attached man page for apex in Jira. Here is the link to it:
> > https://issues.apache.org/jira/secure/attachment/12788403/apex.1
> >
> > Please view it by: man -l apex.1
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > Hi Chinmay,
> > >
> > > I am not able to find the attachment, may be attachments are not
> allowed
> > in
> > > the mailing list. Can you attach the man page to the Jira?
> > >
> > > - Tushar.
> > >
> > >
> > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <chinmay@apache.org
> >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I've attached man page file for the purpose of bigtop integration.
> > > > Please have a look and let me know if there any any feedback about
> it.
> > > >
> > > > You can see the man page content of this file with:
> > > > man -l apex.1
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > >
> > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> > chinmay@apache.org>
> > > > wrote:
> > > >
> > > >> Thanks. Will follow bigtop recommendation. i.e. remove
> "-incubating".
> > > >>
> > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
> thomas@datatorrent.com
> > >
> > > >> wrote:
> > > >>
> > > >>> Please follow the recommendation and continue discussion on the
> > Bigtop
> > > >>> list.
> > > >>>
> > > >>> --
> > > >>> sent from mobile
> > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <ch...@apache.org>
> > > wrote:
> > > >>>
> > > >>> > I got a recommendation from bigtop community to drop
> "-incubating"
> > > from
> > > >>> > version.
> > > >>> >
> > > >>> > Here is the mail thread:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> > > >>> >
> > > >>> > Please let me know if removing "-incubating" is fine.
> > > >>> >
> > > >>> > Thanks,
> > > >>> > Chinmay.
> > > >>> >
> > > >>> >
> > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> > > chinmay@apache.org
> > > >>> >
> > > >>> > wrote:
> > > >>> >
> > > >>> > > I've already started communication there.
> > > >>> > >
> > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> > > >>> thomas@datatorrent.com>
> > > >>> > > wrote:
> > > >>> > >
> > > >>> > >> This is a question you should ask on the Bigtop mailing list.
> > > >>> > >>
> > > >>> > >> --
> > > >>> > >> sent from mobile
> > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org>
> > > >>> wrote:
> > > >>> > >>
> > > >>> > >> > Yes. In case of RPM, the only restriction on version is that
> > it
> > > >>> can
> > > >>> > not
> > > >>> > >> > contain '-'.
> > > >>> > >> > However, as a workaround, one can use 'incubating' as a part
> > of
> > > >>> > >> %{release}.
> > > >>> > >> >
> > > >>> > >> > Version: 3.3.0
> > > >>> > >> > Release: incubating
> > > >>> > >> >
> > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> > > >>> > >> >
> > > >>> > >> > It is the same practice followed for pre-release packages
> e.g.
> > > >>> with
> > > >>> > rcX,
> > > >>> > >> > alphaX, betaX being used in %{release}.
> > > >>> > >> >
> > > >>> > >> > Regards,
> > > >>> > >> > Pradeep A. Dalvi
> > > >>> > >> >
> > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> > > >>> > chinmay@apache.org>
> > > >>> > >> > wrote:
> > > >>> > >> >
> > > >>> > >> > > Hi Everyone,
> > > >>> > >> > >
> > > >>> > >> > > In one of the previous mails in this thread, we decided
> that
> > > >>> version
> > > >>> > >> of
> > > >>> > >> > the
> > > >>> > >> > > package will be "3.3.0-incubating".
> > > >>> > >> > > The problem that I'm facing is in RPM base version "-" is
> > not
> > > >>> > allowed.
> > > >>> > >> > >
> > > >>> > >> > > Can we have package version to "3.3.0"?
> > > >>> > >> > > i.e. the packages created will be as follows:
> > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> > > >>> > >> > >
> > > >>> > >> > > There is another integration for Zepplin in Bigtop which
> is
> > > >>> > incubating
> > > >>> > >> > and
> > > >>> > >> > > is having base version as "0.5.6".
> > > >>> > >> > >
> > > >>> > >> > > Please let me know your thoughts.
> > > >>> > >> > >
> > > >>> > >> > > Thanks,
> > > >>> > >> > > Chinmay.
> > > >>> > >> > >
> > > >>> > >> > >
> > > >>> > >> > >
> > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> > > >>> > >> chinmay@apache.org>
> > > >>> > >> > > wrote:
> > > >>> > >> > >
> > > >>> > >> > > > Thanks everyone for the feedback.
> > > >>> > >> > > >
> > > >>> > >> > > > Also as Ram and David mentioned, I'll start creating the
> > man
> > > >>> page
> > > >>> > as
> > > >>> > >> > > well.
> > > >>> > >> > > >
> > > >>> > >> > > > Thanks,
> > > >>> > >> > > > Chinmay.
> > > >>> > >> > > >
> > > >>> > >> > > >
> > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> > > >>> > >> thomas@datatorrent.com>
> > > >>> > >> > > > wrote:
> > > >>> > >> > > >
> > > >>> > >> > > >> I think we already agreed that we will retain the
> current
> > > >>> script
> > > >>> > >> name
> > > >>> > >> > in
> > > >>> > >> > > >> this bigtop iteration. There is a ticket for renaming,
> > add
> > > >>> you
> > > >>> > >> > > suggestions
> > > >>> > >> > > >> there.
> > > >>> > >> > > >>
> > > >>> > >> > > >> --
> > > >>> > >> > > >> sent from mobile
> > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > > >>> > sandesh@datatorrent.com>
> > > >>> > >> > > wrote:
> > > >>> > >> > > >>
> > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start the
> > transition
> > > >>> > >> process by
> > > >>> > >> > > >> making
> > > >>> > >> > > >> > dtcli as the symlink to apex cli.
> > > >>> > >> > > >> >
> > > >>> > >> > > >> > Renaming is a breaking change should be done during
> > major
> > > >>> > version
> > > >>> > >> > > number
> > > >>> > >> > > >> > change.
> > > >>> > >> > > >> >
> > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> > > >>> > david@datatorrent.com
> > > >>> > >> >
> > > >>> > >> > > >> wrote:
> > > >>> > >> > > >> >
> > > >>> > >> > > >> > > Apologies for hitting the "Send" button too early.
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > What is the scope of the name change?  If we simply
> > > have
> > > >>> a
> > > >>> > >> symlink
> > > >>> > >> > > >> from
> > > >>> > >> > > >> > > apex to dtcli, then it should have no backward
> > > >>> compatibility
> > > >>> > >> > problem
> > > >>> > >> > > >> at
> > > >>> > >> > > >> > > all.
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > But the proper name change would involve:
> > > >>> > >> > > >> > > - Change the name of DTCli class to ApexCli
> > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> > > >>> > >> > > >> > > - Change the initial greeting when running the CLI
> > from
> > > >>> > >> something
> > > >>> > >> > > like
> > > >>> > >> > > >> > "DT
> > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > The above changes may break existing scripts that
> > rely
> > > >>> on the
> > > >>> > >> the
> > > >>> > >> > > >> string
> > > >>> > >> > > >> > > "DT".
> > > >>> > >> > > >> > > But if folks are okay with this potential problem,
> I
> > am
> > > >>> okay
> > > >>> > >> with
> > > >>> > >> > > the
> > > >>> > >> > > >> > name
> > > >>> > >> > > >> > > change.
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > For the man page, we can start with the content of
> > this
> > > >>> page:
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > Plus the command line options when doing dtcli -h:
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > usage: DTCli
> > > >>> > >> > > >> > >  -e <arg>    Commands are read from the argument
> > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at all time
> > > >>> > >> > > >> > >  -h          Print this help
> > > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos principal
> > > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> > > >>> > >> > > >> > >  -r          JSON Raw mode
> > > >>> > >> > > >> > >  -v          Verbose mode level 1
> > > >>> > >> > > >> > >  -vv         Verbose mode level 2
> > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > David
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> > > >>> > >> david@datatorrent.com
> > > >>> > >> > >
> > > >>> > >> > > >> > wrote:
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > > What is the scope of the name change?  If we
> simply
> > > >>> have a
> > > >>> > >> > symlink
> > > >>> > >> > > >> from
> > > >>> > >> > > >> > > > apex to dtcli, then it should have no backward
> > > >>> > compatibility
> > > >>> > >> > > >> problem at
> > > >>> > >> > > >> > > all.
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > But the proper name change would involve:
> > > >>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
> > > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> > > >>> > >> > > >> > > > - Change the initial greeting when running the
> CLI
> > > from
> > > >>> > >> > something
> > > >>> > >> > > >> like
> > > >>> > >> > > >> > > "DT
> > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > The above changes may break existing scripts that
> > > rely
> > > >>> on
> > > >>> > the
> > > >>> > >> > the
> > > >>> > >> > > >> > string
> > > >>> > >> > > >> > > > "DT".
> > > >>> > >> > > >> > > > But if folks are okay with this potential
> problem,
> > I
> > > am
> > > >>> > okay
> > > >>> > >> > with
> > > >>> > >> > > >> the
> > > >>> > >> > > >> > > name
> > > >>> > >> > > >> > > > change.
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > For the man page, we can start with the content
> of
> > > this
> > > >>> > page:
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay
> Kolhatkar
> > <
> > > >>> > >> > > >> chinmay@apache.org
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > > wrote:
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >> Here is a suggestion I have related to name
> > change:
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> 1. While packaging we change the name of "dtcli"
> > > >>> script to
> > > >>> > >> > > "apex".
> > > >>> > >> > > >> > This
> > > >>> > >> > > >> > > is
> > > >>> > >> > > >> > > >> done so that in future when we change the name
> to
> > > >>> "apex",
> > > >>> > >> users
> > > >>> > >> > > of
> > > >>> > >> > > >> > > bigtop
> > > >>> > >> > > >> > > >> apex don't have to transition much.
> > > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli which
> points
> > > to
> > > >>> apex
> > > >>> > >> > > script.
> > > >>> > >> > > >> > This
> > > >>> > >> > > >> > > is
> > > >>> > >> > > >> > > >> for backward compatibility.
> > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first
> > > integration
> > > >>> in
> > > >>> > >> > bigtop.
> > > >>> > >> > > >> This
> > > >>> > >> > > >> > is
> > > >>> > >> > > >> > > >> in
> > > >>> > >> > > >> > > >> the interest of not changing the content of
> dtcli
> > or
> > > >>> apex
> > > >>> > >> > script.
> > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these
> naming
> > > >>> related
> > > >>> > >> > > changes
> > > >>> > >> > > >> and
> > > >>> > >> > > >> > > >> update bigtop repository later to remove
> > references
> > > to
> > > >>> > dtcli
> > > >>> > >> > all
> > > >>> > >> > > >> > > together.
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> Please share your thoughts on above approach.
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> Also, please share what could be the man page
> > > content
> > > >>> for
> > > >>> > >> > "apex".
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> Thanks,
> > > >>> > >> > > >> > > >> Chinmay.
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay
> > Kolhatkar <
> > > >>> > >> > > >> > chinmay@apache.org>
> > > >>> > >> > > >> > > >> wrote:
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> > Yes.. It'll be build from source tar
> downloaded
> > > >>> from one
> > > >>> > >> of
> > > >>> > >> > the
> > > >>> > >> > > >> > apache
> > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced from
> > > dtcli.
> > > >>> > This
> > > >>> > >> > > leaves
> > > >>> > >> > > >> us
> > > >>> > >> > > >> > 3
> > > >>> > >> > > >> > > >> > options:
> > > >>> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli
> would
> > > >>> need a
> > > >>> > >> > change
> > > >>> > >> > > >> while
> > > >>> > >> > > >> > > >> > packaging.
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> > But if we want to use dtcli as it is, then we
> > > would
> > > >>> need
> > > >>> > >> > > >> dt-env.sh
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise
> <
> > > >>> > >> > > >> > thomas@datatorrent.com
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >> > wrote:
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >> Looks good overall, though there shouldn't be
> > any
> > > >>> > >> dt-env.sh
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >> What will the bigtop package be built from,
> the
> > > >>> source
> > > >>> > >> tar?
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay
> > > Kolhatkar
> > > >>> <
> > > >>> > >> > > >> > > chinmay@apache.org
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >> wrote:
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >> > Hi Thomas,
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > First, we're not changing the name
> anywhere.
> > > >>> We'll
> > > >>> > >> follow
> > > >>> > >> > > what
> > > >>> > >> > > >> > > >> >> currently is
> > > >>> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating
> > version.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory
> > structure
> > > >>> below
> > > >>> > >> which
> > > >>> > >> > > is
> > > >>> > >> > > >> > > inline
> > > >>> > >> > > >> > > >> >> with
> > > >>> > >> > > >> > > >> >> > other existing integrations with bigtop.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > The need for each file is as follows:
> > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> > > >>> > >> > > >> engine/src/main/scripts/
> > > >>> > >> > > >> > > of
> > > >>> > >> > > >> > > >> >> source
> > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from
> > dtcli.
> > > >>> This
> > > >>> > >> > > searches
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > >> hadoop
> > > >>> > >> > > >> > > >> >> > binary path and exports an env variable for
> > > >>> dtcli to
> > > >>> > >> use
> > > >>> > >> > it.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is
> > > >>> extracted
> > > >>> > >> in a
> > > >>> > >> > > >> similar
> > > >>> > >> > > >> > > >> way to
> > > >>> > >> > > >> > > >> >> > how DT Community Edition finds dependency
> > jars.
> > > >>> > >> > > >> > > >> >> > Please note that this is the first
> iteration
> > > >>> list of
> > > >>> > >> jars.
> > > >>> > >> > > I'm
> > > >>> > >> > > >> > > >> trying to
> > > >>> > >> > > >> > > >> >> > narrow this down to only those which are
> > really
> > > >>> > >> required.
> > > >>> > >> > > >> > > >> >> > The test that I'm running to check if dtcli
> > > runs
> > > >>> fine
> > > >>> > >> with
> > > >>> > >> > > >> given
> > > >>> > >> > > >> > > >> >> dependency
> > > >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi
> > demo.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Above is the directory structure which is
> > > >>> required
> > > >>> > for
> > > >>> > >> CLI
> > > >>> > >> > > to
> > > >>> > >> > > >> > work.
> > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned
> are
> > > some
> > > >>> > >> > additional
> > > >>> > >> > > >> > files
> > > >>> > >> > > >> > > >> which
> > > >>> > >> > > >> > > >> >> > makes the like of administrator easier. For
> > eg.
> > > >>> > >> /etc/skel
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Please let me know if this is inline with
> > what
> > > >>> you're
> > > >>> > >> > > >> thinking.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Thanks,
> > > >>> > >> > > >> > > >> >> > Chinmay.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > .
> > > >>> > >> > > >> > > >> >> > |-- bin
> > > >>> > >> > > >> > > >> >> > |   `-- dtcli
> > > >>> > >> > > >> > > >> >> > |-- conf
> > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > > >>> > >> > > >> > > >> >> > `-- lib
> > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > > >>> > >> > > >> > > >> >> >     |--
> > apex-bufferserver-3.3.0-incubating.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > > >>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > > >>> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > > >>> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |--
> jetty-websocket-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > > >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas
> > Weise <
> > > >>> > >> > > >> > > >> thomas@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > wrote:
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > > Chinmay,
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > Before discussing where to put the files,
> > > let's
> > > >>> > make
> > > >>> > >> > sure
> > > >>> > >> > > >> they
> > > >>> > >> > > >> > > are
> > > >>> > >> > > >> > > >> >> really
> > > >>> > >> > > >> > > >> >> > > needed for the operation of the CLI. As
> for
> > > >>> names,
> > > >>> > >> > > anything
> > > >>> > >> > > >> new
> > > >>> > >> > > >> > > >> needs
> > > >>> > >> > > >> > > >> >> to
> > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and should
> follow
> > > >>> common
> > > >>> > >> > > >> conventions,
> > > >>> > >> > > >> > > >> >> especially
> > > >>> > >> > > >> > > >> >> > > in Bigtop where there are many existing
> > > >>> > integrations
> > > >>> > >> to
> > > >>> > >> > > look
> > > >>> > >> > > >> > at.
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > Thanks,
> > > >>> > >> > > >> > > >> >> > > Thomas
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay
> > > >>> Kolhatkar
> > > >>> > <
> > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > wrote:
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep and
> Aniruddha.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to change
> the
> > > >>> dtcli
> > > >>> > >> > > >> considering it
> > > >>> > >> > > >> > > >> works
> > > >>> > >> > > >> > > >> >> > with
> > > >>> > >> > > >> > > >> >> > > DT
> > > >>> > >> > > >> > > >> >> > > > Community edition. We can keep the
> > > directory
> > > >>> > >> structure
> > > >>> > >> > > >> > similar
> > > >>> > >> > > >> > > to
> > > >>> > >> > > >> > > >> >> that.
> > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which contains
> > > >>> > information
> > > >>> > >> > > >> required
> > > >>> > >> > > >> > for
> > > >>> > >> > > >> > > >> >> dtcli
> > > >>> > >> > > >> > > >> >> > to
> > > >>> > >> > > >> > > >> >> > > > launch.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop community
> > that
> > > >>> > whether
> > > >>> > >> > they
> > > >>> > >> > > >> > > >> facilitate
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs before user
> is
> > > >>> created.
> > > >>> > >> In
> > > >>> > >> > > >> either
> > > >>> > >> > > >> > > case,
> > > >>> > >> > > >> > > >> >> > current
> > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home
> > > directory.
> > > >>> > Also
> > > >>> > >> > > before
> > > >>> > >> > > >> > > putting
> > > >>> > >> > > >> > > >> >> > > anything
> > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define what are
> > the
> > > >>> > default
> > > >>> > >> > > >> contents
> > > >>> > >> > > >> > > that
> > > >>> > >> > > >> > > >> >> > should
> > > >>> > >> > > >> > > >> >> > > go
> > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no
> > defaults,
> > > >>> > >> probably we
> > > >>> > >> > > >> should
> > > >>> > >> > > >> > > not
> > > >>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice.
> > > This
> > > >>> way
> > > >>> > >> > > contents
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > >> >> dt-env.sh
> > > >>> > >> > > >> > > >> >> > > are
> > > >>> > >> > > >> > > >> >> > > > present as env variables. But I see a
> > catch
> > > >>> > there.
> > > >>> > >> > > Adding
> > > >>> > >> > > >> > > >> dt-env.sh
> > > >>> > >> > > >> > > >> >> to
> > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the
> > variables
> > > >>> > >> available
> > > >>> > >> > at
> > > >>> > >> > > >> > > runtime
> > > >>> > >> > > >> > > >> all
> > > >>> > >> > > >> > > >> >> > the
> > > >>> > >> > > >> > > >> >> > > > time. I feel a little skeptical about
> > that.
> > > >>> > Maybe a
> > > >>> > >> > > >> possible
> > > >>> > >> > > >> > > >> >> collision
> > > >>> > >> > > >> > > >> >> > > can
> > > >>> > >> > > >> > > >> >> > > > occur with other application vars.
> > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
> > > >>> > >> > "../conf/dt-env.sh"
> > > >>> > >> > > >> and
> > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables
> are
> > > >>> anyway
> > > >>> > >> > > >> available.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is
> > > >>> happening
> > > >>> > at
> > > >>> > >> > > Bigtop
> > > >>> > >> > > >> > > >> >> > > Jira/repository.
> > > >>> > >> > > >> > > >> >> > > > Packaging related code usually goes
> > there.
> > > >>> > (That's
> > > >>> > >> > what
> > > >>> > >> > > >> all
> > > >>> > >> > > >> > the
> > > >>> > >> > > >> > > >> >> > > components
> > > >>> > >> > > >> > > >> >> > > > in bigtop does).
> > > >>> > >> > > >> > > >> >> > > > Having said that, once a PR is created,
> > > I'll
> > > >>> be
> > > >>> > >> > sharing
> > > >>> > >> > > >> the
> > > >>> > >> > > >> > > link
> > > >>> > >> > > >> > > >> of
> > > >>> > >> > > >> > > >> >> PR
> > > >>> > >> > > >> > > >> >> > > > here, so that apex community as well
> can
> > > >>> review
> > > >>> > it.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > Chinmay.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM,
> > Aniruddha
> > > >>> > >> Thombare <
> > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> > > >>> > >> > > >> > > >> >> > > > > We may need to iron out details about
> > > exact
> > > >>> > paths
> > > >>> > >> > etc.
> > > >>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR
> > comments.
> > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep
> A.
> > > >>> Dalvi <
> > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > > >>> > >> > > >> > > >> >> > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM,
> > > Chinmay
> > > >>> > >> > Kolhatkar <
> > > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present
> in
> > > DT
> > > >>> > >> community
> > > >>> > >> > > >> > edition
> > > >>> > >> > > >> > > >> will
> > > >>> > >> > > >> > > >> >> be
> > > >>> > >> > > >> > > >> >> > > > > included
> > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I found
> > from
> > > >>> > common
> > > >>> > >> > > >> > dependencies
> > > >>> > >> > > >> > > >> >> between
> > > >>> > >> > > >> > > >> >> > > of
> > > >>> > >> > > >> > > >> >> > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > > >>> > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > > >>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > >
> > jetty-websocket-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some
> > tests
> > > do
> > > >>> > check
> > > >>> > >> > that
> > > >>> > >> > > >> > dtcli
> > > >>> > >> > > >> > > >> works
> > > >>> > >> > > >> > > >> >> > > > properly
> > > >>> > >> > > >> > > >> >> > > > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with
> > > only
> > > >>> > these
> > > >>> > >> > > >> libraries
> > > >>> > >> > > >> > > >> >> present in
> > > >>> > >> > > >> > > >> >> > > > > > isolation
> > > >>> > >> > > >> > > >> >> > > > > > > without dependency on local m2.
> > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there are
> unwanted
> > > >>> jars
> > > >>> > >> which
> > > >>> > >> > are
> > > >>> > >> > > >> used
> > > >>> > >> > > >> > > for
> > > >>> > >> > > >> > > >> >> > compile
> > > >>> > >> > > >> > > >> >> > > > > time
> > > >>> > >> > > >> > > >> >> > > > > > > dependency and not runtime in
> above
> > > >>> list
> > > >>> > >> which I
> > > >>> > >> > > can
> > > >>> > >> > > >> > drop
> > > >>> > >> > > >> > > >> to
> > > >>> > >> > > >> > > >> >> keep
> > > >>> > >> > > >> > > >> >> > > > > package
> > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if
> all
> > > of
> > > >>> the
> > > >>> > >> > > >> dt-env.sh is
> > > >>> > >> > > >> > > >> >> required.
> > > >>> > >> > > >> > > >> >> > > > Only
> > > >>> > >> > > >> > > >> >> > > > > > > required values I see in it are
> > > >>> classpath.
> > > >>> > >> and
> > > >>> > >> > > >> dtcli is
> > > >>> > >> > > >> > > >> >> capable
> > > >>> > >> > > >> > > >> >> > of
> > > >>> > >> > > >> > > >> >> > > > > > building
> > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have
> > to
> > > >>> see
> > > >>> > how
> > > >>> > >> > dtcli
> > > >>> > >> > > >> can
> > > >>> > >> > > >> > > find
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > jar
> > > >>> > >> > > >> > > >> >> > > > > path,
> > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf
> files.
> > If
> > > >>> we
> > > >>> > >> need a
> > > >>> > >> > > >> change
> > > >>> > >> > > >> > in
> > > >>> > >> > > >> > > >> >> dtcli,
> > > >>> > >> > > >> > > >> >> > > then
> > > >>> > >> > > >> > > >> >> > > > > > > community's opinion is required
> for
> > > >>> whether
> > > >>> > >> > dtcli
> > > >>> > >> > > >> > should
> > > >>> > >> > > >> > > >> >> change
> > > >>> > >> > > >> > > >> >> > in
> > > >>> > >> > > >> > > >> >> > > > our
> > > >>> > >> > > >> > > >> >> > > > > > repo
> > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with required
> > > changes
> > > >>> > >> exist in
> > > >>> > >> > > >> bigtop
> > > >>> > >> > > >> > > >> repo
> > > >>> > >> > > >> > > >> >> > until
> > > >>> > >> > > >> > > >> >> > > we
> > > >>> > >> > > >> > > >> >> > > > > > make
> > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Can we have these file paths set
> > using
> > > >>> > >> environment
> > > >>> > >> > > >> > > variables
> > > >>> > >> > > >> > > >> >> with
> > > >>> > >> > > >> > > >> >> > > some
> > > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> > > >>> > >> > > >> > > >> >> > > > > > Then we can set such params in
> > > dt-env.sh.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this
> file
> > > >>> > different
> > > >>> > >> > from
> > > >>> > >> > > >> > > >> >> dt-site.xml?
> > > >>> > >> > > >> > > >> >> > Not
> > > >>> > >> > > >> > > >> >> > > > > sure
> > > >>> > >> > > >> > > >> >> > > > > > if
> > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help.
> > > Files &
> > > >>> > Dirs
> > > >>> > >> > from
> > > >>> > >> > > >> > > /etc/skel
> > > >>> > >> > > >> > > >> >> are
> > > >>> > >> > > >> > > >> >> > > > copied
> > > >>> > >> > > >> > > >> >> > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > home of new user when useradd
> > program
> > > >>> is
> > > >>> > >> called.
> > > >>> > >> > > But
> > > >>> > >> > > >> > for
> > > >>> > >> > > >> > > >> >> existing
> > > >>> > >> > > >> > > >> >> > > > users
> > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any use.
> Moreover,
> > > >>> dtcli
> > > >>> > >> > creates
> > > >>> > >> > > >> > ~/.dt/
> > > >>> > >> > > >> > > on
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > fly
> > > >>> > >> > > >> > > >> >> > > > if
> > > >>> > >> > > >> > > >> >> > > > > > not
> > > >>> > >> > > >> > > >> >> > > > > > > encountered for the first time.
> > Again
> > > >>> > >> correct me
> > > >>> > >> > > if
> > > >>> > >> > > >> I'm
> > > >>> > >> > > >> > > >> wrong.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right. However this
> > would
> > > >>> > >> probably be
> > > >>> > >> > > >> > > necessary
> > > >>> > >> > > >> > > >> >> step
> > > >>> > >> > > >> > > >> >> > for
> > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get
> > installed
> > > >>> > during
> > > >>> > >> OS
> > > >>> > >> > > >> install
> > > >>> > >> > > >> > > and
> > > >>> > >> > > >> > > >> >> > before
> > > >>> > >> > > >> > > >> >> > > > user
> > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile
> > ->
> > > >>> I'm
> > > >>> > not
> > > >>> > >> > sure
> > > >>> > >> > > >> its
> > > >>> > >> > > >> > the
> > > >>> > >> > > >> > > >> best
> > > >>> > >> > > >> > > >> >> > > idea.
> > > >>> > >> > > >> > > >> >> > > > > > We'll
> > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location
> which
> > is
> > > >>> by
> > > >>> > >> default
> > > >>> > >> > > >> > present
> > > >>> > >> > > >> > > in
> > > >>> > >> > > >> > > >> >> path
> > > >>> > >> > > >> > > >> >> > > i.e.
> > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > For other env variables specific to
> > > apex,
> > > >>> > I'll
> > > >>> > >> > > prefer
> > > >>> > >> > > >> to
> > > >>> > >> > > >> > > use
> > > >>> > >> > > >> > > >> >> > > dt-env.sh
> > > >>> > >> > > >> > > >> >> > > > > and
> > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather than
> > > changing
> > > >>> > >> bashrc
> > > >>> > >> > > >> etc...
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
> > > >>> /etc/profile.d/
> > > >>> > >> shall
> > > >>> > >> > > be
> > > >>> > >> > > >> the
> > > >>> > >> > > >> > > >> >> cleaner
> > > >>> > >> > > >> > > >> >> > way
> > > >>> > >> > > >> > > >> >> > > > to
> > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM,
> > > >>> Aniruddha
> > > >>> > >> > > Thombare
> > > >>> > >> > > >> <
> > > >>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com>
> wrote:
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > We need to consider following:
> > > >>> > >> > > >> > > >> >> > > > > > > > System wide default config
> files
> > > can
> > > >>> be
> > > >>> > >> > located
> > > >>> > >> > > at
> > > >>> > >> > > >> > > >> following
> > > >>> > >> > > >> > > >> >> > > > > locations:
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may
> have
> > > to
> > > >>> > change
> > > >>> > >> > > dtcli
> > > >>> > >> > > >> > > >> behaviour
> > > >>> > >> > > >> > > >> >> on
> > > >>> > >> > > >> > > >> >> > > how
> > > >>> > >> > > >> > > >> >> > > > it
> > > >>> > >> > > >> > > >> >> > > > > > > finds
> > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may
> > have
> > > to
> > > >>> > >> change
> > > >>> > >> > > dtcli
> > > >>> > >> > > >> > > >> >> behaviour on
> > > >>> > >> > > >> > > >> >> > > how
> > > >>> > >> > > >> > > >> >> > > > > it
> > > >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
> > > >>> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and
> > > other
> > > >>> > files
> > > >>> > >> > (for
> > > >>> > >> > > >> new
> > > >>> > >> > > >> > > users
> > > >>> > >> > > >> > > >> >> that
> > > >>> > >> > > >> > > >> >> > > will
> > > >>> > >> > > >> > > >> >> > > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > > created in system in future.)
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > We may also have to modify
> > bashrc /
> > > >>> > >> > bashprofile
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > >> >> population
> > > >>> > >> > > >> > > >> >> > > any
> > > >>> > >> > > >> > > >> >> > > > > > > > variables that are required.
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your
> > comments /
> > > >>> > >> > suggestion.
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16
> AM,
> > > >>> Thomas
> > > >>> > >> Weise
> > > >>> > >> > <
> > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > -->
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05
> AM,
> > > >>> Chinmay
> > > >>> > >> > > >> Kolhatkar <
> > > >>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging
> > > effort
> > > >>> > >> (rpm+deb)
> > > >>> > >> > > of
> > > >>> > >> > > >> > apex,
> > > >>> > >> > > >> > > >> here
> > > >>> > >> > > >> > > >> >> > are
> > > >>> > >> > > >> > > >> >> > > > some
> > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> > > >>> > >> > > >> > > >> >> > > > > > > > > > about package structure
> etc..
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on
> bbigtop
> > > >>> mailing
> > > >>> > >> > list, I
> > > >>> > >> > > >> have
> > > >>> > >> > > >> > > >> some
> > > >>> > >> > > >> > > >> >> > > question
> > > >>> > >> > > >> > > >> >> > > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory
> structure
> > of
> > > >>> apex
> > > >>> > >> > package
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > both
> > > >>> > >> > > >> > > >> >> deb &
> > > >>> > >> > > >> > > >> >> > > > rpm:
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other
> > > >>> dependent
> > > >>> > >> jars>
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> > > >>> > /usr/lib/apex/bin/dtcli
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> /usr/share/doc/man/man1/dtcli.1.gz
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > /usr/share/doc/apex/license/LICENSE.txt
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> /usr/share/doc/apex/CHANGELOG
> > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
> 3.3.0-incubating.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> =
> All
> > > the
> > > >>> 3rd
> > > >>> > >> party
> > > >>> > >> > > >> jars
> > > >>> > >> > > >> > > which
> > > >>> > >> > > >> > > >> >> are
> > > >>> > >> > > >> > > >> >> > > > > required
> > > >>> > >> > > >> > > >> >> > > > > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the
> > > dependencies
> > > >>> are
> > > >>> > >> > > packaged
> > > >>> > >> > > >> as
> > > >>> > >> > > >> > > >> part of
> > > >>> > >> > > >> > > >> >> > > > rpm/deb
> > > >>> > >> > > >> > > >> >> > > > > by
> > > >>> > >> > > >> > > >> >> > > > > > > any
> > > >>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what those
> jars
> > > >>> are in
> > > >>> > >> > Bigtop
> > > >>> > >> > > >> > > context?
> > > >>> > >> > > >> > > >> >> Same
> > > >>> > >> > > >> > > >> >> > as
> > > >>> > >> > > >> > > >> >> > > > > > shipped
> > > >>> > >> > > >> > > >> >> > > > > > > > with
> > > >>> > >> > > >> > > >> >> > > > > > > > > DT community addition under
> > lib/
> > > ?
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> =
> Licenses
> > of
> > > >>> > >> > > corresponding
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > 3rd
> > > >>> > >> > > >> > > >> >> party
> > > >>> > >> > > >> > > >> >> > > jars
> > > >>> > >> > > >> > > >> >> > > > > > which
> > > >>> > >> > > >> > > >> >> > > > > > > > > needs
> > > >>> > >> > > >> > > >> >> > > > > > > > > > to included while
> packaging.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions related to this:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli
> of
> > > >>> apex as
> > > >>> > >> > "apex"
> > > >>> > >> > > >> > instead
> > > >>> > >> > > >> > > >> of
> > > >>> > >> > > >> > > >> >> > > "dtcli"
> > > >>> > >> > > >> > > >> >> > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > I think we should keep the
> name
> > > >>> until
> > > >>> > we
> > > >>> > >> are
> > > >>> > >> > > >> able
> > > >>> > >> > > >> > to
> > > >>> > >> > > >> > > >> >> change
> > > >>> > >> > > >> > > >> >> > it
> > > >>> > >> > > >> > > >> >> > > in
> > > >>> > >> > > >> > > >> >> > > > > > Apex.
> > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get
> > confusing.
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares
> > in
> > > >>> bigtop
> > > >>> > >> have
> > > >>> > >> > > man
> > > >>> > >> > > >> > page
> > > >>> > >> > > >> > > >> for
> > > >>> > >> > > >> > > >> >> > their
> > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we should have it
> too
> > > for
> > > >>> > >> dtcli.
> > > >>> > >> > Is
> > > >>> > >> > > >> there
> > > >>> > >> > > >> > > any
> > > >>> > >> > > >> > > >> >> > > > > documentation
> > > >>> > >> > > >> > > >> >> > > > > > > > > which I
> > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man page? or
> > can
> > > >>> I use
> > > >>> > >> > output
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > >> "dtcli
> > > >>> > >> > > >> > > >> >> > > --help"?
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call
> version
> > > of
> > > >>> apex
> > > >>> > >> in
> > > >>> > >> > > >> Bigtop
> > > >>> > >> > > >> > as
> > > >>> > >> > > >> > > >> >> 3.3.0 OR
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex
> package
> > of
> > > >>> > bigtop
> > > >>> > >> to
> > > >>> > >> > > >> depend
> > > >>> > >> > > >> > on
> > > >>> > >> > > >> > > >> >> 2.7.1
> > > >>> > >> > > >> > > >> >> > > > > version
> > > >>> > >> > > >> > > >> >> > > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems
> > > that
> > > >>> we
> > > >>> > see
> > > >>> > >> > with
> > > >>> > >> > > >> this
> > > >>> > >> > > >> > > >> >> > dependency?
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > Should work without changed.
> I
> > > >>> thought
> > > >>> > we
> > > >>> > >> > > >> certified
> > > >>> > >> > > >> > > >> >> against
> > > >>> > >> > > >> > > >> >> > > > 2.7.0?
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache
> > > mirror
> > > >>> > from
> > > >>> > >> > which
> > > >>> > >> > > >> > bigtop
> > > >>> > >> > > >> > > >> will
> > > >>> > >> > > >> > > >> >> > pick
> > > >>> > >> > > >> > > >> >> > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > source for compilation and
> > > >>> packaging.
> > > >>> > >> > Please
> > > >>> > >> > > >> > > correct
> > > >>> > >> > > >> > > >> if
> > > >>> > >> > > >> > > >> >> > > > > incorrect:
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > >
> http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
> apache.osuosl.org
> > is
> > > >>> the
> > > >>> > >> mirror
> > > >>> > >> > > >> used
> > > >>> > >> > > >> > by
> > > >>> > >> > > >> > > >> all
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > > > > > softwares
> > > >>> > >> > > >> > > >> >> > > > > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share your opinion.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at
> 11:19
> > > AM,
> > > >>> > >> Chinmay
> > > >>> > >> > > >> > Kolhatkar
> > > >>> > >> > > >> > > <
> > > >>> > >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on
> > the
> > > >>> > >> packaging.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already started
> > > >>> discussion on
> > > >>> > >> > bigtop
> > > >>> > >> > > >> dev
> > > >>> > >> > > >> > > >> mailing
> > > >>> > >> > > >> > > >> >> > list
> > > >>> > >> > > >> > > >> >> > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also
> created a
> > > >>> Jira
> > > >>> > for
> > > >>> > >> the
> > > >>> > >> > > >> same.
> > > >>> > >> > > >> > > For
> > > >>> > >> > > >> > > >> >> this
> > > >>> > >> > > >> > > >> >> > > > > > communities
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is the
> > bigtop
> > > >>> Jira:
> > > >>> > >> > > >> > BIGTOP-2313.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at
> 1:13
> > > AM,
> > > >>> > >> Thomas
> > > >>> > >> > > >> Weise <
> > > >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything
> under
> > > >>> > >> > prerequisites
> > > >>> > >> > > >> that
> > > >>> > >> > > >> > > >> would
> > > >>> > >> > > >> > > >> >> be a
> > > >>> > >> > > >> > > >> >> > > > > > problem.
> > > >>> > >> > > >> > > >> >> > > > > > > We
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
> > > >>> compatibility
> > > >>> > >> as
> > > >>> > >> > > part
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > >> >> becoming
> > > >>> > >> > > >> > > >> >> > an
> > > >>> > >> > > >> > > >> >> > > > > > > incubator
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the
> > > packaging
> > > >>> > during
> > > >>> > >> > the
> > > >>> > >> > > >> next
> > > >>> > >> > > >> > > >> weeks.
> > > >>> > >> > > >> > > >> >> > Since
> > > >>> > >> > > >> > > >> >> > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > work
> > > >>> > >> > > >> > > >> >> > > > > > > > > will
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related
> > > >>> > discussions
> > > >>> > >> and
> > > >>> > >> > > >> JIRAs
> > > >>> > >> > > >> > > >> should
> > > >>> > >> > > >> > > >> >> > also
> > > >>> > >> > > >> > > >> >> > > be
> > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have
> the
> > > >>> > packaging
> > > >>> > >> in
> > > >>> > >> > > >> place
> > > >>> > >> > > >> > by
> > > >>> > >> > > >> > > >> end
> > > >>> > >> > > >> > > >> >> Feb.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at
> > 10:14
> > > >>> AM,
> > > >>> > >> > Chinmay
> > > >>> > >> > > >> > > >> Kolhatkar <
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com
> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work
> on
> > > >>> adding
> > > >>> > >> > Apache
> > > >>> > >> > > >> Apex
> > > >>> > >> > > >> > > as a
> > > >>> > >> > > >> > > >> >> > > component
> > > >>> > >> > > >> > > >> >> > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > > Apache
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the
> packaging
> > > >>> system
> > > >>> > for
> > > >>> > >> > the
> > > >>> > >> > > >> > Apache
> > > >>> > >> > > >> > > >> big
> > > >>> > >> > > >> > > >> >> data
> > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it,
> > > most
> > > >>> > >> recently
> > > >>> > >> > > EMR.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking
> > Jira
> > > >>> task
> > > >>> > in
> > > >>> > >> > > >> APEXCORE
> > > >>> > >> > > >> > for
> > > >>> > >> > > >> > > >> the
> > > >>> > >> > > >> > > >> >> > same:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >>
> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of
> > execution
> > > >>> is as
> > > >>> > >> > > follows:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
> > > >>> prerequisites*
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some
> > > hard
> > > >>> and
> > > >>> > >> soft
> > > >>> > >> > > >> > > expectation
> > > >>> > >> > > >> > > >> >> for
> > > >>> > >> > > >> > > >> >> > new
> > > >>> > >> > > >> > > >> >> > > > > > > > components
> > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into
> Bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of
> it:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> >
> > > >>> > >> > > >>
> > > >>> > >> > >
> > > >>> > >> >
> > > >>> > >>
> > > >>> >
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to
> be
> > > >>> > standard
> > > >>> > >> ASF
> > > >>> > >> > > >> based
> > > >>> > >> > > >> > > >> >> > > requirements,
> > > >>> > >> > > >> > > >> >> > > > > but
> > > >>> > >> > > >> > > >> >> > > > > > > few
> > > >>> > >> > > >> > > >> >> > > > > > > > > need
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects
> are
> > > >>> > expected
> > > >>> > >> to
> > > >>> > >> > be
> > > >>> > >> > > >> > > Licensed
> > > >>> > >> > > >> > > >> >> under
> > > >>> > >> > > >> > > >> >> > > > > Apache
> > > >>> > >> > > >> > > >> >> > > > > > > > > License,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
> > > >>> > >> dependencies
> > > >>> > >> > are
> > > >>> > >> > > >> > > expected
> > > >>> > >> > > >> > > >> >> to be
> > > >>> > >> > > >> > > >> >> > > > > > > compatible
> > > >>> > >> > > >> > > >> >> > > > > > > > > with
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under
> ASL
> > > 2.0
> > > >>> but
> > > >>> > >> need
> > > >>> > >> > to
> > > >>> > >> > > >> > check
> > > >>> > >> > > >> > > if
> > > >>> > >> > > >> > > >> >> > > > > dependencies
> > > >>> > >> > > >> > > >> >> > > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL
> 2.0.
> > > >>> This I
> > > >>> > >> guess
> > > >>> > >> > > >> would
> > > >>> > >> > > >> > > be a
> > > >>> > >> > > >> > > >> >> > > > > verification
> > > >>> > >> > > >> > > >> >> > > > > > > > check.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects
> are
> > > >>> > expected
> > > >>> > >> to
> > > >>> > >> > be
> > > >>> > >> > > >> > > >> compatible
> > > >>> > >> > > >> > > >> >> > with
> > > >>> > >> > > >> > > >> >> > > > all
> > > >>> > >> > > >> > > >> >> > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms
> that
> > > >>> Bigtop
> > > >>> > >> > > >> distribution
> > > >>> > >> > > >> > > is
> > > >>> > >> > > >> > > >> >> > > targeting
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > verifying
> > > >>> > whether
> > > >>> > >> > our
> > > >>> > >> > > >> > > software
> > > >>> > >> > > >> > > >> >> runs
> > > >>> > >> > > >> > > >> >> > > fine
> > > >>> > >> > > >> > > >> >> > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
> > > >>> ubuntu-14.04
> > > >>> > >> > > debian-8
> > > >>> > >> > > >> > > >> >> > opensuse-13.2.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests
> that
> > > >>> should
> > > >>> > be
> > > >>> > >> > > added
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > >> >> > deployment.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the
> test
> > > >>> > artifacts
> > > >>> > >> > which
> > > >>> > >> > > >> goes
> > > >>> > >> > > >> > > >> beyond
> > > >>> > >> > > >> > > >> >> > > smoke
> > > >>> > >> > > >> > > >> >> > > > > test
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are
> > basically
> > > >>> the
> > > >>> > >> > > integration
> > > >>> > >> > > >> > > tests
> > > >>> > >> > > >> > > >> for
> > > >>> > >> > > >> > > >> >> > > > > > verification
> > > >>> > >> > > >> > > >> >> > > > > > > > > after
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a
> > soft
> > > >>> > >> > requirement,
> > > >>> > >> > > >> but
> > > >>> > >> > > >> > > aim
> > > >>> > >> > > >> > > >> is
> > > >>> > >> > > >> > > >> >> to
> > > >>> > >> > > >> > > >> >> > > > > achieve
> > > >>> > >> > > >> > > >> >> > > > > > > this
> > > >>> > >> > > >> > > >> >> > > > > > > > > as
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have
> > > >>> explanation why
> > > >>> > >> not
> > > >>> > >> > to
> > > >>> > >> > > >> > > include.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any from
> > the
> > > >>> link
> > > >>> > >> which
> > > >>> > >> > > >> > > explicitly
> > > >>> > >> > > >> > > >> >> needs
> > > >>> > >> > > >> > > >> >> > to
> > > >>> > >> > > >> > > >> >> > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > > checked
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please
> let
> > > us
> > > >>> > know.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex
> as
> > > >>> > component
> > > >>> > >> to
> > > >>> > >> > > >> Bigtop*
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail
> > > >>> archive of
> > > >>> > >> > Bigtop,
> > > >>> > >> > > >> it
> > > >>> > >> > > >> > was
> > > >>> > >> > > >> > > >> >> learnt
> > > >>> > >> > > >> > > >> >> > > that
> > > >>> > >> > > >> > > >> >> > > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community want to see
> > the
> > > >>> > >> addition of
> > > >>> > >> > > new
> > > >>> > >> > > >> > > >> >> components
> > > >>> > >> > > >> > > >> >> > in
> > > >>> > >> > > >> > > >> >> > > > > > phases.
> > > >>> > >> > > >> > > >> >> > > > > > > > Here
> > > >>> > >> > > >> > > >> >> > > > > > > > > > are
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > creating
> > > of
> > > >>> > >> package
> > > >>> > >> > > i.e.
> > > >>> > >> > > >> > rpm
> > > >>> > >> > > >> > > &
> > > >>> > >> > > >> > > >> deb
> > > >>> > >> > > >> > > >> >> > > files.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > > >>> documentations/READMEs,
> > > >>> > >> > LICENSE,
> > > >>> > >> > > >> > > >> DISCLAMER,
> > > >>> > >> > > >> > > >> >> > NOTES
> > > >>> > >> > > >> > > >> >> > > > etc
> > > >>> > >> > > >> > > >> >> > > > > if
> > > >>> > >> > > >> > > >> >> > > > > > > any
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> documentation
> > > that
> > > >>> > need
> > > >>> > >> to
> > > >>> > >> > be
> > > >>> > >> > > >> > added
> > > >>> > >> > > >> > > to
> > > >>> > >> > > >> > > >> >> > > > > distribution
> > > >>> > >> > > >> > > >> >> > > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > > our
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license
> > > >>> information of
> > > >>> > >> > > >> > dependencies
> > > >>> > >> > > >> > > >> >> required
> > > >>> > >> > > >> > > >> >> > > to
> > > >>> > >> > > >> > > >> >> > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > added
> > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at
> very
> > > >>> least)
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke
> test
> > > for
> > > >>> > >> > packaging.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for
> > > >>> automatic
> > > >>> > >> > > >> deployment
> > > >>> > >> > > >> > and
> > > >>> > >> > > >> > > >> >> > > > configuration
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet
> recipes
> > > for
> > > >>> > >> > automatic
> > > >>> > >> > > >> > > >> deployment
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For verification
> > of
> > > >>> > >> > deployments.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat'
> to
> > > make
> > > >>> > sure
> > > >>> > >> all
> > > >>> > >> > > new
> > > >>> > >> > > >> > > stuff
> > > >>> > >> > > >> > > >> is
> > > >>> > >> > > >> > > >> >> > > > compliant
> > > >>> > >> > > >> > > >> >> > > > > > > with
> > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license requirements.
> If
> > > >>> you add
> > > >>> > >> code
> > > >>> > >> > > >> > licenses
> > > >>> > >> > > >> > > >> >> under
> > > >>> > >> > > >> > > >> >> > > > > different
> > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need to be
> > > >>> listed in
> > > >>> > >> the
> > > >>> > >> > > >> NOTICE.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your
> > thoughts
> > > >>> on
> > > >>> > the
> > > >>> > >> > > >> approach.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
> > corresponding
> > > >>> > >> > communication
> > > >>> > >> > > >> on
> > > >>> > >> > > >> > > >> bigtop
> > > >>> > >> > > >> > > >> >> > > mailing
> > > >>> > >> > > >> > > >> >> > > > > list
> > > >>> > >> > > >> > > >> >> > > > > > > as
> > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some specific
> > > >>> > >> > > >> questions/suggestions
> > > >>> > >> > > >> > > >> >> related to
> > > >>> > >> > > >> > > >> >> > > > what
> > > >>> > >> > > >> > > >> >> > > > > > > should
> > > >>> > >> > > >> > > >> >> > > > > > > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the package
> > and
> > > >>> what
> > > >>> > >> > should
> > > >>> > >> > > be
> > > >>> > >> > > >> > the
> > > >>> > >> > > >> > > >> smoke
> > > >>> > >> > > >> > > >> >> > > tests,
> > > >>> > >> > > >> > > >> >> > > > > but
> > > >>> > >> > > >> > > >> >> > > > > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not having
> > too
> > > >>> much
> > > >>> > >> > content
> > > >>> > >> > > >> > here,
> > > >>> > >> > > >> > > >> we'll
> > > >>> > >> > > >> > > >> >> > put
> > > >>> > >> > > >> > > >> >> > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > questions
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this
> > > >>> > mailthread.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> >
> > > >>> > >> > > >>
> > > >>> > >> > > >
> > > >>> > >> > > >
> > > >>> > >> > >
> > > >>> > >> >
> > > >>> > >>
> > > >>> > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Pradeep Kumbhar <pr...@datatorrent.com>]

Added couple of comments
<https://issues.apache.org/jira/browse/APEXCORE-331?focusedCommentId=15151959&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15151959>
for
apex man page on JIRA. Please review.

On Thu, Feb 18, 2016 at 1:37 PM, Tushar Gosavi <tu...@datatorrent.com>
wrote:

> In following statement, can you replace dtGateway with ApplicationMaster
> and Yarn. or may be remove this statement entirely.
> apex is a wrapper around the REST API provided by dtGatway,
>
> - Tushar.
>
>
>
>
> On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Hi,
> >
> > I've attached man page for apex in Jira. Here is the link to it:
> > https://issues.apache.org/jira/secure/attachment/12788403/apex.1
> >
> > Please view it by: man -l apex.1
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > Hi Chinmay,
> > >
> > > I am not able to find the attachment, may be attachments are not
> allowed
> > in
> > > the mailing list. Can you attach the man page to the Jira?
> > >
> > > - Tushar.
> > >
> > >
> > > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <chinmay@apache.org
> >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I've attached man page file for the purpose of bigtop integration.
> > > > Please have a look and let me know if there any any feedback about
> it.
> > > >
> > > > You can see the man page content of this file with:
> > > > man -l apex.1
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > >
> > > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> > chinmay@apache.org>
> > > > wrote:
> > > >
> > > >> Thanks. Will follow bigtop recommendation. i.e. remove
> "-incubating".
> > > >>
> > > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <
> thomas@datatorrent.com
> > >
> > > >> wrote:
> > > >>
> > > >>> Please follow the recommendation and continue discussion on the
> > Bigtop
> > > >>> list.
> > > >>>
> > > >>> --
> > > >>> sent from mobile
> > > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <ch...@apache.org>
> > > wrote:
> > > >>>
> > > >>> > I got a recommendation from bigtop community to drop
> "-incubating"
> > > from
> > > >>> > version.
> > > >>> >
> > > >>> > Here is the mail thread:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> > > >>> >
> > > >>> > Please let me know if removing "-incubating" is fine.
> > > >>> >
> > > >>> > Thanks,
> > > >>> > Chinmay.
> > > >>> >
> > > >>> >
> > > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> > > chinmay@apache.org
> > > >>> >
> > > >>> > wrote:
> > > >>> >
> > > >>> > > I've already started communication there.
> > > >>> > >
> > > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> > > >>> thomas@datatorrent.com>
> > > >>> > > wrote:
> > > >>> > >
> > > >>> > >> This is a question you should ask on the Bigtop mailing list.
> > > >>> > >>
> > > >>> > >> --
> > > >>> > >> sent from mobile
> > > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org>
> > > >>> wrote:
> > > >>> > >>
> > > >>> > >> > Yes. In case of RPM, the only restriction on version is that
> > it
> > > >>> can
> > > >>> > not
> > > >>> > >> > contain '-'.
> > > >>> > >> > However, as a workaround, one can use 'incubating' as a part
> > of
> > > >>> > >> %{release}.
> > > >>> > >> >
> > > >>> > >> > Version: 3.3.0
> > > >>> > >> > Release: incubating
> > > >>> > >> >
> > > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> > > >>> > >> >
> > > >>> > >> > It is the same practice followed for pre-release packages
> e.g.
> > > >>> with
> > > >>> > rcX,
> > > >>> > >> > alphaX, betaX being used in %{release}.
> > > >>> > >> >
> > > >>> > >> > Regards,
> > > >>> > >> > Pradeep A. Dalvi
> > > >>> > >> >
> > > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> > > >>> > chinmay@apache.org>
> > > >>> > >> > wrote:
> > > >>> > >> >
> > > >>> > >> > > Hi Everyone,
> > > >>> > >> > >
> > > >>> > >> > > In one of the previous mails in this thread, we decided
> that
> > > >>> version
> > > >>> > >> of
> > > >>> > >> > the
> > > >>> > >> > > package will be "3.3.0-incubating".
> > > >>> > >> > > The problem that I'm facing is in RPM base version "-" is
> > not
> > > >>> > allowed.
> > > >>> > >> > >
> > > >>> > >> > > Can we have package version to "3.3.0"?
> > > >>> > >> > > i.e. the packages created will be as follows:
> > > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> > > >>> > >> > >
> > > >>> > >> > > There is another integration for Zepplin in Bigtop which
> is
> > > >>> > incubating
> > > >>> > >> > and
> > > >>> > >> > > is having base version as "0.5.6".
> > > >>> > >> > >
> > > >>> > >> > > Please let me know your thoughts.
> > > >>> > >> > >
> > > >>> > >> > > Thanks,
> > > >>> > >> > > Chinmay.
> > > >>> > >> > >
> > > >>> > >> > >
> > > >>> > >> > >
> > > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> > > >>> > >> chinmay@apache.org>
> > > >>> > >> > > wrote:
> > > >>> > >> > >
> > > >>> > >> > > > Thanks everyone for the feedback.
> > > >>> > >> > > >
> > > >>> > >> > > > Also as Ram and David mentioned, I'll start creating the
> > man
> > > >>> page
> > > >>> > as
> > > >>> > >> > > well.
> > > >>> > >> > > >
> > > >>> > >> > > > Thanks,
> > > >>> > >> > > > Chinmay.
> > > >>> > >> > > >
> > > >>> > >> > > >
> > > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> > > >>> > >> thomas@datatorrent.com>
> > > >>> > >> > > > wrote:
> > > >>> > >> > > >
> > > >>> > >> > > >> I think we already agreed that we will retain the
> current
> > > >>> script
> > > >>> > >> name
> > > >>> > >> > in
> > > >>> > >> > > >> this bigtop iteration. There is a ticket for renaming,
> > add
> > > >>> you
> > > >>> > >> > > suggestions
> > > >>> > >> > > >> there.
> > > >>> > >> > > >>
> > > >>> > >> > > >> --
> > > >>> > >> > > >> sent from mobile
> > > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > > >>> > sandesh@datatorrent.com>
> > > >>> > >> > > wrote:
> > > >>> > >> > > >>
> > > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start the
> > transition
> > > >>> > >> process by
> > > >>> > >> > > >> making
> > > >>> > >> > > >> > dtcli as the symlink to apex cli.
> > > >>> > >> > > >> >
> > > >>> > >> > > >> > Renaming is a breaking change should be done during
> > major
> > > >>> > version
> > > >>> > >> > > number
> > > >>> > >> > > >> > change.
> > > >>> > >> > > >> >
> > > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> > > >>> > david@datatorrent.com
> > > >>> > >> >
> > > >>> > >> > > >> wrote:
> > > >>> > >> > > >> >
> > > >>> > >> > > >> > > Apologies for hitting the "Send" button too early.
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > What is the scope of the name change?  If we simply
> > > have
> > > >>> a
> > > >>> > >> symlink
> > > >>> > >> > > >> from
> > > >>> > >> > > >> > > apex to dtcli, then it should have no backward
> > > >>> compatibility
> > > >>> > >> > problem
> > > >>> > >> > > >> at
> > > >>> > >> > > >> > > all.
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > But the proper name change would involve:
> > > >>> > >> > > >> > > - Change the name of DTCli class to ApexCli
> > > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> > > >>> > >> > > >> > > - Change the initial greeting when running the CLI
> > from
> > > >>> > >> something
> > > >>> > >> > > like
> > > >>> > >> > > >> > "DT
> > > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > The above changes may break existing scripts that
> > rely
> > > >>> on the
> > > >>> > >> the
> > > >>> > >> > > >> string
> > > >>> > >> > > >> > > "DT".
> > > >>> > >> > > >> > > But if folks are okay with this potential problem,
> I
> > am
> > > >>> okay
> > > >>> > >> with
> > > >>> > >> > > the
> > > >>> > >> > > >> > name
> > > >>> > >> > > >> > > change.
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > For the man page, we can start with the content of
> > this
> > > >>> page:
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > Plus the command line options when doing dtcli -h:
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > usage: DTCli
> > > >>> > >> > > >> > >  -e <arg>    Commands are read from the argument
> > > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at all time
> > > >>> > >> > > >> > >  -h          Print this help
> > > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos principal
> > > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> > > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> > > >>> > >> > > >> > >  -r          JSON Raw mode
> > > >>> > >> > > >> > >  -v          Verbose mode level 1
> > > >>> > >> > > >> > >  -vv         Verbose mode level 2
> > > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> > > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > David
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> > > >>> > >> david@datatorrent.com
> > > >>> > >> > >
> > > >>> > >> > > >> > wrote:
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > > What is the scope of the name change?  If we
> simply
> > > >>> have a
> > > >>> > >> > symlink
> > > >>> > >> > > >> from
> > > >>> > >> > > >> > > > apex to dtcli, then it should have no backward
> > > >>> > compatibility
> > > >>> > >> > > >> problem at
> > > >>> > >> > > >> > > all.
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > But the proper name change would involve:
> > > >>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
> > > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> > > >>> > >> > > >> > > > - Change the initial greeting when running the
> CLI
> > > from
> > > >>> > >> > something
> > > >>> > >> > > >> like
> > > >>> > >> > > >> > > "DT
> > > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > The above changes may break existing scripts that
> > > rely
> > > >>> on
> > > >>> > the
> > > >>> > >> > the
> > > >>> > >> > > >> > string
> > > >>> > >> > > >> > > > "DT".
> > > >>> > >> > > >> > > > But if folks are okay with this potential
> problem,
> > I
> > > am
> > > >>> > okay
> > > >>> > >> > with
> > > >>> > >> > > >> the
> > > >>> > >> > > >> > > name
> > > >>> > >> > > >> > > > change.
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > For the man page, we can start with the content
> of
> > > this
> > > >>> > page:
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay
> Kolhatkar
> > <
> > > >>> > >> > > >> chinmay@apache.org
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> > > > wrote:
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >> Here is a suggestion I have related to name
> > change:
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> 1. While packaging we change the name of "dtcli"
> > > >>> script to
> > > >>> > >> > > "apex".
> > > >>> > >> > > >> > This
> > > >>> > >> > > >> > > is
> > > >>> > >> > > >> > > >> done so that in future when we change the name
> to
> > > >>> "apex",
> > > >>> > >> users
> > > >>> > >> > > of
> > > >>> > >> > > >> > > bigtop
> > > >>> > >> > > >> > > >> apex don't have to transition much.
> > > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli which
> points
> > > to
> > > >>> apex
> > > >>> > >> > > script.
> > > >>> > >> > > >> > This
> > > >>> > >> > > >> > > is
> > > >>> > >> > > >> > > >> for backward compatibility.
> > > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first
> > > integration
> > > >>> in
> > > >>> > >> > bigtop.
> > > >>> > >> > > >> This
> > > >>> > >> > > >> > is
> > > >>> > >> > > >> > > >> in
> > > >>> > >> > > >> > > >> the interest of not changing the content of
> dtcli
> > or
> > > >>> apex
> > > >>> > >> > script.
> > > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these
> naming
> > > >>> related
> > > >>> > >> > > changes
> > > >>> > >> > > >> and
> > > >>> > >> > > >> > > >> update bigtop repository later to remove
> > references
> > > to
> > > >>> > dtcli
> > > >>> > >> > all
> > > >>> > >> > > >> > > together.
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> Please share your thoughts on above approach.
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> Also, please share what could be the man page
> > > content
> > > >>> for
> > > >>> > >> > "apex".
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> Thanks,
> > > >>> > >> > > >> > > >> Chinmay.
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay
> > Kolhatkar <
> > > >>> > >> > > >> > chinmay@apache.org>
> > > >>> > >> > > >> > > >> wrote:
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >> > Yes.. It'll be build from source tar
> downloaded
> > > >>> from one
> > > >>> > >> of
> > > >>> > >> > the
> > > >>> > >> > > >> > apache
> > > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced from
> > > dtcli.
> > > >>> > This
> > > >>> > >> > > leaves
> > > >>> > >> > > >> us
> > > >>> > >> > > >> > 3
> > > >>> > >> > > >> > > >> > options:
> > > >>> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli
> would
> > > >>> need a
> > > >>> > >> > change
> > > >>> > >> > > >> while
> > > >>> > >> > > >> > > >> > packaging.
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> > But if we want to use dtcli as it is, then we
> > > would
> > > >>> need
> > > >>> > >> > > >> dt-env.sh
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise
> <
> > > >>> > >> > > >> > thomas@datatorrent.com
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >> > wrote:
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >> Looks good overall, though there shouldn't be
> > any
> > > >>> > >> dt-env.sh
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >> What will the bigtop package be built from,
> the
> > > >>> source
> > > >>> > >> tar?
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay
> > > Kolhatkar
> > > >>> <
> > > >>> > >> > > >> > > chinmay@apache.org
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >> wrote:
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >> > Hi Thomas,
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > First, we're not changing the name
> anywhere.
> > > >>> We'll
> > > >>> > >> follow
> > > >>> > >> > > what
> > > >>> > >> > > >> > > >> >> currently is
> > > >>> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating
> > version.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory
> > structure
> > > >>> below
> > > >>> > >> which
> > > >>> > >> > > is
> > > >>> > >> > > >> > > inline
> > > >>> > >> > > >> > > >> >> with
> > > >>> > >> > > >> > > >> >> > other existing integrations with bigtop.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > The need for each file is as follows:
> > > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> > > >>> > >> > > >> engine/src/main/scripts/
> > > >>> > >> > > >> > > of
> > > >>> > >> > > >> > > >> >> source
> > > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from
> > dtcli.
> > > >>> This
> > > >>> > >> > > searches
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > >> hadoop
> > > >>> > >> > > >> > > >> >> > binary path and exports an env variable for
> > > >>> dtcli to
> > > >>> > >> use
> > > >>> > >> > it.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is
> > > >>> extracted
> > > >>> > >> in a
> > > >>> > >> > > >> similar
> > > >>> > >> > > >> > > >> way to
> > > >>> > >> > > >> > > >> >> > how DT Community Edition finds dependency
> > jars.
> > > >>> > >> > > >> > > >> >> > Please note that this is the first
> iteration
> > > >>> list of
> > > >>> > >> jars.
> > > >>> > >> > > I'm
> > > >>> > >> > > >> > > >> trying to
> > > >>> > >> > > >> > > >> >> > narrow this down to only those which are
> > really
> > > >>> > >> required.
> > > >>> > >> > > >> > > >> >> > The test that I'm running to check if dtcli
> > > runs
> > > >>> fine
> > > >>> > >> with
> > > >>> > >> > > >> given
> > > >>> > >> > > >> > > >> >> dependency
> > > >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi
> > demo.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Above is the directory structure which is
> > > >>> required
> > > >>> > for
> > > >>> > >> CLI
> > > >>> > >> > > to
> > > >>> > >> > > >> > work.
> > > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned
> are
> > > some
> > > >>> > >> > additional
> > > >>> > >> > > >> > files
> > > >>> > >> > > >> > > >> which
> > > >>> > >> > > >> > > >> >> > makes the like of administrator easier. For
> > eg.
> > > >>> > >> /etc/skel
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Please let me know if this is inline with
> > what
> > > >>> you're
> > > >>> > >> > > >> thinking.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > Thanks,
> > > >>> > >> > > >> > > >> >> > Chinmay.
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > .
> > > >>> > >> > > >> > > >> >> > |-- bin
> > > >>> > >> > > >> > > >> >> > |   `-- dtcli
> > > >>> > >> > > >> > > >> >> > |-- conf
> > > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > > >>> > >> > > >> > > >> >> > `-- lib
> > > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > > >>> > >> > > >> > > >> >> >     |--
> > apex-bufferserver-3.3.0-incubating.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > > >>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > > >>> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > > >>> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |--
> jetty-websocket-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > > >>> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > > >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas
> > Weise <
> > > >>> > >> > > >> > > >> thomas@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > wrote:
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >> > > Chinmay,
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > Before discussing where to put the files,
> > > let's
> > > >>> > make
> > > >>> > >> > sure
> > > >>> > >> > > >> they
> > > >>> > >> > > >> > > are
> > > >>> > >> > > >> > > >> >> really
> > > >>> > >> > > >> > > >> >> > > needed for the operation of the CLI. As
> for
> > > >>> names,
> > > >>> > >> > > anything
> > > >>> > >> > > >> new
> > > >>> > >> > > >> > > >> needs
> > > >>> > >> > > >> > > >> >> to
> > > >>> > >> > > >> > > >> >> > > reflect Apex in the name and should
> follow
> > > >>> common
> > > >>> > >> > > >> conventions,
> > > >>> > >> > > >> > > >> >> especially
> > > >>> > >> > > >> > > >> >> > > in Bigtop where there are many existing
> > > >>> > integrations
> > > >>> > >> to
> > > >>> > >> > > look
> > > >>> > >> > > >> > at.
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > Thanks,
> > > >>> > >> > > >> > > >> >> > > Thomas
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay
> > > >>> Kolhatkar
> > > >>> > <
> > > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > wrote:
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> > > > Really good points Pradeep and
> Aniruddha.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to change
> the
> > > >>> dtcli
> > > >>> > >> > > >> considering it
> > > >>> > >> > > >> > > >> works
> > > >>> > >> > > >> > > >> >> > with
> > > >>> > >> > > >> > > >> >> > > DT
> > > >>> > >> > > >> > > >> >> > > > Community edition. We can keep the
> > > directory
> > > >>> > >> structure
> > > >>> > >> > > >> > similar
> > > >>> > >> > > >> > > to
> > > >>> > >> > > >> > > >> >> that.
> > > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which contains
> > > >>> > information
> > > >>> > >> > > >> required
> > > >>> > >> > > >> > for
> > > >>> > >> > > >> > > >> >> dtcli
> > > >>> > >> > > >> > > >> >> > to
> > > >>> > >> > > >> > > >> >> > > > launch.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop community
> > that
> > > >>> > whether
> > > >>> > >> > they
> > > >>> > >> > > >> > > >> facilitate
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > > > installation of rpms/debs before user
> is
> > > >>> created.
> > > >>> > >> In
> > > >>> > >> > > >> either
> > > >>> > >> > > >> > > case,
> > > >>> > >> > > >> > > >> >> > current
> > > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home
> > > directory.
> > > >>> > Also
> > > >>> > >> > > before
> > > >>> > >> > > >> > > putting
> > > >>> > >> > > >> > > >> >> > > anything
> > > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define what are
> > the
> > > >>> > default
> > > >>> > >> > > >> contents
> > > >>> > >> > > >> > > that
> > > >>> > >> > > >> > > >> >> > should
> > > >>> > >> > > >> > > >> >> > > go
> > > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no
> > defaults,
> > > >>> > >> probably we
> > > >>> > >> > > >> should
> > > >>> > >> > > >> > > not
> > > >>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice.
> > > This
> > > >>> way
> > > >>> > >> > > contents
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > >> >> dt-env.sh
> > > >>> > >> > > >> > > >> >> > > are
> > > >>> > >> > > >> > > >> >> > > > present as env variables. But I see a
> > catch
> > > >>> > there.
> > > >>> > >> > > Adding
> > > >>> > >> > > >> > > >> dt-env.sh
> > > >>> > >> > > >> > > >> >> to
> > > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the
> > variables
> > > >>> > >> available
> > > >>> > >> > at
> > > >>> > >> > > >> > > runtime
> > > >>> > >> > > >> > > >> all
> > > >>> > >> > > >> > > >> >> > the
> > > >>> > >> > > >> > > >> >> > > > time. I feel a little skeptical about
> > that.
> > > >>> > Maybe a
> > > >>> > >> > > >> possible
> > > >>> > >> > > >> > > >> >> collision
> > > >>> > >> > > >> > > >> >> > > can
> > > >>> > >> > > >> > > >> >> > > > occur with other application vars.
> > > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
> > > >>> > >> > "../conf/dt-env.sh"
> > > >>> > >> > > >> and
> > > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables
> are
> > > >>> anyway
> > > >>> > >> > > >> available.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is
> > > >>> happening
> > > >>> > at
> > > >>> > >> > > Bigtop
> > > >>> > >> > > >> > > >> >> > > Jira/repository.
> > > >>> > >> > > >> > > >> >> > > > Packaging related code usually goes
> > there.
> > > >>> > (That's
> > > >>> > >> > what
> > > >>> > >> > > >> all
> > > >>> > >> > > >> > the
> > > >>> > >> > > >> > > >> >> > > components
> > > >>> > >> > > >> > > >> >> > > > in bigtop does).
> > > >>> > >> > > >> > > >> >> > > > Having said that, once a PR is created,
> > > I'll
> > > >>> be
> > > >>> > >> > sharing
> > > >>> > >> > > >> the
> > > >>> > >> > > >> > > link
> > > >>> > >> > > >> > > >> of
> > > >>> > >> > > >> > > >> >> PR
> > > >>> > >> > > >> > > >> >> > > > here, so that apex community as well
> can
> > > >>> review
> > > >>> > it.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > Chinmay.
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM,
> > Aniruddha
> > > >>> > >> Thombare <
> > > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> > > >>> > >> > > >> > > >> >> > > > > We may need to iron out details about
> > > exact
> > > >>> > paths
> > > >>> > >> > etc.
> > > >>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR
> > comments.
> > > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep
> A.
> > > >>> Dalvi <
> > > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > > >>> > >> > > >> > > >> >> > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM,
> > > Chinmay
> > > >>> > >> > Kolhatkar <
> > > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present
> in
> > > DT
> > > >>> > >> community
> > > >>> > >> > > >> > edition
> > > >>> > >> > > >> > > >> will
> > > >>> > >> > > >> > > >> >> be
> > > >>> > >> > > >> > > >> >> > > > > included
> > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I found
> > from
> > > >>> > common
> > > >>> > >> > > >> > dependencies
> > > >>> > >> > > >> > > >> >> between
> > > >>> > >> > > >> > > >> >> > > of
> > > >>> > >> > > >> > > >> >> > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> > > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > > >>> > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > > >>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > >
> > jetty-websocket-8.1.10.v20130312.jar
> > > >>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > > >>> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some
> > tests
> > > do
> > > >>> > check
> > > >>> > >> > that
> > > >>> > >> > > >> > dtcli
> > > >>> > >> > > >> > > >> works
> > > >>> > >> > > >> > > >> >> > > > properly
> > > >>> > >> > > >> > > >> >> > > > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with
> > > only
> > > >>> > these
> > > >>> > >> > > >> libraries
> > > >>> > >> > > >> > > >> >> present in
> > > >>> > >> > > >> > > >> >> > > > > > isolation
> > > >>> > >> > > >> > > >> >> > > > > > > without dependency on local m2.
> > > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there are
> unwanted
> > > >>> jars
> > > >>> > >> which
> > > >>> > >> > are
> > > >>> > >> > > >> used
> > > >>> > >> > > >> > > for
> > > >>> > >> > > >> > > >> >> > compile
> > > >>> > >> > > >> > > >> >> > > > > time
> > > >>> > >> > > >> > > >> >> > > > > > > dependency and not runtime in
> above
> > > >>> list
> > > >>> > >> which I
> > > >>> > >> > > can
> > > >>> > >> > > >> > drop
> > > >>> > >> > > >> > > >> to
> > > >>> > >> > > >> > > >> >> keep
> > > >>> > >> > > >> > > >> >> > > > > package
> > > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if
> all
> > > of
> > > >>> the
> > > >>> > >> > > >> dt-env.sh is
> > > >>> > >> > > >> > > >> >> required.
> > > >>> > >> > > >> > > >> >> > > > Only
> > > >>> > >> > > >> > > >> >> > > > > > > required values I see in it are
> > > >>> classpath.
> > > >>> > >> and
> > > >>> > >> > > >> dtcli is
> > > >>> > >> > > >> > > >> >> capable
> > > >>> > >> > > >> > > >> >> > of
> > > >>> > >> > > >> > > >> >> > > > > > building
> > > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have
> > to
> > > >>> see
> > > >>> > how
> > > >>> > >> > dtcli
> > > >>> > >> > > >> can
> > > >>> > >> > > >> > > find
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > jar
> > > >>> > >> > > >> > > >> >> > > > > path,
> > > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf
> files.
> > If
> > > >>> we
> > > >>> > >> need a
> > > >>> > >> > > >> change
> > > >>> > >> > > >> > in
> > > >>> > >> > > >> > > >> >> dtcli,
> > > >>> > >> > > >> > > >> >> > > then
> > > >>> > >> > > >> > > >> >> > > > > > > community's opinion is required
> for
> > > >>> whether
> > > >>> > >> > dtcli
> > > >>> > >> > > >> > should
> > > >>> > >> > > >> > > >> >> change
> > > >>> > >> > > >> > > >> >> > in
> > > >>> > >> > > >> > > >> >> > > > our
> > > >>> > >> > > >> > > >> >> > > > > > repo
> > > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with required
> > > changes
> > > >>> > >> exist in
> > > >>> > >> > > >> bigtop
> > > >>> > >> > > >> > > >> repo
> > > >>> > >> > > >> > > >> >> > until
> > > >>> > >> > > >> > > >> >> > > we
> > > >>> > >> > > >> > > >> >> > > > > > make
> > > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Can we have these file paths set
> > using
> > > >>> > >> environment
> > > >>> > >> > > >> > > variables
> > > >>> > >> > > >> > > >> >> with
> > > >>> > >> > > >> > > >> >> > > some
> > > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> > > >>> > >> > > >> > > >> >> > > > > > Then we can set such params in
> > > dt-env.sh.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this
> file
> > > >>> > different
> > > >>> > >> > from
> > > >>> > >> > > >> > > >> >> dt-site.xml?
> > > >>> > >> > > >> > > >> >> > Not
> > > >>> > >> > > >> > > >> >> > > > > sure
> > > >>> > >> > > >> > > >> >> > > > > > if
> > > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help.
> > > Files &
> > > >>> > Dirs
> > > >>> > >> > from
> > > >>> > >> > > >> > > /etc/skel
> > > >>> > >> > > >> > > >> >> are
> > > >>> > >> > > >> > > >> >> > > > copied
> > > >>> > >> > > >> > > >> >> > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > home of new user when useradd
> > program
> > > >>> is
> > > >>> > >> called.
> > > >>> > >> > > But
> > > >>> > >> > > >> > for
> > > >>> > >> > > >> > > >> >> existing
> > > >>> > >> > > >> > > >> >> > > > users
> > > >>> > >> > > >> > > >> >> > > > > > > this won't be of any use.
> Moreover,
> > > >>> dtcli
> > > >>> > >> > creates
> > > >>> > >> > > >> > ~/.dt/
> > > >>> > >> > > >> > > on
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > fly
> > > >>> > >> > > >> > > >> >> > > > if
> > > >>> > >> > > >> > > >> >> > > > > > not
> > > >>> > >> > > >> > > >> >> > > > > > > encountered for the first time.
> > Again
> > > >>> > >> correct me
> > > >>> > >> > > if
> > > >>> > >> > > >> I'm
> > > >>> > >> > > >> > > >> wrong.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Yes, you are right. However this
> > would
> > > >>> > >> probably be
> > > >>> > >> > > >> > > necessary
> > > >>> > >> > > >> > > >> >> step
> > > >>> > >> > > >> > > >> >> > for
> > > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get
> > installed
> > > >>> > during
> > > >>> > >> OS
> > > >>> > >> > > >> install
> > > >>> > >> > > >> > > and
> > > >>> > >> > > >> > > >> >> > before
> > > >>> > >> > > >> > > >> >> > > > user
> > > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile
> > ->
> > > >>> I'm
> > > >>> > not
> > > >>> > >> > sure
> > > >>> > >> > > >> its
> > > >>> > >> > > >> > the
> > > >>> > >> > > >> > > >> best
> > > >>> > >> > > >> > > >> >> > > idea.
> > > >>> > >> > > >> > > >> >> > > > > > We'll
> > > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location
> which
> > is
> > > >>> by
> > > >>> > >> default
> > > >>> > >> > > >> > present
> > > >>> > >> > > >> > > in
> > > >>> > >> > > >> > > >> >> path
> > > >>> > >> > > >> > > >> >> > > i.e.
> > > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > For other env variables specific to
> > > apex,
> > > >>> > I'll
> > > >>> > >> > > prefer
> > > >>> > >> > > >> to
> > > >>> > >> > > >> > > use
> > > >>> > >> > > >> > > >> >> > > dt-env.sh
> > > >>> > >> > > >> > > >> >> > > > > and
> > > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather than
> > > changing
> > > >>> > >> bashrc
> > > >>> > >> > > >> etc...
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
> > > >>> /etc/profile.d/
> > > >>> > >> shall
> > > >>> > >> > > be
> > > >>> > >> > > >> the
> > > >>> > >> > > >> > > >> >> cleaner
> > > >>> > >> > > >> > > >> >> > way
> > > >>> > >> > > >> > > >> >> > > > to
> > > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM,
> > > >>> Aniruddha
> > > >>> > >> > > Thombare
> > > >>> > >> > > >> <
> > > >>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com>
> wrote:
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > We need to consider following:
> > > >>> > >> > > >> > > >> >> > > > > > > > System wide default config
> files
> > > can
> > > >>> be
> > > >>> > >> > located
> > > >>> > >> > > at
> > > >>> > >> > > >> > > >> following
> > > >>> > >> > > >> > > >> >> > > > > locations:
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may
> have
> > > to
> > > >>> > change
> > > >>> > >> > > dtcli
> > > >>> > >> > > >> > > >> behaviour
> > > >>> > >> > > >> > > >> >> on
> > > >>> > >> > > >> > > >> >> > > how
> > > >>> > >> > > >> > > >> >> > > > it
> > > >>> > >> > > >> > > >> >> > > > > > > finds
> > > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> > > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may
> > have
> > > to
> > > >>> > >> change
> > > >>> > >> > > dtcli
> > > >>> > >> > > >> > > >> >> behaviour on
> > > >>> > >> > > >> > > >> >> > > how
> > > >>> > >> > > >> > > >> >> > > > > it
> > > >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
> > > >>> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and
> > > other
> > > >>> > files
> > > >>> > >> > (for
> > > >>> > >> > > >> new
> > > >>> > >> > > >> > > users
> > > >>> > >> > > >> > > >> >> that
> > > >>> > >> > > >> > > >> >> > > will
> > > >>> > >> > > >> > > >> >> > > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > > created in system in future.)
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > We may also have to modify
> > bashrc /
> > > >>> > >> > bashprofile
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > >> >> population
> > > >>> > >> > > >> > > >> >> > > any
> > > >>> > >> > > >> > > >> >> > > > > > > > variables that are required.
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your
> > comments /
> > > >>> > >> > suggestion.
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16
> AM,
> > > >>> Thomas
> > > >>> > >> Weise
> > > >>> > >> > <
> > > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > -->
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05
> AM,
> > > >>> Chinmay
> > > >>> > >> > > >> Kolhatkar <
> > > >>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging
> > > effort
> > > >>> > >> (rpm+deb)
> > > >>> > >> > > of
> > > >>> > >> > > >> > apex,
> > > >>> > >> > > >> > > >> here
> > > >>> > >> > > >> > > >> >> > are
> > > >>> > >> > > >> > > >> >> > > > some
> > > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> > > >>> > >> > > >> > > >> >> > > > > > > > > > about package structure
> etc..
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on
> bbigtop
> > > >>> mailing
> > > >>> > >> > list, I
> > > >>> > >> > > >> have
> > > >>> > >> > > >> > > >> some
> > > >>> > >> > > >> > > >> >> > > question
> > > >>> > >> > > >> > > >> >> > > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory
> structure
> > of
> > > >>> apex
> > > >>> > >> > package
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > both
> > > >>> > >> > > >> > > >> >> deb &
> > > >>> > >> > > >> > > >> >> > > > rpm:
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other
> > > >>> dependent
> > > >>> > >> jars>
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> > > >>> > /usr/lib/apex/bin/dtcli
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> /usr/share/doc/man/man1/dtcli.1.gz
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > /usr/share/doc/apex/license/LICENSE.txt
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> /usr/share/doc/apex/CHANGELOG
> > > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > <version> =
> 3.3.0-incubating.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> =
> All
> > > the
> > > >>> 3rd
> > > >>> > >> party
> > > >>> > >> > > >> jars
> > > >>> > >> > > >> > > which
> > > >>> > >> > > >> > > >> >> are
> > > >>> > >> > > >> > > >> >> > > > > required
> > > >>> > >> > > >> > > >> >> > > > > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the
> > > dependencies
> > > >>> are
> > > >>> > >> > > packaged
> > > >>> > >> > > >> as
> > > >>> > >> > > >> > > >> part of
> > > >>> > >> > > >> > > >> >> > > > rpm/deb
> > > >>> > >> > > >> > > >> >> > > > > by
> > > >>> > >> > > >> > > >> >> > > > > > > any
> > > >>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what those
> jars
> > > >>> are in
> > > >>> > >> > Bigtop
> > > >>> > >> > > >> > > context?
> > > >>> > >> > > >> > > >> >> Same
> > > >>> > >> > > >> > > >> >> > as
> > > >>> > >> > > >> > > >> >> > > > > > shipped
> > > >>> > >> > > >> > > >> >> > > > > > > > with
> > > >>> > >> > > >> > > >> >> > > > > > > > > DT community addition under
> > lib/
> > > ?
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> =
> Licenses
> > of
> > > >>> > >> > > corresponding
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > 3rd
> > > >>> > >> > > >> > > >> >> party
> > > >>> > >> > > >> > > >> >> > > jars
> > > >>> > >> > > >> > > >> >> > > > > > which
> > > >>> > >> > > >> > > >> >> > > > > > > > > needs
> > > >>> > >> > > >> > > >> >> > > > > > > > > > to included while
> packaging.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Questions related to this:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli
> of
> > > >>> apex as
> > > >>> > >> > "apex"
> > > >>> > >> > > >> > instead
> > > >>> > >> > > >> > > >> of
> > > >>> > >> > > >> > > >> >> > > "dtcli"
> > > >>> > >> > > >> > > >> >> > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> > > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > I think we should keep the
> name
> > > >>> until
> > > >>> > we
> > > >>> > >> are
> > > >>> > >> > > >> able
> > > >>> > >> > > >> > to
> > > >>> > >> > > >> > > >> >> change
> > > >>> > >> > > >> > > >> >> > it
> > > >>> > >> > > >> > > >> >> > > in
> > > >>> > >> > > >> > > >> >> > > > > > Apex.
> > > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get
> > confusing.
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares
> > in
> > > >>> bigtop
> > > >>> > >> have
> > > >>> > >> > > man
> > > >>> > >> > > >> > page
> > > >>> > >> > > >> > > >> for
> > > >>> > >> > > >> > > >> >> > their
> > > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > I think we should have it
> too
> > > for
> > > >>> > >> dtcli.
> > > >>> > >> > Is
> > > >>> > >> > > >> there
> > > >>> > >> > > >> > > any
> > > >>> > >> > > >> > > >> >> > > > > documentation
> > > >>> > >> > > >> > > >> >> > > > > > > > > which I
> > > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man page? or
> > can
> > > >>> I use
> > > >>> > >> > output
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > >> "dtcli
> > > >>> > >> > > >> > > >> >> > > --help"?
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call
> version
> > > of
> > > >>> apex
> > > >>> > >> in
> > > >>> > >> > > >> Bigtop
> > > >>> > >> > > >> > as
> > > >>> > >> > > >> > > >> >> 3.3.0 OR
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex
> package
> > of
> > > >>> > bigtop
> > > >>> > >> to
> > > >>> > >> > > >> depend
> > > >>> > >> > > >> > on
> > > >>> > >> > > >> > > >> >> 2.7.1
> > > >>> > >> > > >> > > >> >> > > > > version
> > > >>> > >> > > >> > > >> >> > > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems
> > > that
> > > >>> we
> > > >>> > see
> > > >>> > >> > with
> > > >>> > >> > > >> this
> > > >>> > >> > > >> > > >> >> > dependency?
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > Should work without changed.
> I
> > > >>> thought
> > > >>> > we
> > > >>> > >> > > >> certified
> > > >>> > >> > > >> > > >> >> against
> > > >>> > >> > > >> > > >> >> > > > 2.7.0?
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache
> > > mirror
> > > >>> > from
> > > >>> > >> > which
> > > >>> > >> > > >> > bigtop
> > > >>> > >> > > >> > > >> will
> > > >>> > >> > > >> > > >> >> > pick
> > > >>> > >> > > >> > > >> >> > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > source for compilation and
> > > >>> packaging.
> > > >>> > >> > Please
> > > >>> > >> > > >> > > correct
> > > >>> > >> > > >> > > >> if
> > > >>> > >> > > >> > > >> >> > > > > incorrect:
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > >
> http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE:
> apache.osuosl.org
> > is
> > > >>> the
> > > >>> > >> mirror
> > > >>> > >> > > >> used
> > > >>> > >> > > >> > by
> > > >>> > >> > > >> > > >> all
> > > >>> > >> > > >> > > >> >> the
> > > >>> > >> > > >> > > >> >> > > > > > softwares
> > > >>> > >> > > >> > > >> >> > > > > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Please share your opinion.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at
> 11:19
> > > AM,
> > > >>> > >> Chinmay
> > > >>> > >> > > >> > Kolhatkar
> > > >>> > >> > > >> > > <
> > > >>> > >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on
> > the
> > > >>> > >> packaging.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already started
> > > >>> discussion on
> > > >>> > >> > bigtop
> > > >>> > >> > > >> dev
> > > >>> > >> > > >> > > >> mailing
> > > >>> > >> > > >> > > >> >> > list
> > > >>> > >> > > >> > > >> >> > > > for
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also
> created a
> > > >>> Jira
> > > >>> > for
> > > >>> > >> the
> > > >>> > >> > > >> same.
> > > >>> > >> > > >> > > For
> > > >>> > >> > > >> > > >> >> this
> > > >>> > >> > > >> > > >> >> > > > > > communities
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is the
> > bigtop
> > > >>> Jira:
> > > >>> > >> > > >> > BIGTOP-2313.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at
> 1:13
> > > AM,
> > > >>> > >> Thomas
> > > >>> > >> > > >> Weise <
> > > >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything
> under
> > > >>> > >> > prerequisites
> > > >>> > >> > > >> that
> > > >>> > >> > > >> > > >> would
> > > >>> > >> > > >> > > >> >> be a
> > > >>> > >> > > >> > > >> >> > > > > > problem.
> > > >>> > >> > > >> > > >> >> > > > > > > We
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
> > > >>> compatibility
> > > >>> > >> as
> > > >>> > >> > > part
> > > >>> > >> > > >> of
> > > >>> > >> > > >> > > >> >> becoming
> > > >>> > >> > > >> > > >> >> > an
> > > >>> > >> > > >> > > >> >> > > > > > > incubator
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the
> > > packaging
> > > >>> > during
> > > >>> > >> > the
> > > >>> > >> > > >> next
> > > >>> > >> > > >> > > >> weeks.
> > > >>> > >> > > >> > > >> >> > Since
> > > >>> > >> > > >> > > >> >> > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > work
> > > >>> > >> > > >> > > >> >> > > > > > > > > will
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related
> > > >>> > discussions
> > > >>> > >> and
> > > >>> > >> > > >> JIRAs
> > > >>> > >> > > >> > > >> should
> > > >>> > >> > > >> > > >> >> > also
> > > >>> > >> > > >> > > >> >> > > be
> > > >>> > >> > > >> > > >> >> > > > > > > there.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have
> the
> > > >>> > packaging
> > > >>> > >> in
> > > >>> > >> > > >> place
> > > >>> > >> > > >> > by
> > > >>> > >> > > >> > > >> end
> > > >>> > >> > > >> > > >> >> Feb.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at
> > 10:14
> > > >>> AM,
> > > >>> > >> > Chinmay
> > > >>> > >> > > >> > > >> Kolhatkar <
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com
> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work
> on
> > > >>> adding
> > > >>> > >> > Apache
> > > >>> > >> > > >> Apex
> > > >>> > >> > > >> > > as a
> > > >>> > >> > > >> > > >> >> > > component
> > > >>> > >> > > >> > > >> >> > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > > Apache
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the
> packaging
> > > >>> system
> > > >>> > for
> > > >>> > >> > the
> > > >>> > >> > > >> > Apache
> > > >>> > >> > > >> > > >> big
> > > >>> > >> > > >> > > >> >> data
> > > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it,
> > > most
> > > >>> > >> recently
> > > >>> > >> > > EMR.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking
> > Jira
> > > >>> task
> > > >>> > in
> > > >>> > >> > > >> APEXCORE
> > > >>> > >> > > >> > for
> > > >>> > >> > > >> > > >> the
> > > >>> > >> > > >> > > >> >> > same:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >>
> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of
> > execution
> > > >>> is as
> > > >>> > >> > > follows:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
> > > >>> prerequisites*
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some
> > > hard
> > > >>> and
> > > >>> > >> soft
> > > >>> > >> > > >> > > expectation
> > > >>> > >> > > >> > > >> >> for
> > > >>> > >> > > >> > > >> >> > new
> > > >>> > >> > > >> > > >> >> > > > > > > > components
> > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into
> Bigtop.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of
> it:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> >
> > > >>> > >> > > >>
> > > >>> > >> > >
> > > >>> > >> >
> > > >>> > >>
> > > >>> >
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to
> be
> > > >>> > standard
> > > >>> > >> ASF
> > > >>> > >> > > >> based
> > > >>> > >> > > >> > > >> >> > > requirements,
> > > >>> > >> > > >> > > >> >> > > > > but
> > > >>> > >> > > >> > > >> >> > > > > > > few
> > > >>> > >> > > >> > > >> >> > > > > > > > > need
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects
> are
> > > >>> > expected
> > > >>> > >> to
> > > >>> > >> > be
> > > >>> > >> > > >> > > Licensed
> > > >>> > >> > > >> > > >> >> under
> > > >>> > >> > > >> > > >> >> > > > > Apache
> > > >>> > >> > > >> > > >> >> > > > > > > > > License,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
> > > >>> > >> dependencies
> > > >>> > >> > are
> > > >>> > >> > > >> > > expected
> > > >>> > >> > > >> > > >> >> to be
> > > >>> > >> > > >> > > >> >> > > > > > > compatible
> > > >>> > >> > > >> > > >> >> > > > > > > > > with
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under
> ASL
> > > 2.0
> > > >>> but
> > > >>> > >> need
> > > >>> > >> > to
> > > >>> > >> > > >> > check
> > > >>> > >> > > >> > > if
> > > >>> > >> > > >> > > >> >> > > > > dependencies
> > > >>> > >> > > >> > > >> >> > > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL
> 2.0.
> > > >>> This I
> > > >>> > >> guess
> > > >>> > >> > > >> would
> > > >>> > >> > > >> > > be a
> > > >>> > >> > > >> > > >> >> > > > > verification
> > > >>> > >> > > >> > > >> >> > > > > > > > check.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects
> are
> > > >>> > expected
> > > >>> > >> to
> > > >>> > >> > be
> > > >>> > >> > > >> > > >> compatible
> > > >>> > >> > > >> > > >> >> > with
> > > >>> > >> > > >> > > >> >> > > > all
> > > >>> > >> > > >> > > >> >> > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms
> that
> > > >>> Bigtop
> > > >>> > >> > > >> distribution
> > > >>> > >> > > >> > > is
> > > >>> > >> > > >> > > >> >> > > targeting
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > verifying
> > > >>> > whether
> > > >>> > >> > our
> > > >>> > >> > > >> > > software
> > > >>> > >> > > >> > > >> >> runs
> > > >>> > >> > > >> > > >> >> > > fine
> > > >>> > >> > > >> > > >> >> > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
> > > >>> ubuntu-14.04
> > > >>> > >> > > debian-8
> > > >>> > >> > > >> > > >> >> > opensuse-13.2.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests
> that
> > > >>> should
> > > >>> > be
> > > >>> > >> > > added
> > > >>> > >> > > >> for
> > > >>> > >> > > >> > > >> >> > deployment.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the
> test
> > > >>> > artifacts
> > > >>> > >> > which
> > > >>> > >> > > >> goes
> > > >>> > >> > > >> > > >> beyond
> > > >>> > >> > > >> > > >> >> > > smoke
> > > >>> > >> > > >> > > >> >> > > > > test
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are
> > basically
> > > >>> the
> > > >>> > >> > > integration
> > > >>> > >> > > >> > > tests
> > > >>> > >> > > >> > > >> for
> > > >>> > >> > > >> > > >> >> > > > > > verification
> > > >>> > >> > > >> > > >> >> > > > > > > > > after
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a
> > soft
> > > >>> > >> > requirement,
> > > >>> > >> > > >> but
> > > >>> > >> > > >> > > aim
> > > >>> > >> > > >> > > >> is
> > > >>> > >> > > >> > > >> >> to
> > > >>> > >> > > >> > > >> >> > > > > achieve
> > > >>> > >> > > >> > > >> >> > > > > > > this
> > > >>> > >> > > >> > > >> >> > > > > > > > > as
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have
> > > >>> explanation why
> > > >>> > >> not
> > > >>> > >> > to
> > > >>> > >> > > >> > > include.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any from
> > the
> > > >>> link
> > > >>> > >> which
> > > >>> > >> > > >> > > explicitly
> > > >>> > >> > > >> > > >> >> needs
> > > >>> > >> > > >> > > >> >> > to
> > > >>> > >> > > >> > > >> >> > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > > checked
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please
> let
> > > us
> > > >>> > know.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex
> as
> > > >>> > component
> > > >>> > >> to
> > > >>> > >> > > >> Bigtop*
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail
> > > >>> archive of
> > > >>> > >> > Bigtop,
> > > >>> > >> > > >> it
> > > >>> > >> > > >> > was
> > > >>> > >> > > >> > > >> >> learnt
> > > >>> > >> > > >> > > >> >> > > that
> > > >>> > >> > > >> > > >> >> > > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community want to see
> > the
> > > >>> > >> addition of
> > > >>> > >> > > new
> > > >>> > >> > > >> > > >> >> components
> > > >>> > >> > > >> > > >> >> > in
> > > >>> > >> > > >> > > >> >> > > > > > phases.
> > > >>> > >> > > >> > > >> >> > > > > > > > Here
> > > >>> > >> > > >> > > >> >> > > > > > > > > > are
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> > creating
> > > of
> > > >>> > >> package
> > > >>> > >> > > i.e.
> > > >>> > >> > > >> > rpm
> > > >>> > >> > > >> > > &
> > > >>> > >> > > >> > > >> deb
> > > >>> > >> > > >> > > >> >> > > files.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > > >>> documentations/READMEs,
> > > >>> > >> > LICENSE,
> > > >>> > >> > > >> > > >> DISCLAMER,
> > > >>> > >> > > >> > > >> >> > NOTES
> > > >>> > >> > > >> > > >> >> > > > etc
> > > >>> > >> > > >> > > >> >> > > > > if
> > > >>> > >> > > >> > > >> >> > > > > > > any
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any
> documentation
> > > that
> > > >>> > need
> > > >>> > >> to
> > > >>> > >> > be
> > > >>> > >> > > >> > added
> > > >>> > >> > > >> > > to
> > > >>> > >> > > >> > > >> >> > > > > distribution
> > > >>> > >> > > >> > > >> >> > > > > > of
> > > >>> > >> > > >> > > >> >> > > > > > > > our
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license
> > > >>> information of
> > > >>> > >> > > >> > dependencies
> > > >>> > >> > > >> > > >> >> required
> > > >>> > >> > > >> > > >> >> > > to
> > > >>> > >> > > >> > > >> >> > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > added
> > > >>> > >> > > >> > > >> >> > > > > > > > > to
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at
> very
> > > >>> least)
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke
> test
> > > for
> > > >>> > >> > packaging.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for
> > > >>> automatic
> > > >>> > >> > > >> deployment
> > > >>> > >> > > >> > and
> > > >>> > >> > > >> > > >> >> > > > configuration
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet
> recipes
> > > for
> > > >>> > >> > automatic
> > > >>> > >> > > >> > > >> deployment
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For verification
> > of
> > > >>> > >> > deployments.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat'
> to
> > > make
> > > >>> > sure
> > > >>> > >> all
> > > >>> > >> > > new
> > > >>> > >> > > >> > > stuff
> > > >>> > >> > > >> > > >> is
> > > >>> > >> > > >> > > >> >> > > > compliant
> > > >>> > >> > > >> > > >> >> > > > > > > with
> > > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license requirements.
> If
> > > >>> you add
> > > >>> > >> code
> > > >>> > >> > > >> > licenses
> > > >>> > >> > > >> > > >> >> under
> > > >>> > >> > > >> > > >> >> > > > > different
> > > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need to be
> > > >>> listed in
> > > >>> > >> the
> > > >>> > >> > > >> NOTICE.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your
> > thoughts
> > > >>> on
> > > >>> > the
> > > >>> > >> > > >> approach.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
> > corresponding
> > > >>> > >> > communication
> > > >>> > >> > > >> on
> > > >>> > >> > > >> > > >> bigtop
> > > >>> > >> > > >> > > >> >> > > mailing
> > > >>> > >> > > >> > > >> >> > > > > list
> > > >>> > >> > > >> > > >> >> > > > > > > as
> > > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some specific
> > > >>> > >> > > >> questions/suggestions
> > > >>> > >> > > >> > > >> >> related to
> > > >>> > >> > > >> > > >> >> > > > what
> > > >>> > >> > > >> > > >> >> > > > > > > should
> > > >>> > >> > > >> > > >> >> > > > > > > > > be
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the package
> > and
> > > >>> what
> > > >>> > >> > should
> > > >>> > >> > > be
> > > >>> > >> > > >> > the
> > > >>> > >> > > >> > > >> smoke
> > > >>> > >> > > >> > > >> >> > > tests,
> > > >>> > >> > > >> > > >> >> > > > > but
> > > >>> > >> > > >> > > >> >> > > > > > > in
> > > >>> > >> > > >> > > >> >> > > > > > > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not having
> > too
> > > >>> much
> > > >>> > >> > content
> > > >>> > >> > > >> > here,
> > > >>> > >> > > >> > > >> we'll
> > > >>> > >> > > >> > > >> >> > put
> > > >>> > >> > > >> > > >> >> > > > the
> > > >>> > >> > > >> > > >> >> > > > > > > > > questions
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this
> > > >>> > mailthread.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > > >
> > > >>> > >> > > >> > > >> >> > > > > >
> > > >>> > >> > > >> > > >> >> > > > >
> > > >>> > >> > > >> > > >> >> > > >
> > > >>> > >> > > >> > > >> >> > >
> > > >>> > >> > > >> > > >> >> >
> > > >>> > >> > > >> > > >> >>
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >> >
> > > >>> > >> > > >> > > >>
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > > >
> > > >>> > >> > > >> > >
> > > >>> > >> > > >> >
> > > >>> > >> > > >>
> > > >>> > >> > > >
> > > >>> > >> > > >
> > > >>> > >> > >
> > > >>> > >> >
> > > >>> > >>
> > > >>> > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>



-- 
*regards,*
*~pradeep*

Re: Apache Apex as component in Apache bigtop

[Tushar Gosavi <tu...@datatorrent.com>]

In following statement, can you replace dtGateway with ApplicationMaster
and Yarn. or may be remove this statement entirely.
apex is a wrapper around the REST API provided by dtGatway,

- Tushar.




On Thu, Feb 18, 2016 at 12:41 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Hi,
>
> I've attached man page for apex in Jira. Here is the link to it:
> https://issues.apache.org/jira/secure/attachment/12788403/apex.1
>
> Please view it by: man -l apex.1
>
> Thanks,
> Chinmay.
>
>
> On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > Hi Chinmay,
> >
> > I am not able to find the attachment, may be attachments are not allowed
> in
> > the mailing list. Can you attach the man page to the Jira?
> >
> > - Tushar.
> >
> >
> > On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > I've attached man page file for the purpose of bigtop integration.
> > > Please have a look and let me know if there any any feedback about it.
> > >
> > > You can see the man page content of this file with:
> > > man -l apex.1
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > >
> > > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <
> chinmay@apache.org>
> > > wrote:
> > >
> > >> Thanks. Will follow bigtop recommendation. i.e. remove "-incubating".
> > >>
> > >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <thomas@datatorrent.com
> >
> > >> wrote:
> > >>
> > >>> Please follow the recommendation and continue discussion on the
> Bigtop
> > >>> list.
> > >>>
> > >>> --
> > >>> sent from mobile
> > >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <ch...@apache.org>
> > wrote:
> > >>>
> > >>> > I got a recommendation from bigtop community to drop "-incubating"
> > from
> > >>> > version.
> > >>> >
> > >>> > Here is the mail thread:
> > >>> >
> > >>> >
> > >>>
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> > >>> >
> > >>> > Please let me know if removing "-incubating" is fine.
> > >>> >
> > >>> > Thanks,
> > >>> > Chinmay.
> > >>> >
> > >>> >
> > >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> > chinmay@apache.org
> > >>> >
> > >>> > wrote:
> > >>> >
> > >>> > > I've already started communication there.
> > >>> > >
> > >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> > >>> thomas@datatorrent.com>
> > >>> > > wrote:
> > >>> > >
> > >>> > >> This is a question you should ask on the Bigtop mailing list.
> > >>> > >>
> > >>> > >> --
> > >>> > >> sent from mobile
> > >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org>
> > >>> wrote:
> > >>> > >>
> > >>> > >> > Yes. In case of RPM, the only restriction on version is that
> it
> > >>> can
> > >>> > not
> > >>> > >> > contain '-'.
> > >>> > >> > However, as a workaround, one can use 'incubating' as a part
> of
> > >>> > >> %{release}.
> > >>> > >> >
> > >>> > >> > Version: 3.3.0
> > >>> > >> > Release: incubating
> > >>> > >> >
> > >>> > >> > apex-3.3.0-incubating.noarch.rpm
> > >>> > >> >
> > >>> > >> > It is the same practice followed for pre-release packages e.g.
> > >>> with
> > >>> > rcX,
> > >>> > >> > alphaX, betaX being used in %{release}.
> > >>> > >> >
> > >>> > >> > Regards,
> > >>> > >> > Pradeep A. Dalvi
> > >>> > >> >
> > >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> > >>> > chinmay@apache.org>
> > >>> > >> > wrote:
> > >>> > >> >
> > >>> > >> > > Hi Everyone,
> > >>> > >> > >
> > >>> > >> > > In one of the previous mails in this thread, we decided that
> > >>> version
> > >>> > >> of
> > >>> > >> > the
> > >>> > >> > > package will be "3.3.0-incubating".
> > >>> > >> > > The problem that I'm facing is in RPM base version "-" is
> not
> > >>> > allowed.
> > >>> > >> > >
> > >>> > >> > > Can we have package version to "3.3.0"?
> > >>> > >> > > i.e. the packages created will be as follows:
> > >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> > >>> > >> > >
> > >>> > >> > > There is another integration for Zepplin in Bigtop which is
> > >>> > incubating
> > >>> > >> > and
> > >>> > >> > > is having base version as "0.5.6".
> > >>> > >> > >
> > >>> > >> > > Please let me know your thoughts.
> > >>> > >> > >
> > >>> > >> > > Thanks,
> > >>> > >> > > Chinmay.
> > >>> > >> > >
> > >>> > >> > >
> > >>> > >> > >
> > >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> > >>> > >> chinmay@apache.org>
> > >>> > >> > > wrote:
> > >>> > >> > >
> > >>> > >> > > > Thanks everyone for the feedback.
> > >>> > >> > > >
> > >>> > >> > > > Also as Ram and David mentioned, I'll start creating the
> man
> > >>> page
> > >>> > as
> > >>> > >> > > well.
> > >>> > >> > > >
> > >>> > >> > > > Thanks,
> > >>> > >> > > > Chinmay.
> > >>> > >> > > >
> > >>> > >> > > >
> > >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> > >>> > >> thomas@datatorrent.com>
> > >>> > >> > > > wrote:
> > >>> > >> > > >
> > >>> > >> > > >> I think we already agreed that we will retain the current
> > >>> script
> > >>> > >> name
> > >>> > >> > in
> > >>> > >> > > >> this bigtop iteration. There is a ticket for renaming,
> add
> > >>> you
> > >>> > >> > > suggestions
> > >>> > >> > > >> there.
> > >>> > >> > > >>
> > >>> > >> > > >> --
> > >>> > >> > > >> sent from mobile
> > >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > >>> > sandesh@datatorrent.com>
> > >>> > >> > > wrote:
> > >>> > >> > > >>
> > >>> > >> > > >> > +1 for Symlink from apex to dtcli or start the
> transition
> > >>> > >> process by
> > >>> > >> > > >> making
> > >>> > >> > > >> > dtcli as the symlink to apex cli.
> > >>> > >> > > >> >
> > >>> > >> > > >> > Renaming is a breaking change should be done during
> major
> > >>> > version
> > >>> > >> > > number
> > >>> > >> > > >> > change.
> > >>> > >> > > >> >
> > >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> > >>> > david@datatorrent.com
> > >>> > >> >
> > >>> > >> > > >> wrote:
> > >>> > >> > > >> >
> > >>> > >> > > >> > > Apologies for hitting the "Send" button too early.
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > What is the scope of the name change?  If we simply
> > have
> > >>> a
> > >>> > >> symlink
> > >>> > >> > > >> from
> > >>> > >> > > >> > > apex to dtcli, then it should have no backward
> > >>> compatibility
> > >>> > >> > problem
> > >>> > >> > > >> at
> > >>> > >> > > >> > > all.
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > But the proper name change would involve:
> > >>> > >> > > >> > > - Change the name of DTCli class to ApexCli
> > >>> > >> > > >> > > - Change the prompt from dt> to apex>
> > >>> > >> > > >> > > - Change the initial greeting when running the CLI
> from
> > >>> > >> something
> > >>> > >> > > like
> > >>> > >> > > >> > "DT
> > >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > The above changes may break existing scripts that
> rely
> > >>> on the
> > >>> > >> the
> > >>> > >> > > >> string
> > >>> > >> > > >> > > "DT".
> > >>> > >> > > >> > > But if folks are okay with this potential problem, I
> am
> > >>> okay
> > >>> > >> with
> > >>> > >> > > the
> > >>> > >> > > >> > name
> > >>> > >> > > >> > > change.
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > For the man page, we can start with the content of
> this
> > >>> page:
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > Plus the command line options when doing dtcli -h:
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > usage: DTCli
> > >>> > >> > > >> > >  -e <arg>    Commands are read from the argument
> > >>> > >> > > >> > >  -f <arg>    Use the specified prompt at all time
> > >>> > >> > > >> > >  -h          Print this help
> > >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos principal
> > >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> > >>> > >> > > >> > >  -p <arg>    JSONP padding function
> > >>> > >> > > >> > >  -r          JSON Raw mode
> > >>> > >> > > >> > >  -v          Verbose mode level 1
> > >>> > >> > > >> > >  -vv         Verbose mode level 2
> > >>> > >> > > >> > >  -vvv        Verbose mode level 3
> > >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > David
> > >>> > >> > > >> > >
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> > >>> > >> david@datatorrent.com
> > >>> > >> > >
> > >>> > >> > > >> > wrote:
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > > What is the scope of the name change?  If we simply
> > >>> have a
> > >>> > >> > symlink
> > >>> > >> > > >> from
> > >>> > >> > > >> > > > apex to dtcli, then it should have no backward
> > >>> > compatibility
> > >>> > >> > > >> problem at
> > >>> > >> > > >> > > all.
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > > > But the proper name change would involve:
> > >>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
> > >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> > >>> > >> > > >> > > > - Change the initial greeting when running the CLI
> > from
> > >>> > >> > something
> > >>> > >> > > >> like
> > >>> > >> > > >> > > "DT
> > >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > > > The above changes may break existing scripts that
> > rely
> > >>> on
> > >>> > the
> > >>> > >> > the
> > >>> > >> > > >> > string
> > >>> > >> > > >> > > > "DT".
> > >>> > >> > > >> > > > But if folks are okay with this potential problem,
> I
> > am
> > >>> > okay
> > >>> > >> > with
> > >>> > >> > > >> the
> > >>> > >> > > >> > > name
> > >>> > >> > > >> > > > change.
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > > > For the man page, we can start with the content of
> > this
> > >>> > page:
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar
> <
> > >>> > >> > > >> chinmay@apache.org
> > >>> > >> > > >> > >
> > >>> > >> > > >> > > > wrote:
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > > >> Here is a suggestion I have related to name
> change:
> > >>> > >> > > >> > > >>
> > >>> > >> > > >> > > >> 1. While packaging we change the name of "dtcli"
> > >>> script to
> > >>> > >> > > "apex".
> > >>> > >> > > >> > This
> > >>> > >> > > >> > > is
> > >>> > >> > > >> > > >> done so that in future when we change the name to
> > >>> "apex",
> > >>> > >> users
> > >>> > >> > > of
> > >>> > >> > > >> > > bigtop
> > >>> > >> > > >> > > >> apex don't have to transition much.
> > >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli which points
> > to
> > >>> apex
> > >>> > >> > > script.
> > >>> > >> > > >> > This
> > >>> > >> > > >> > > is
> > >>> > >> > > >> > > >> for backward compatibility.
> > >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first
> > integration
> > >>> in
> > >>> > >> > bigtop.
> > >>> > >> > > >> This
> > >>> > >> > > >> > is
> > >>> > >> > > >> > > >> in
> > >>> > >> > > >> > > >> the interest of not changing the content of dtcli
> or
> > >>> apex
> > >>> > >> > script.
> > >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these naming
> > >>> related
> > >>> > >> > > changes
> > >>> > >> > > >> and
> > >>> > >> > > >> > > >> update bigtop repository later to remove
> references
> > to
> > >>> > dtcli
> > >>> > >> > all
> > >>> > >> > > >> > > together.
> > >>> > >> > > >> > > >>
> > >>> > >> > > >> > > >> Please share your thoughts on above approach.
> > >>> > >> > > >> > > >>
> > >>> > >> > > >> > > >> Also, please share what could be the man page
> > content
> > >>> for
> > >>> > >> > "apex".
> > >>> > >> > > >> > > >>
> > >>> > >> > > >> > > >> Thanks,
> > >>> > >> > > >> > > >> Chinmay.
> > >>> > >> > > >> > > >>
> > >>> > >> > > >> > > >>
> > >>> > >> > > >> > > >>
> > >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay
> Kolhatkar <
> > >>> > >> > > >> > chinmay@apache.org>
> > >>> > >> > > >> > > >> wrote:
> > >>> > >> > > >> > > >>
> > >>> > >> > > >> > > >> > Yes.. It'll be build from source tar downloaded
> > >>> from one
> > >>> > >> of
> > >>> > >> > the
> > >>> > >> > > >> > apache
> > >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> > >>> > >> > > >> > > >> >
> > >>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced from
> > dtcli.
> > >>> > This
> > >>> > >> > > leaves
> > >>> > >> > > >> us
> > >>> > >> > > >> > 3
> > >>> > >> > > >> > > >> > options:
> > >>> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli would
> > >>> need a
> > >>> > >> > change
> > >>> > >> > > >> while
> > >>> > >> > > >> > > >> > packaging.
> > >>> > >> > > >> > > >> >
> > >>> > >> > > >> > > >> > But if we want to use dtcli as it is, then we
> > would
> > >>> need
> > >>> > >> > > >> dt-env.sh
> > >>> > >> > > >> > > >> >
> > >>> > >> > > >> > > >> >
> > >>> > >> > > >> > > >> >
> > >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> > >>> > >> > > >> > thomas@datatorrent.com
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > > >> > wrote:
> > >>> > >> > > >> > > >> >
> > >>> > >> > > >> > > >> >> Looks good overall, though there shouldn't be
> any
> > >>> > >> dt-env.sh
> > >>> > >> > > >> > > >> >>
> > >>> > >> > > >> > > >> >> What will the bigtop package be built from, the
> > >>> source
> > >>> > >> tar?
> > >>> > >> > > >> > > >> >>
> > >>> > >> > > >> > > >> >>
> > >>> > >> > > >> > > >> >>
> > >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay
> > Kolhatkar
> > >>> <
> > >>> > >> > > >> > > chinmay@apache.org
> > >>> > >> > > >> > > >> >
> > >>> > >> > > >> > > >> >> wrote:
> > >>> > >> > > >> > > >> >>
> > >>> > >> > > >> > > >> >> > Hi Thomas,
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > Thanks for the feedback.
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > First, we're not changing the name anywhere.
> > >>> We'll
> > >>> > >> follow
> > >>> > >> > > what
> > >>> > >> > > >> > > >> >> currently is
> > >>> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating
> version.
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory
> structure
> > >>> below
> > >>> > >> which
> > >>> > >> > > is
> > >>> > >> > > >> > > inline
> > >>> > >> > > >> > > >> >> with
> > >>> > >> > > >> > > >> >> > other existing integrations with bigtop.
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > The need for each file is as follows:
> > >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> > >>> > >> > > >> engine/src/main/scripts/
> > >>> > >> > > >> > > of
> > >>> > >> > > >> > > >> >> source
> > >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from
> dtcli.
> > >>> This
> > >>> > >> > > searches
> > >>> > >> > > >> for
> > >>> > >> > > >> > > >> hadoop
> > >>> > >> > > >> > > >> >> > binary path and exports an env variable for
> > >>> dtcli to
> > >>> > >> use
> > >>> > >> > it.
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is
> > >>> extracted
> > >>> > >> in a
> > >>> > >> > > >> similar
> > >>> > >> > > >> > > >> way to
> > >>> > >> > > >> > > >> >> > how DT Community Edition finds dependency
> jars.
> > >>> > >> > > >> > > >> >> > Please note that this is the first iteration
> > >>> list of
> > >>> > >> jars.
> > >>> > >> > > I'm
> > >>> > >> > > >> > > >> trying to
> > >>> > >> > > >> > > >> >> > narrow this down to only those which are
> really
> > >>> > >> required.
> > >>> > >> > > >> > > >> >> > The test that I'm running to check if dtcli
> > runs
> > >>> fine
> > >>> > >> with
> > >>> > >> > > >> given
> > >>> > >> > > >> > > >> >> dependency
> > >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi
> demo.
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > Above is the directory structure which is
> > >>> required
> > >>> > for
> > >>> > >> CLI
> > >>> > >> > > to
> > >>> > >> > > >> > work.
> > >>> > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned are
> > some
> > >>> > >> > additional
> > >>> > >> > > >> > files
> > >>> > >> > > >> > > >> which
> > >>> > >> > > >> > > >> >> > makes the like of administrator easier. For
> eg.
> > >>> > >> /etc/skel
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > Please let me know if this is inline with
> what
> > >>> you're
> > >>> > >> > > >> thinking.
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > Thanks,
> > >>> > >> > > >> > > >> >> > Chinmay.
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > .
> > >>> > >> > > >> > > >> >> > |-- bin
> > >>> > >> > > >> > > >> >> > |   `-- dtcli
> > >>> > >> > > >> > > >> >> > |-- conf
> > >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > >>> > >> > > >> > > >> >> > `-- lib
> > >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > >>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > >>> > >> > > >> > > >> >> >     |--
> apex-bufferserver-3.3.0-incubating.jar
> > >>> > >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > >>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > >>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > >>> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > >>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > >>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > >>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > >>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > >>> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > >>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > >>> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > >>> > >> > > >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > >>> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > 3 directories, 40 files
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas
> Weise <
> > >>> > >> > > >> > > >> thomas@datatorrent.com>
> > >>> > >> > > >> > > >> >> > wrote:
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >> > > Chinmay,
> > >>> > >> > > >> > > >> >> > >
> > >>> > >> > > >> > > >> >> > > Before discussing where to put the files,
> > let's
> > >>> > make
> > >>> > >> > sure
> > >>> > >> > > >> they
> > >>> > >> > > >> > > are
> > >>> > >> > > >> > > >> >> really
> > >>> > >> > > >> > > >> >> > > needed for the operation of the CLI. As for
> > >>> names,
> > >>> > >> > > anything
> > >>> > >> > > >> new
> > >>> > >> > > >> > > >> needs
> > >>> > >> > > >> > > >> >> to
> > >>> > >> > > >> > > >> >> > > reflect Apex in the name and should follow
> > >>> common
> > >>> > >> > > >> conventions,
> > >>> > >> > > >> > > >> >> especially
> > >>> > >> > > >> > > >> >> > > in Bigtop where there are many existing
> > >>> > integrations
> > >>> > >> to
> > >>> > >> > > look
> > >>> > >> > > >> > at.
> > >>> > >> > > >> > > >> >> > >
> > >>> > >> > > >> > > >> >> > > Thanks,
> > >>> > >> > > >> > > >> >> > > Thomas
> > >>> > >> > > >> > > >> >> > >
> > >>> > >> > > >> > > >> >> > >
> > >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay
> > >>> Kolhatkar
> > >>> > <
> > >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > >>> > >> > > >> > > >> >> > > wrote:
> > >>> > >> > > >> > > >> >> > >
> > >>> > >> > > >> > > >> >> > > > Really good points Pradeep and Aniruddha.
> > >>> > >> > > >> > > >> >> > > >
> > >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to change the
> > >>> dtcli
> > >>> > >> > > >> considering it
> > >>> > >> > > >> > > >> works
> > >>> > >> > > >> > > >> >> > with
> > >>> > >> > > >> > > >> >> > > DT
> > >>> > >> > > >> > > >> >> > > > Community edition. We can keep the
> > directory
> > >>> > >> structure
> > >>> > >> > > >> > similar
> > >>> > >> > > >> > > to
> > >>> > >> > > >> > > >> >> that.
> > >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which contains
> > >>> > information
> > >>> > >> > > >> required
> > >>> > >> > > >> > for
> > >>> > >> > > >> > > >> >> dtcli
> > >>> > >> > > >> > > >> >> > to
> > >>> > >> > > >> > > >> >> > > > launch.
> > >>> > >> > > >> > > >> >> > > >
> > >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop community
> that
> > >>> > whether
> > >>> > >> > they
> > >>> > >> > > >> > > >> facilitate
> > >>> > >> > > >> > > >> >> the
> > >>> > >> > > >> > > >> >> > > > installation of rpms/debs before user is
> > >>> created.
> > >>> > >> In
> > >>> > >> > > >> either
> > >>> > >> > > >> > > case,
> > >>> > >> > > >> > > >> >> > current
> > >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home
> > directory.
> > >>> > Also
> > >>> > >> > > before
> > >>> > >> > > >> > > putting
> > >>> > >> > > >> > > >> >> > > anything
> > >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define what are
> the
> > >>> > default
> > >>> > >> > > >> contents
> > >>> > >> > > >> > > that
> > >>> > >> > > >> > > >> >> > should
> > >>> > >> > > >> > > >> >> > > go
> > >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no
> defaults,
> > >>> > >> probably we
> > >>> > >> > > >> should
> > >>> > >> > > >> > > not
> > >>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > >>> > >> > > >> > > >> >> > > >
> > >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice.
> > This
> > >>> way
> > >>> > >> > > contents
> > >>> > >> > > >> of
> > >>> > >> > > >> > > >> >> dt-env.sh
> > >>> > >> > > >> > > >> >> > > are
> > >>> > >> > > >> > > >> >> > > > present as env variables. But I see a
> catch
> > >>> > there.
> > >>> > >> > > Adding
> > >>> > >> > > >> > > >> dt-env.sh
> > >>> > >> > > >> > > >> >> to
> > >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the
> variables
> > >>> > >> available
> > >>> > >> > at
> > >>> > >> > > >> > > runtime
> > >>> > >> > > >> > > >> all
> > >>> > >> > > >> > > >> >> > the
> > >>> > >> > > >> > > >> >> > > > time. I feel a little skeptical about
> that.
> > >>> > Maybe a
> > >>> > >> > > >> possible
> > >>> > >> > > >> > > >> >> collision
> > >>> > >> > > >> > > >> >> > > can
> > >>> > >> > > >> > > >> >> > > > occur with other application vars.
> > >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
> > >>> > >> > "../conf/dt-env.sh"
> > >>> > >> > > >> and
> > >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are
> > >>> anyway
> > >>> > >> > > >> available.
> > >>> > >> > > >> > > >> >> > > >
> > >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is
> > >>> happening
> > >>> > at
> > >>> > >> > > Bigtop
> > >>> > >> > > >> > > >> >> > > Jira/repository.
> > >>> > >> > > >> > > >> >> > > > Packaging related code usually goes
> there.
> > >>> > (That's
> > >>> > >> > what
> > >>> > >> > > >> all
> > >>> > >> > > >> > the
> > >>> > >> > > >> > > >> >> > > components
> > >>> > >> > > >> > > >> >> > > > in bigtop does).
> > >>> > >> > > >> > > >> >> > > > Having said that, once a PR is created,
> > I'll
> > >>> be
> > >>> > >> > sharing
> > >>> > >> > > >> the
> > >>> > >> > > >> > > link
> > >>> > >> > > >> > > >> of
> > >>> > >> > > >> > > >> >> PR
> > >>> > >> > > >> > > >> >> > > > here, so that apex community as well can
> > >>> review
> > >>> > it.
> > >>> > >> > > >> > > >> >> > > >
> > >>> > >> > > >> > > >> >> > > > Thanks,
> > >>> > >> > > >> > > >> >> > > > Chinmay.
> > >>> > >> > > >> > > >> >> > > >
> > >>> > >> > > >> > > >> >> > > >
> > >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM,
> Aniruddha
> > >>> > >> Thombare <
> > >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > >>> > >> > > >> > > >> >> > > >
> > >>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> > >>> > >> > > >> > > >> >> > > > > We may need to iron out details about
> > exact
> > >>> > paths
> > >>> > >> > etc.
> > >>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR
> comments.
> > >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> > >>> > >> > > >> > > >> >> > > > >
> > >>> > >> > > >> > > >> >> > > > >
> > >>> > >> > > >> > > >> >> > > > >
> > >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A.
> > >>> Dalvi <
> > >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > >>> > >> > > >> > > >> >> > > > > wrote:
> > >>> > >> > > >> > > >> >> > > > >
> > >>> > >> > > >> > > >> >> > > > > > Inline comments...
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM,
> > Chinmay
> > >>> > >> > Kolhatkar <
> > >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > >>> > >> > > >> > > >> >> > > > > > wrote:
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present in
> > DT
> > >>> > >> community
> > >>> > >> > > >> > edition
> > >>> > >> > > >> > > >> will
> > >>> > >> > > >> > > >> >> be
> > >>> > >> > > >> > > >> >> > > > > included
> > >>> > >> > > >> > > >> >> > > > > > > there.
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I found
> from
> > >>> > common
> > >>> > >> > > >> > dependencies
> > >>> > >> > > >> > > >> >> between
> > >>> > >> > > >> > > >> >> > > of
> > >>> > >> > > >> > > >> >> > > > > apex
> > >>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> > >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > >>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > >>> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > >>> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > >>> > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > >>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > >>> > >> > > >> > > >> >> > > > > > >
> jetty-websocket-8.1.10.v20130312.jar
> > >>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > >>> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > >>> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some
> tests
> > do
> > >>> > check
> > >>> > >> > that
> > >>> > >> > > >> > dtcli
> > >>> > >> > > >> > > >> works
> > >>> > >> > > >> > > >> >> > > > properly
> > >>> > >> > > >> > > >> >> > > > > > for
> > >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with
> > only
> > >>> > these
> > >>> > >> > > >> libraries
> > >>> > >> > > >> > > >> >> present in
> > >>> > >> > > >> > > >> >> > > > > > isolation
> > >>> > >> > > >> > > >> >> > > > > > > without dependency on local m2.
> > >>> > >> > > >> > > >> >> > > > > > > I'm believe that there are unwanted
> > >>> jars
> > >>> > >> which
> > >>> > >> > are
> > >>> > >> > > >> used
> > >>> > >> > > >> > > for
> > >>> > >> > > >> > > >> >> > compile
> > >>> > >> > > >> > > >> >> > > > > time
> > >>> > >> > > >> > > >> >> > > > > > > dependency and not runtime in above
> > >>> list
> > >>> > >> which I
> > >>> > >> > > can
> > >>> > >> > > >> > drop
> > >>> > >> > > >> > > >> to
> > >>> > >> > > >> > > >> >> keep
> > >>> > >> > > >> > > >> >> > > > > package
> > >>> > >> > > >> > > >> >> > > > > > > size minimal.
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all
> > of
> > >>> the
> > >>> > >> > > >> dt-env.sh is
> > >>> > >> > > >> > > >> >> required.
> > >>> > >> > > >> > > >> >> > > > Only
> > >>> > >> > > >> > > >> >> > > > > > > required values I see in it are
> > >>> classpath.
> > >>> > >> and
> > >>> > >> > > >> dtcli is
> > >>> > >> > > >> > > >> >> capable
> > >>> > >> > > >> > > >> >> > of
> > >>> > >> > > >> > > >> >> > > > > > building
> > >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have
> to
> > >>> see
> > >>> > how
> > >>> > >> > dtcli
> > >>> > >> > > >> can
> > >>> > >> > > >> > > find
> > >>> > >> > > >> > > >> >> the
> > >>> > >> > > >> > > >> >> > jar
> > >>> > >> > > >> > > >> >> > > > > path,
> > >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf files.
> If
> > >>> we
> > >>> > >> need a
> > >>> > >> > > >> change
> > >>> > >> > > >> > in
> > >>> > >> > > >> > > >> >> dtcli,
> > >>> > >> > > >> > > >> >> > > then
> > >>> > >> > > >> > > >> >> > > > > > > community's opinion is required for
> > >>> whether
> > >>> > >> > dtcli
> > >>> > >> > > >> > should
> > >>> > >> > > >> > > >> >> change
> > >>> > >> > > >> > > >> >> > in
> > >>> > >> > > >> > > >> >> > > > our
> > >>> > >> > > >> > > >> >> > > > > > repo
> > >>> > >> > > >> > > >> >> > > > > > > or a copy of that with required
> > changes
> > >>> > >> exist in
> > >>> > >> > > >> bigtop
> > >>> > >> > > >> > > >> repo
> > >>> > >> > > >> > > >> >> > until
> > >>> > >> > > >> > > >> >> > > we
> > >>> > >> > > >> > > >> >> > > > > > make
> > >>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > > Can we have these file paths set
> using
> > >>> > >> environment
> > >>> > >> > > >> > > variables
> > >>> > >> > > >> > > >> >> with
> > >>> > >> > > >> > > >> >> > > some
> > >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> > >>> > >> > > >> > > >> >> > > > > > Then we can set such params in
> > dt-env.sh.
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file
> > >>> > different
> > >>> > >> > from
> > >>> > >> > > >> > > >> >> dt-site.xml?
> > >>> > >> > > >> > > >> >> > Not
> > >>> > >> > > >> > > >> >> > > > > sure
> > >>> > >> > > >> > > >> >> > > > > > if
> > >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help.
> > Files &
> > >>> > Dirs
> > >>> > >> > from
> > >>> > >> > > >> > > /etc/skel
> > >>> > >> > > >> > > >> >> are
> > >>> > >> > > >> > > >> >> > > > copied
> > >>> > >> > > >> > > >> >> > > > > to
> > >>> > >> > > >> > > >> >> > > > > > > home of new user when useradd
> program
> > >>> is
> > >>> > >> called.
> > >>> > >> > > But
> > >>> > >> > > >> > for
> > >>> > >> > > >> > > >> >> existing
> > >>> > >> > > >> > > >> >> > > > users
> > >>> > >> > > >> > > >> >> > > > > > > this won't be of any use. Moreover,
> > >>> dtcli
> > >>> > >> > creates
> > >>> > >> > > >> > ~/.dt/
> > >>> > >> > > >> > > on
> > >>> > >> > > >> > > >> >> the
> > >>> > >> > > >> > > >> >> > fly
> > >>> > >> > > >> > > >> >> > > > if
> > >>> > >> > > >> > > >> >> > > > > > not
> > >>> > >> > > >> > > >> >> > > > > > > encountered for the first time.
> Again
> > >>> > >> correct me
> > >>> > >> > > if
> > >>> > >> > > >> I'm
> > >>> > >> > > >> > > >> wrong.
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > > Yes, you are right. However this
> would
> > >>> > >> probably be
> > >>> > >> > > >> > > necessary
> > >>> > >> > > >> > > >> >> step
> > >>> > >> > > >> > > >> >> > for
> > >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get
> installed
> > >>> > during
> > >>> > >> OS
> > >>> > >> > > >> install
> > >>> > >> > > >> > > and
> > >>> > >> > > >> > > >> >> > before
> > >>> > >> > > >> > > >> >> > > > user
> > >>> > >> > > >> > > >> >> > > > > > accounts were created.
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile
> ->
> > >>> I'm
> > >>> > not
> > >>> > >> > sure
> > >>> > >> > > >> its
> > >>> > >> > > >> > the
> > >>> > >> > > >> > > >> best
> > >>> > >> > > >> > > >> >> > > idea.
> > >>> > >> > > >> > > >> >> > > > > > We'll
> > >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location which
> is
> > >>> by
> > >>> > >> default
> > >>> > >> > > >> > present
> > >>> > >> > > >> > > in
> > >>> > >> > > >> > > >> >> path
> > >>> > >> > > >> > > >> >> > > i.e.
> > >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > > For other env variables specific to
> > apex,
> > >>> > I'll
> > >>> > >> > > prefer
> > >>> > >> > > >> to
> > >>> > >> > > >> > > use
> > >>> > >> > > >> > > >> >> > > dt-env.sh
> > >>> > >> > > >> > > >> >> > > > > and
> > >>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather than
> > changing
> > >>> > >> bashrc
> > >>> > >> > > >> etc...
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
> > >>> /etc/profile.d/
> > >>> > >> shall
> > >>> > >> > > be
> > >>> > >> > > >> the
> > >>> > >> > > >> > > >> >> cleaner
> > >>> > >> > > >> > > >> >> > way
> > >>> > >> > > >> > > >> >> > > > to
> > >>> > >> > > >> > > >> >> > > > > > achieve the same.
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > Thanks,
> > >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM,
> > >>> Aniruddha
> > >>> > >> > > Thombare
> > >>> > >> > > >> <
> > >>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > Hi,
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > We need to consider following:
> > >>> > >> > > >> > > >> >> > > > > > > > System wide default config files
> > can
> > >>> be
> > >>> > >> > located
> > >>> > >> > > at
> > >>> > >> > > >> > > >> following
> > >>> > >> > > >> > > >> >> > > > > locations:
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have
> > to
> > >>> > change
> > >>> > >> > > dtcli
> > >>> > >> > > >> > > >> behaviour
> > >>> > >> > > >> > > >> >> on
> > >>> > >> > > >> > > >> >> > > how
> > >>> > >> > > >> > > >> >> > > > it
> > >>> > >> > > >> > > >> >> > > > > > > finds
> > >>> > >> > > >> > > >> >> > > > > > > > those locations)
> > >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may
> have
> > to
> > >>> > >> change
> > >>> > >> > > dtcli
> > >>> > >> > > >> > > >> >> behaviour on
> > >>> > >> > > >> > > >> >> > > how
> > >>> > >> > > >> > > >> >> > > > > it
> > >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
> > >>> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and
> > other
> > >>> > files
> > >>> > >> > (for
> > >>> > >> > > >> new
> > >>> > >> > > >> > > users
> > >>> > >> > > >> > > >> >> that
> > >>> > >> > > >> > > >> >> > > will
> > >>> > >> > > >> > > >> >> > > > > be
> > >>> > >> > > >> > > >> >> > > > > > > > created in system in future.)
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > We may also have to modify
> bashrc /
> > >>> > >> > bashprofile
> > >>> > >> > > >> for
> > >>> > >> > > >> > > >> >> population
> > >>> > >> > > >> > > >> >> > > any
> > >>> > >> > > >> > > >> >> > > > > > > > variables that are required.
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your
> comments /
> > >>> > >> > suggestion.
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM,
> > >>> Thomas
> > >>> > >> Weise
> > >>> > >> > <
> > >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > >>> > >> > > >> > > >> >> > > > > > > > wrote:
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > -->
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM,
> > >>> Chinmay
> > >>> > >> > > >> Kolhatkar <
> > >>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging
> > effort
> > >>> > >> (rpm+deb)
> > >>> > >> > > of
> > >>> > >> > > >> > apex,
> > >>> > >> > > >> > > >> here
> > >>> > >> > > >> > > >> >> > are
> > >>> > >> > > >> > > >> >> > > > some
> > >>> > >> > > >> > > >> >> > > > > > > > > proposals
> > >>> > >> > > >> > > >> >> > > > > > > > > > about package structure etc..
> > >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on bbigtop
> > >>> mailing
> > >>> > >> > list, I
> > >>> > >> > > >> have
> > >>> > >> > > >> > > >> some
> > >>> > >> > > >> > > >> >> > > question
> > >>> > >> > > >> > > >> >> > > > > for
> > >>> > >> > > >> > > >> >> > > > > > > > apex
> > >>> > >> > > >> > > >> >> > > > > > > > > > community.
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory structure
> of
> > >>> apex
> > >>> > >> > package
> > >>> > >> > > >> for
> > >>> > >> > > >> > > both
> > >>> > >> > > >> > > >> >> deb &
> > >>> > >> > > >> > > >> >> > > > rpm:
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other
> > >>> dependent
> > >>> > >> jars>
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> > >>> > /usr/lib/apex/bin/dtcli
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> /usr/share/doc/man/man1/dtcli.1.gz
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > /usr/share/doc/apex/license/LICENSE.txt
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > >>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> > >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> = All
> > the
> > >>> 3rd
> > >>> > >> party
> > >>> > >> > > >> jars
> > >>> > >> > > >> > > which
> > >>> > >> > > >> > > >> >> are
> > >>> > >> > > >> > > >> >> > > > > required
> > >>> > >> > > >> > > >> >> > > > > > > for
> > >>> > >> > > >> > > >> >> > > > > > > > > apex
> > >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the
> > dependencies
> > >>> are
> > >>> > >> > > packaged
> > >>> > >> > > >> as
> > >>> > >> > > >> > > >> part of
> > >>> > >> > > >> > > >> >> > > > rpm/deb
> > >>> > >> > > >> > > >> >> > > > > by
> > >>> > >> > > >> > > >> >> > > > > > > any
> > >>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what those jars
> > >>> are in
> > >>> > >> > Bigtop
> > >>> > >> > > >> > > context?
> > >>> > >> > > >> > > >> >> Same
> > >>> > >> > > >> > > >> >> > as
> > >>> > >> > > >> > > >> >> > > > > > shipped
> > >>> > >> > > >> > > >> >> > > > > > > > with
> > >>> > >> > > >> > > >> >> > > > > > > > > DT community addition under
> lib/
> > ?
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses
> of
> > >>> > >> > > corresponding
> > >>> > >> > > >> of
> > >>> > >> > > >> > > 3rd
> > >>> > >> > > >> > > >> >> party
> > >>> > >> > > >> > > >> >> > > jars
> > >>> > >> > > >> > > >> >> > > > > > which
> > >>> > >> > > >> > > >> >> > > > > > > > > needs
> > >>> > >> > > >> > > >> >> > > > > > > > > > to included while packaging.
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > Questions related to this:
> > >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli of
> > >>> apex as
> > >>> > >> > "apex"
> > >>> > >> > > >> > instead
> > >>> > >> > > >> > > >> of
> > >>> > >> > > >> > > >> >> > > "dtcli"
> > >>> > >> > > >> > > >> >> > > > in
> > >>> > >> > > >> > > >> >> > > > > > > > bigtop
> > >>> > >> > > >> > > >> >> > > > > > > > > > package?
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > I think we should keep the name
> > >>> until
> > >>> > we
> > >>> > >> are
> > >>> > >> > > >> able
> > >>> > >> > > >> > to
> > >>> > >> > > >> > > >> >> change
> > >>> > >> > > >> > > >> >> > it
> > >>> > >> > > >> > > >> >> > > in
> > >>> > >> > > >> > > >> >> > > > > > Apex.
> > >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get
> confusing.
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares
> in
> > >>> bigtop
> > >>> > >> have
> > >>> > >> > > man
> > >>> > >> > > >> > page
> > >>> > >> > > >> > > >> for
> > >>> > >> > > >> > > >> >> > their
> > >>> > >> > > >> > > >> >> > > > > > > > > executables.
> > >>> > >> > > >> > > >> >> > > > > > > > > > I think we should have it too
> > for
> > >>> > >> dtcli.
> > >>> > >> > Is
> > >>> > >> > > >> there
> > >>> > >> > > >> > > any
> > >>> > >> > > >> > > >> >> > > > > documentation
> > >>> > >> > > >> > > >> >> > > > > > > > > which I
> > >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man page? or
> can
> > >>> I use
> > >>> > >> > output
> > >>> > >> > > >> of
> > >>> > >> > > >> > > >> "dtcli
> > >>> > >> > > >> > > >> >> > > --help"?
> > >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call version
> > of
> > >>> apex
> > >>> > >> in
> > >>> > >> > > >> Bigtop
> > >>> > >> > > >> > as
> > >>> > >> > > >> > > >> >> 3.3.0 OR
> > >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package
> of
> > >>> > bigtop
> > >>> > >> to
> > >>> > >> > > >> depend
> > >>> > >> > > >> > on
> > >>> > >> > > >> > > >> >> 2.7.1
> > >>> > >> > > >> > > >> >> > > > > version
> > >>> > >> > > >> > > >> >> > > > > > of
> > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems
> > that
> > >>> we
> > >>> > see
> > >>> > >> > with
> > >>> > >> > > >> this
> > >>> > >> > > >> > > >> >> > dependency?
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > Should work without changed. I
> > >>> thought
> > >>> > we
> > >>> > >> > > >> certified
> > >>> > >> > > >> > > >> >> against
> > >>> > >> > > >> > > >> >> > > > 2.7.0?
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache
> > mirror
> > >>> > from
> > >>> > >> > which
> > >>> > >> > > >> > bigtop
> > >>> > >> > > >> > > >> will
> > >>> > >> > > >> > > >> >> > pick
> > >>> > >> > > >> > > >> >> > > > the
> > >>> > >> > > >> > > >> >> > > > > > > apex
> > >>> > >> > > >> > > >> >> > > > > > > > > > source for compilation and
> > >>> packaging.
> > >>> > >> > Please
> > >>> > >> > > >> > > correct
> > >>> > >> > > >> > > >> if
> > >>> > >> > > >> > > >> >> > > > > incorrect:
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > >
> > >>> > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org
> is
> > >>> the
> > >>> > >> mirror
> > >>> > >> > > >> used
> > >>> > >> > > >> > by
> > >>> > >> > > >> > > >> all
> > >>> > >> > > >> > > >> >> the
> > >>> > >> > > >> > > >> >> > > > > > softwares
> > >>> > >> > > >> > > >> >> > > > > > > in
> > >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > Please share your opinion.
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19
> > AM,
> > >>> > >> Chinmay
> > >>> > >> > > >> > Kolhatkar
> > >>> > >> > > >> > > <
> > >>> > >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on
> the
> > >>> > >> packaging.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > > I've already started
> > >>> discussion on
> > >>> > >> > bigtop
> > >>> > >> > > >> dev
> > >>> > >> > > >> > > >> mailing
> > >>> > >> > > >> > > >> >> > list
> > >>> > >> > > >> > > >> >> > > > for
> > >>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also created a
> > >>> Jira
> > >>> > for
> > >>> > >> the
> > >>> > >> > > >> same.
> > >>> > >> > > >> > > For
> > >>> > >> > > >> > > >> >> this
> > >>> > >> > > >> > > >> >> > > > > > communities
> > >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is the
> bigtop
> > >>> Jira:
> > >>> > >> > > >> > BIGTOP-2313.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13
> > AM,
> > >>> > >> Thomas
> > >>> > >> > > >> Weise <
> > >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything under
> > >>> > >> > prerequisites
> > >>> > >> > > >> that
> > >>> > >> > > >> > > >> would
> > >>> > >> > > >> > > >> >> be a
> > >>> > >> > > >> > > >> >> > > > > > problem.
> > >>> > >> > > >> > > >> >> > > > > > > We
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
> > >>> compatibility
> > >>> > >> as
> > >>> > >> > > part
> > >>> > >> > > >> of
> > >>> > >> > > >> > > >> >> becoming
> > >>> > >> > > >> > > >> >> > an
> > >>> > >> > > >> > > >> >> > > > > > > incubator
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the
> > packaging
> > >>> > during
> > >>> > >> > the
> > >>> > >> > > >> next
> > >>> > >> > > >> > > >> weeks.
> > >>> > >> > > >> > > >> >> > Since
> > >>> > >> > > >> > > >> >> > > > the
> > >>> > >> > > >> > > >> >> > > > > > > work
> > >>> > >> > > >> > > >> >> > > > > > > > > will
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related
> > >>> > discussions
> > >>> > >> and
> > >>> > >> > > >> JIRAs
> > >>> > >> > > >> > > >> should
> > >>> > >> > > >> > > >> >> > also
> > >>> > >> > > >> > > >> >> > > be
> > >>> > >> > > >> > > >> >> > > > > > > there.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have the
> > >>> > packaging
> > >>> > >> in
> > >>> > >> > > >> place
> > >>> > >> > > >> > by
> > >>> > >> > > >> > > >> end
> > >>> > >> > > >> > > >> >> Feb.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at
> 10:14
> > >>> AM,
> > >>> > >> > Chinmay
> > >>> > >> > > >> > > >> Kolhatkar <
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work on
> > >>> adding
> > >>> > >> > Apache
> > >>> > >> > > >> Apex
> > >>> > >> > > >> > > as a
> > >>> > >> > > >> > > >> >> > > component
> > >>> > >> > > >> > > >> >> > > > > to
> > >>> > >> > > >> > > >> >> > > > > > > > Apache
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging
> > >>> system
> > >>> > for
> > >>> > >> > the
> > >>> > >> > > >> > Apache
> > >>> > >> > > >> > > >> big
> > >>> > >> > > >> > > >> >> data
> > >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it,
> > most
> > >>> > >> recently
> > >>> > >> > > EMR.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking
> Jira
> > >>> task
> > >>> > in
> > >>> > >> > > >> APEXCORE
> > >>> > >> > > >> > for
> > >>> > >> > > >> > > >> the
> > >>> > >> > > >> > > >> >> > same:
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >>
> https://issues.apache.org/jira/browse/APEXCORE-331
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of
> execution
> > >>> is as
> > >>> > >> > > follows:
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
> > >>> prerequisites*
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some
> > hard
> > >>> and
> > >>> > >> soft
> > >>> > >> > > >> > > expectation
> > >>> > >> > > >> > > >> >> for
> > >>> > >> > > >> > > >> >> > new
> > >>> > >> > > >> > > >> >> > > > > > > > components
> > >>> > >> > > >> > > >> >> > > > > > > > > to
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of it:
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > >
> > >>> > >> > > >> > > >> >> > > >
> > >>> > >> > > >> > > >> >> > >
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >>
> > >>> > >> > > >> > > >>
> > >>> > >> > > >> > >
> > >>> > >> > > >> >
> > >>> > >> > > >>
> > >>> > >> > >
> > >>> > >> >
> > >>> > >>
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to be
> > >>> > standard
> > >>> > >> ASF
> > >>> > >> > > >> based
> > >>> > >> > > >> > > >> >> > > requirements,
> > >>> > >> > > >> > > >> >> > > > > but
> > >>> > >> > > >> > > >> >> > > > > > > few
> > >>> > >> > > >> > > >> >> > > > > > > > > need
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects are
> > >>> > expected
> > >>> > >> to
> > >>> > >> > be
> > >>> > >> > > >> > > Licensed
> > >>> > >> > > >> > > >> >> under
> > >>> > >> > > >> > > >> >> > > > > Apache
> > >>> > >> > > >> > > >> >> > > > > > > > > License,
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
> > >>> > >> dependencies
> > >>> > >> > are
> > >>> > >> > > >> > > expected
> > >>> > >> > > >> > > >> >> to be
> > >>> > >> > > >> > > >> >> > > > > > > compatible
> > >>> > >> > > >> > > >> >> > > > > > > > > with
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL
> > 2.0
> > >>> but
> > >>> > >> need
> > >>> > >> > to
> > >>> > >> > > >> > check
> > >>> > >> > > >> > > if
> > >>> > >> > > >> > > >> >> > > > > dependencies
> > >>> > >> > > >> > > >> >> > > > > > of
> > >>> > >> > > >> > > >> >> > > > > > > > > Apex
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0.
> > >>> This I
> > >>> > >> guess
> > >>> > >> > > >> would
> > >>> > >> > > >> > > be a
> > >>> > >> > > >> > > >> >> > > > > verification
> > >>> > >> > > >> > > >> >> > > > > > > > check.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects are
> > >>> > expected
> > >>> > >> to
> > >>> > >> > be
> > >>> > >> > > >> > > >> compatible
> > >>> > >> > > >> > > >> >> > with
> > >>> > >> > > >> > > >> >> > > > all
> > >>> > >> > > >> > > >> >> > > > > of
> > >>> > >> > > >> > > >> >> > > > > > > the
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms that
> > >>> Bigtop
> > >>> > >> > > >> distribution
> > >>> > >> > > >> > > is
> > >>> > >> > > >> > > >> >> > > targeting
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> verifying
> > >>> > whether
> > >>> > >> > our
> > >>> > >> > > >> > > software
> > >>> > >> > > >> > > >> >> runs
> > >>> > >> > > >> > > >> >> > > fine
> > >>> > >> > > >> > > >> >> > > > in
> > >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
> > >>> ubuntu-14.04
> > >>> > >> > > debian-8
> > >>> > >> > > >> > > >> >> > opensuse-13.2.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that
> > >>> should
> > >>> > be
> > >>> > >> > > added
> > >>> > >> > > >> for
> > >>> > >> > > >> > > >> >> > deployment.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test
> > >>> > artifacts
> > >>> > >> > which
> > >>> > >> > > >> goes
> > >>> > >> > > >> > > >> beyond
> > >>> > >> > > >> > > >> >> > > smoke
> > >>> > >> > > >> > > >> >> > > > > test
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are
> basically
> > >>> the
> > >>> > >> > > integration
> > >>> > >> > > >> > > tests
> > >>> > >> > > >> > > >> for
> > >>> > >> > > >> > > >> >> > > > > > verification
> > >>> > >> > > >> > > >> >> > > > > > > > > after
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a
> soft
> > >>> > >> > requirement,
> > >>> > >> > > >> but
> > >>> > >> > > >> > > aim
> > >>> > >> > > >> > > >> is
> > >>> > >> > > >> > > >> >> to
> > >>> > >> > > >> > > >> >> > > > > achieve
> > >>> > >> > > >> > > >> >> > > > > > > this
> > >>> > >> > > >> > > >> >> > > > > > > > > as
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have
> > >>> explanation why
> > >>> > >> not
> > >>> > >> > to
> > >>> > >> > > >> > > include.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any from
> the
> > >>> link
> > >>> > >> which
> > >>> > >> > > >> > > explicitly
> > >>> > >> > > >> > > >> >> needs
> > >>> > >> > > >> > > >> >> > to
> > >>> > >> > > >> > > >> >> > > > be
> > >>> > >> > > >> > > >> >> > > > > > > > checked
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please let
> > us
> > >>> > know.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as
> > >>> > component
> > >>> > >> to
> > >>> > >> > > >> Bigtop*
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail
> > >>> archive of
> > >>> > >> > Bigtop,
> > >>> > >> > > >> it
> > >>> > >> > > >> > was
> > >>> > >> > > >> > > >> >> learnt
> > >>> > >> > > >> > > >> >> > > that
> > >>> > >> > > >> > > >> >> > > > > the
> > >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > community want to see
> the
> > >>> > >> addition of
> > >>> > >> > > new
> > >>> > >> > > >> > > >> >> components
> > >>> > >> > > >> > > >> >> > in
> > >>> > >> > > >> > > >> >> > > > > > phases.
> > >>> > >> > > >> > > >> >> > > > > > > > Here
> > >>> > >> > > >> > > >> >> > > > > > > > > > are
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs
> creating
> > of
> > >>> > >> package
> > >>> > >> > > i.e.
> > >>> > >> > > >> > rpm
> > >>> > >> > > >> > > &
> > >>> > >> > > >> > > >> deb
> > >>> > >> > > >> > > >> >> > > files.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> > >>> documentations/READMEs,
> > >>> > >> > LICENSE,
> > >>> > >> > > >> > > >> DISCLAMER,
> > >>> > >> > > >> > > >> >> > NOTES
> > >>> > >> > > >> > > >> >> > > > etc
> > >>> > >> > > >> > > >> >> > > > > if
> > >>> > >> > > >> > > >> >> > > > > > > any
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any documentation
> > that
> > >>> > need
> > >>> > >> to
> > >>> > >> > be
> > >>> > >> > > >> > added
> > >>> > >> > > >> > > to
> > >>> > >> > > >> > > >> >> > > > > distribution
> > >>> > >> > > >> > > >> >> > > > > > of
> > >>> > >> > > >> > > >> >> > > > > > > > our
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license
> > >>> information of
> > >>> > >> > > >> > dependencies
> > >>> > >> > > >> > > >> >> required
> > >>> > >> > > >> > > >> >> > > to
> > >>> > >> > > >> > > >> >> > > > be
> > >>> > >> > > >> > > >> >> > > > > > > added
> > >>> > >> > > >> > > >> >> > > > > > > > > to
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very
> > >>> least)
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke test
> > for
> > >>> > >> > packaging.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for
> > >>> automatic
> > >>> > >> > > >> deployment
> > >>> > >> > > >> > and
> > >>> > >> > > >> > > >> >> > > > configuration
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes
> > for
> > >>> > >> > automatic
> > >>> > >> > > >> > > >> deployment
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For verification
> of
> > >>> > >> > deployments.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to
> > make
> > >>> > sure
> > >>> > >> all
> > >>> > >> > > new
> > >>> > >> > > >> > > stuff
> > >>> > >> > > >> > > >> is
> > >>> > >> > > >> > > >> >> > > > compliant
> > >>> > >> > > >> > > >> >> > > > > > > with
> > >>> > >> > > >> > > >> >> > > > > > > > > ASF
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > license requirements. If
> > >>> you add
> > >>> > >> code
> > >>> > >> > > >> > licenses
> > >>> > >> > > >> > > >> >> under
> > >>> > >> > > >> > > >> >> > > > > different
> > >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need to be
> > >>> listed in
> > >>> > >> the
> > >>> > >> > > >> NOTICE.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your
> thoughts
> > >>> on
> > >>> > the
> > >>> > >> > > >> approach.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start
> corresponding
> > >>> > >> > communication
> > >>> > >> > > >> on
> > >>> > >> > > >> > > >> bigtop
> > >>> > >> > > >> > > >> >> > > mailing
> > >>> > >> > > >> > > >> >> > > > > list
> > >>> > >> > > >> > > >> >> > > > > > > as
> > >>> > >> > > >> > > >> >> > > > > > > > > > well.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some specific
> > >>> > >> > > >> questions/suggestions
> > >>> > >> > > >> > > >> >> related to
> > >>> > >> > > >> > > >> >> > > > what
> > >>> > >> > > >> > > >> >> > > > > > > should
> > >>> > >> > > >> > > >> >> > > > > > > > > be
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the package
> and
> > >>> what
> > >>> > >> > should
> > >>> > >> > > be
> > >>> > >> > > >> > the
> > >>> > >> > > >> > > >> smoke
> > >>> > >> > > >> > > >> >> > > tests,
> > >>> > >> > > >> > > >> >> > > > > but
> > >>> > >> > > >> > > >> >> > > > > > > in
> > >>> > >> > > >> > > >> >> > > > > > > > > the
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not having
> too
> > >>> much
> > >>> > >> > content
> > >>> > >> > > >> > here,
> > >>> > >> > > >> > > >> we'll
> > >>> > >> > > >> > > >> >> > put
> > >>> > >> > > >> > > >> >> > > > the
> > >>> > >> > > >> > > >> >> > > > > > > > > questions
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this
> > >>> > mailthread.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >>
> > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > > >
> > >>> > >> > > >> > > >> >> > > > > > >
> > >>> > >> > > >> > > >> >> > > > > >
> > >>> > >> > > >> > > >> >> > > > >
> > >>> > >> > > >> > > >> >> > > >
> > >>> > >> > > >> > > >> >> > >
> > >>> > >> > > >> > > >> >> >
> > >>> > >> > > >> > > >> >>
> > >>> > >> > > >> > > >> >
> > >>> > >> > > >> > > >> >
> > >>> > >> > > >> > > >>
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > > >
> > >>> > >> > > >> > >
> > >>> > >> > > >> >
> > >>> > >> > > >>
> > >>> > >> > > >
> > >>> > >> > > >
> > >>> > >> > >
> > >>> > >> >
> > >>> > >>
> > >>> > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Hi,

I've attached man page for apex in Jira. Here is the link to it:
https://issues.apache.org/jira/secure/attachment/12788403/apex.1

Please view it by: man -l apex.1

Thanks,
Chinmay.


On Thu, Feb 18, 2016 at 12:51 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:

> Hi Chinmay,
>
> I am not able to find the attachment, may be attachments are not allowed in
> the mailing list. Can you attach the man page to the Jira?
>
> - Tushar.
>
>
> On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Hi All,
> >
> > I've attached man page file for the purpose of bigtop integration.
> > Please have a look and let me know if there any any feedback about it.
> >
> > You can see the man page content of this file with:
> > man -l apex.1
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> > On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> >> Thanks. Will follow bigtop recommendation. i.e. remove "-incubating".
> >>
> >> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <th...@datatorrent.com>
> >> wrote:
> >>
> >>> Please follow the recommendation and continue discussion on the Bigtop
> >>> list.
> >>>
> >>> --
> >>> sent from mobile
> >>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <ch...@apache.org>
> wrote:
> >>>
> >>> > I got a recommendation from bigtop community to drop "-incubating"
> from
> >>> > version.
> >>> >
> >>> > Here is the mail thread:
> >>> >
> >>> >
> >>>
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> >>> >
> >>> > Please let me know if removing "-incubating" is fine.
> >>> >
> >>> > Thanks,
> >>> > Chinmay.
> >>> >
> >>> >
> >>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <
> chinmay@apache.org
> >>> >
> >>> > wrote:
> >>> >
> >>> > > I've already started communication there.
> >>> > >
> >>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
> >>> thomas@datatorrent.com>
> >>> > > wrote:
> >>> > >
> >>> > >> This is a question you should ask on the Bigtop mailing list.
> >>> > >>
> >>> > >> --
> >>> > >> sent from mobile
> >>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org>
> >>> wrote:
> >>> > >>
> >>> > >> > Yes. In case of RPM, the only restriction on version is that it
> >>> can
> >>> > not
> >>> > >> > contain '-'.
> >>> > >> > However, as a workaround, one can use 'incubating' as a part of
> >>> > >> %{release}.
> >>> > >> >
> >>> > >> > Version: 3.3.0
> >>> > >> > Release: incubating
> >>> > >> >
> >>> > >> > apex-3.3.0-incubating.noarch.rpm
> >>> > >> >
> >>> > >> > It is the same practice followed for pre-release packages e.g.
> >>> with
> >>> > rcX,
> >>> > >> > alphaX, betaX being used in %{release}.
> >>> > >> >
> >>> > >> > Regards,
> >>> > >> > Pradeep A. Dalvi
> >>> > >> >
> >>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> >>> > chinmay@apache.org>
> >>> > >> > wrote:
> >>> > >> >
> >>> > >> > > Hi Everyone,
> >>> > >> > >
> >>> > >> > > In one of the previous mails in this thread, we decided that
> >>> version
> >>> > >> of
> >>> > >> > the
> >>> > >> > > package will be "3.3.0-incubating".
> >>> > >> > > The problem that I'm facing is in RPM base version "-" is not
> >>> > allowed.
> >>> > >> > >
> >>> > >> > > Can we have package version to "3.3.0"?
> >>> > >> > > i.e. the packages created will be as follows:
> >>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> >>> > >> > > For DEB: apex_3.3.0-1_all.deb
> >>> > >> > >
> >>> > >> > > There is another integration for Zepplin in Bigtop which is
> >>> > incubating
> >>> > >> > and
> >>> > >> > > is having base version as "0.5.6".
> >>> > >> > >
> >>> > >> > > Please let me know your thoughts.
> >>> > >> > >
> >>> > >> > > Thanks,
> >>> > >> > > Chinmay.
> >>> > >> > >
> >>> > >> > >
> >>> > >> > >
> >>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> >>> > >> chinmay@apache.org>
> >>> > >> > > wrote:
> >>> > >> > >
> >>> > >> > > > Thanks everyone for the feedback.
> >>> > >> > > >
> >>> > >> > > > Also as Ram and David mentioned, I'll start creating the man
> >>> page
> >>> > as
> >>> > >> > > well.
> >>> > >> > > >
> >>> > >> > > > Thanks,
> >>> > >> > > > Chinmay.
> >>> > >> > > >
> >>> > >> > > >
> >>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> >>> > >> thomas@datatorrent.com>
> >>> > >> > > > wrote:
> >>> > >> > > >
> >>> > >> > > >> I think we already agreed that we will retain the current
> >>> script
> >>> > >> name
> >>> > >> > in
> >>> > >> > > >> this bigtop iteration. There is a ticket for renaming, add
> >>> you
> >>> > >> > > suggestions
> >>> > >> > > >> there.
> >>> > >> > > >>
> >>> > >> > > >> --
> >>> > >> > > >> sent from mobile
> >>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> >>> > sandesh@datatorrent.com>
> >>> > >> > > wrote:
> >>> > >> > > >>
> >>> > >> > > >> > +1 for Symlink from apex to dtcli or start the transition
> >>> > >> process by
> >>> > >> > > >> making
> >>> > >> > > >> > dtcli as the symlink to apex cli.
> >>> > >> > > >> >
> >>> > >> > > >> > Renaming is a breaking change should be done during major
> >>> > version
> >>> > >> > > number
> >>> > >> > > >> > change.
> >>> > >> > > >> >
> >>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> >>> > david@datatorrent.com
> >>> > >> >
> >>> > >> > > >> wrote:
> >>> > >> > > >> >
> >>> > >> > > >> > > Apologies for hitting the "Send" button too early.
> >>> > >> > > >> > >
> >>> > >> > > >> > > What is the scope of the name change?  If we simply
> have
> >>> a
> >>> > >> symlink
> >>> > >> > > >> from
> >>> > >> > > >> > > apex to dtcli, then it should have no backward
> >>> compatibility
> >>> > >> > problem
> >>> > >> > > >> at
> >>> > >> > > >> > > all.
> >>> > >> > > >> > >
> >>> > >> > > >> > > But the proper name change would involve:
> >>> > >> > > >> > > - Change the name of DTCli class to ApexCli
> >>> > >> > > >> > > - Change the prompt from dt> to apex>
> >>> > >> > > >> > > - Change the initial greeting when running the CLI from
> >>> > >> something
> >>> > >> > > like
> >>> > >> > > >> > "DT
> >>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> >>> > >> > > >> > >
> >>> > >> > > >> > > The above changes may break existing scripts that rely
> >>> on the
> >>> > >> the
> >>> > >> > > >> string
> >>> > >> > > >> > > "DT".
> >>> > >> > > >> > > But if folks are okay with this potential problem, I am
> >>> okay
> >>> > >> with
> >>> > >> > > the
> >>> > >> > > >> > name
> >>> > >> > > >> > > change.
> >>> > >> > > >> > >
> >>> > >> > > >> > > For the man page, we can start with the content of this
> >>> page:
> >>> > >> > > >> > >
> >>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> >>> > >> > > >> > >
> >>> > >> > > >> > > Plus the command line options when doing dtcli -h:
> >>> > >> > > >> > >
> >>> > >> > > >> > > usage: DTCli
> >>> > >> > > >> > >  -e <arg>    Commands are read from the argument
> >>> > >> > > >> > >  -f <arg>    Use the specified prompt at all time
> >>> > >> > > >> > >  -h          Print this help
> >>> > >> > > >> > >  -kp <arg>   Use the specified kerberos principal
> >>> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> >>> > >> > > >> > >  -p <arg>    JSONP padding function
> >>> > >> > > >> > >  -r          JSON Raw mode
> >>> > >> > > >> > >  -v          Verbose mode level 1
> >>> > >> > > >> > >  -vv         Verbose mode level 2
> >>> > >> > > >> > >  -vvv        Verbose mode level 3
> >>> > >> > > >> > >  -vvvv       Verbose mode level 4
> >>> > >> > > >> > >
> >>> > >> > > >> > > David
> >>> > >> > > >> > >
> >>> > >> > > >> > >
> >>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> >>> > >> david@datatorrent.com
> >>> > >> > >
> >>> > >> > > >> > wrote:
> >>> > >> > > >> > >
> >>> > >> > > >> > > > What is the scope of the name change?  If we simply
> >>> have a
> >>> > >> > symlink
> >>> > >> > > >> from
> >>> > >> > > >> > > > apex to dtcli, then it should have no backward
> >>> > compatibility
> >>> > >> > > >> problem at
> >>> > >> > > >> > > all.
> >>> > >> > > >> > > >
> >>> > >> > > >> > > > But the proper name change would involve:
> >>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
> >>> > >> > > >> > > > - Change the prompt from dt> to apex>
> >>> > >> > > >> > > > - Change the initial greeting when running the CLI
> from
> >>> > >> > something
> >>> > >> > > >> like
> >>> > >> > > >> > > "DT
> >>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> >>> > >> > > >> > > >
> >>> > >> > > >> > > > The above changes may break existing scripts that
> rely
> >>> on
> >>> > the
> >>> > >> > the
> >>> > >> > > >> > string
> >>> > >> > > >> > > > "DT".
> >>> > >> > > >> > > > But if folks are okay with this potential problem, I
> am
> >>> > okay
> >>> > >> > with
> >>> > >> > > >> the
> >>> > >> > > >> > > name
> >>> > >> > > >> > > > change.
> >>> > >> > > >> > > >
> >>> > >> > > >> > > > For the man page, we can start with the content of
> this
> >>> > page:
> >>> > >> > > >> > > >
> >>> > >> > > >> > > >
> >>> > >> > > >> > > >
> >>> > >> > > >> > > >
> >>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> >>> > >> > > >> chinmay@apache.org
> >>> > >> > > >> > >
> >>> > >> > > >> > > > wrote:
> >>> > >> > > >> > > >
> >>> > >> > > >> > > >> Here is a suggestion I have related to name change:
> >>> > >> > > >> > > >>
> >>> > >> > > >> > > >> 1. While packaging we change the name of "dtcli"
> >>> script to
> >>> > >> > > "apex".
> >>> > >> > > >> > This
> >>> > >> > > >> > > is
> >>> > >> > > >> > > >> done so that in future when we change the name to
> >>> "apex",
> >>> > >> users
> >>> > >> > > of
> >>> > >> > > >> > > bigtop
> >>> > >> > > >> > > >> apex don't have to transition much.
> >>> > >> > > >> > > >> 2. We also keep a symlink named dtcli which points
> to
> >>> apex
> >>> > >> > > script.
> >>> > >> > > >> > This
> >>> > >> > > >> > > is
> >>> > >> > > >> > > >> for backward compatibility.
> >>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first
> integration
> >>> in
> >>> > >> > bigtop.
> >>> > >> > > >> This
> >>> > >> > > >> > is
> >>> > >> > > >> > > >> in
> >>> > >> > > >> > > >> the interest of not changing the content of dtcli or
> >>> apex
> >>> > >> > script.
> >>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these naming
> >>> related
> >>> > >> > > changes
> >>> > >> > > >> and
> >>> > >> > > >> > > >> update bigtop repository later to remove references
> to
> >>> > dtcli
> >>> > >> > all
> >>> > >> > > >> > > together.
> >>> > >> > > >> > > >>
> >>> > >> > > >> > > >> Please share your thoughts on above approach.
> >>> > >> > > >> > > >>
> >>> > >> > > >> > > >> Also, please share what could be the man page
> content
> >>> for
> >>> > >> > "apex".
> >>> > >> > > >> > > >>
> >>> > >> > > >> > > >> Thanks,
> >>> > >> > > >> > > >> Chinmay.
> >>> > >> > > >> > > >>
> >>> > >> > > >> > > >>
> >>> > >> > > >> > > >>
> >>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> >>> > >> > > >> > chinmay@apache.org>
> >>> > >> > > >> > > >> wrote:
> >>> > >> > > >> > > >>
> >>> > >> > > >> > > >> > Yes.. It'll be build from source tar downloaded
> >>> from one
> >>> > >> of
> >>> > >> > the
> >>> > >> > > >> > apache
> >>> > >> > > >> > > >> > mirror: apache.osuosl.org
> >>> > >> > > >> > > >> >
> >>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced from
> dtcli.
> >>> > This
> >>> > >> > > leaves
> >>> > >> > > >> us
> >>> > >> > > >> > 3
> >>> > >> > > >> > > >> > options:
> >>> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli would
> >>> need a
> >>> > >> > change
> >>> > >> > > >> while
> >>> > >> > > >> > > >> > packaging.
> >>> > >> > > >> > > >> >
> >>> > >> > > >> > > >> > But if we want to use dtcli as it is, then we
> would
> >>> need
> >>> > >> > > >> dt-env.sh
> >>> > >> > > >> > > >> >
> >>> > >> > > >> > > >> >
> >>> > >> > > >> > > >> >
> >>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> >>> > >> > > >> > thomas@datatorrent.com
> >>> > >> > > >> > > >
> >>> > >> > > >> > > >> > wrote:
> >>> > >> > > >> > > >> >
> >>> > >> > > >> > > >> >> Looks good overall, though there shouldn't be any
> >>> > >> dt-env.sh
> >>> > >> > > >> > > >> >>
> >>> > >> > > >> > > >> >> What will the bigtop package be built from, the
> >>> source
> >>> > >> tar?
> >>> > >> > > >> > > >> >>
> >>> > >> > > >> > > >> >>
> >>> > >> > > >> > > >> >>
> >>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay
> Kolhatkar
> >>> <
> >>> > >> > > >> > > chinmay@apache.org
> >>> > >> > > >> > > >> >
> >>> > >> > > >> > > >> >> wrote:
> >>> > >> > > >> > > >> >>
> >>> > >> > > >> > > >> >> > Hi Thomas,
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > Thanks for the feedback.
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > First, we're not changing the name anywhere.
> >>> We'll
> >>> > >> follow
> >>> > >> > > what
> >>> > >> > > >> > > >> >> currently is
> >>> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating version.
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory structure
> >>> below
> >>> > >> which
> >>> > >> > > is
> >>> > >> > > >> > > inline
> >>> > >> > > >> > > >> >> with
> >>> > >> > > >> > > >> >> > other existing integrations with bigtop.
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > The need for each file is as follows:
> >>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> >>> > >> > > >> engine/src/main/scripts/
> >>> > >> > > >> > > of
> >>> > >> > > >> > > >> >> source
> >>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli.
> >>> This
> >>> > >> > > searches
> >>> > >> > > >> for
> >>> > >> > > >> > > >> hadoop
> >>> > >> > > >> > > >> >> > binary path and exports an env variable for
> >>> dtcli to
> >>> > >> use
> >>> > >> > it.
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is
> >>> extracted
> >>> > >> in a
> >>> > >> > > >> similar
> >>> > >> > > >> > > >> way to
> >>> > >> > > >> > > >> >> > how DT Community Edition finds dependency jars.
> >>> > >> > > >> > > >> >> > Please note that this is the first iteration
> >>> list of
> >>> > >> jars.
> >>> > >> > > I'm
> >>> > >> > > >> > > >> trying to
> >>> > >> > > >> > > >> >> > narrow this down to only those which are really
> >>> > >> required.
> >>> > >> > > >> > > >> >> > The test that I'm running to check if dtcli
> runs
> >>> fine
> >>> > >> with
> >>> > >> > > >> given
> >>> > >> > > >> > > >> >> dependency
> >>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi demo.
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > Above is the directory structure which is
> >>> required
> >>> > for
> >>> > >> CLI
> >>> > >> > > to
> >>> > >> > > >> > work.
> >>> > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned are
> some
> >>> > >> > additional
> >>> > >> > > >> > files
> >>> > >> > > >> > > >> which
> >>> > >> > > >> > > >> >> > makes the like of administrator easier. For eg.
> >>> > >> /etc/skel
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > Please let me know if this is inline with what
> >>> you're
> >>> > >> > > >> thinking.
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > Thanks,
> >>> > >> > > >> > > >> >> > Chinmay.
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > .
> >>> > >> > > >> > > >> >> > |-- bin
> >>> > >> > > >> > > >> >> > |   `-- dtcli
> >>> > >> > > >> > > >> >> > |-- conf
> >>> > >> > > >> > > >> >> > |   `-- dt-env.sh
> >>> > >> > > >> > > >> >> > `-- lib
> >>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> >>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> >>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> >>> > >> > > >> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> >>> > >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> >>> > >> > > >> > > >> >> >     |-- apex-engine.jar
> >>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> >>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> >>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> >>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> >>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> >>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> >>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> >>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> >>> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> >>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> >>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> >>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> >>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> >>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> >>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> >>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> >>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> >>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> >>> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> >>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> >>> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> >>> > >> > > >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> >>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> >>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> >>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> >>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> >>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> >>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> >>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> >>> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> >>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> >>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > 3 directories, 40 files
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> >>> > >> > > >> > > >> thomas@datatorrent.com>
> >>> > >> > > >> > > >> >> > wrote:
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >> > > Chinmay,
> >>> > >> > > >> > > >> >> > >
> >>> > >> > > >> > > >> >> > > Before discussing where to put the files,
> let's
> >>> > make
> >>> > >> > sure
> >>> > >> > > >> they
> >>> > >> > > >> > > are
> >>> > >> > > >> > > >> >> really
> >>> > >> > > >> > > >> >> > > needed for the operation of the CLI. As for
> >>> names,
> >>> > >> > > anything
> >>> > >> > > >> new
> >>> > >> > > >> > > >> needs
> >>> > >> > > >> > > >> >> to
> >>> > >> > > >> > > >> >> > > reflect Apex in the name and should follow
> >>> common
> >>> > >> > > >> conventions,
> >>> > >> > > >> > > >> >> especially
> >>> > >> > > >> > > >> >> > > in Bigtop where there are many existing
> >>> > integrations
> >>> > >> to
> >>> > >> > > look
> >>> > >> > > >> > at.
> >>> > >> > > >> > > >> >> > >
> >>> > >> > > >> > > >> >> > > Thanks,
> >>> > >> > > >> > > >> >> > > Thomas
> >>> > >> > > >> > > >> >> > >
> >>> > >> > > >> > > >> >> > >
> >>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay
> >>> Kolhatkar
> >>> > <
> >>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> >>> > >> > > >> > > >> >> > > wrote:
> >>> > >> > > >> > > >> >> > >
> >>> > >> > > >> > > >> >> > > > Really good points Pradeep and Aniruddha.
> >>> > >> > > >> > > >> >> > > >
> >>> > >> > > >> > > >> >> > > > 1. I believe we won't need to change the
> >>> dtcli
> >>> > >> > > >> considering it
> >>> > >> > > >> > > >> works
> >>> > >> > > >> > > >> >> > with
> >>> > >> > > >> > > >> >> > > DT
> >>> > >> > > >> > > >> >> > > > Community edition. We can keep the
> directory
> >>> > >> structure
> >>> > >> > > >> > similar
> >>> > >> > > >> > > to
> >>> > >> > > >> > > >> >> that.
> >>> > >> > > >> > > >> >> > > > dt-env.sh has variables which contains
> >>> > information
> >>> > >> > > >> required
> >>> > >> > > >> > for
> >>> > >> > > >> > > >> >> dtcli
> >>> > >> > > >> > > >> >> > to
> >>> > >> > > >> > > >> >> > > > launch.
> >>> > >> > > >> > > >> >> > > >
> >>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop community that
> >>> > whether
> >>> > >> > they
> >>> > >> > > >> > > >> facilitate
> >>> > >> > > >> > > >> >> the
> >>> > >> > > >> > > >> >> > > > installation of rpms/debs before user is
> >>> created.
> >>> > >> In
> >>> > >> > > >> either
> >>> > >> > > >> > > case,
> >>> > >> > > >> > > >> >> > current
> >>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home
> directory.
> >>> > Also
> >>> > >> > > before
> >>> > >> > > >> > > putting
> >>> > >> > > >> > > >> >> > > anything
> >>> > >> > > >> > > >> >> > > > in /etc/skel we need to define what are the
> >>> > default
> >>> > >> > > >> contents
> >>> > >> > > >> > > that
> >>> > >> > > >> > > >> >> > should
> >>> > >> > > >> > > >> >> > > go
> >>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults,
> >>> > >> probably we
> >>> > >> > > >> should
> >>> > >> > > >> > > not
> >>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> >>> > >> > > >> > > >> >> > > >
> >>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice.
> This
> >>> way
> >>> > >> > > contents
> >>> > >> > > >> of
> >>> > >> > > >> > > >> >> dt-env.sh
> >>> > >> > > >> > > >> >> > > are
> >>> > >> > > >> > > >> >> > > > present as env variables. But I see a catch
> >>> > there.
> >>> > >> > > Adding
> >>> > >> > > >> > > >> dt-env.sh
> >>> > >> > > >> > > >> >> to
> >>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the variables
> >>> > >> available
> >>> > >> > at
> >>> > >> > > >> > > runtime
> >>> > >> > > >> > > >> all
> >>> > >> > > >> > > >> >> > the
> >>> > >> > > >> > > >> >> > > > time. I feel a little skeptical about that.
> >>> > Maybe a
> >>> > >> > > >> possible
> >>> > >> > > >> > > >> >> collision
> >>> > >> > > >> > > >> >> > > can
> >>> > >> > > >> > > >> >> > > > occur with other application vars.
> >>> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
> >>> > >> > "../conf/dt-env.sh"
> >>> > >> > > >> and
> >>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are
> >>> anyway
> >>> > >> > > >> available.
> >>> > >> > > >> > > >> >> > > >
> >>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is
> >>> happening
> >>> > at
> >>> > >> > > Bigtop
> >>> > >> > > >> > > >> >> > > Jira/repository.
> >>> > >> > > >> > > >> >> > > > Packaging related code usually goes there.
> >>> > (That's
> >>> > >> > what
> >>> > >> > > >> all
> >>> > >> > > >> > the
> >>> > >> > > >> > > >> >> > > components
> >>> > >> > > >> > > >> >> > > > in bigtop does).
> >>> > >> > > >> > > >> >> > > > Having said that, once a PR is created,
> I'll
> >>> be
> >>> > >> > sharing
> >>> > >> > > >> the
> >>> > >> > > >> > > link
> >>> > >> > > >> > > >> of
> >>> > >> > > >> > > >> >> PR
> >>> > >> > > >> > > >> >> > > > here, so that apex community as well can
> >>> review
> >>> > it.
> >>> > >> > > >> > > >> >> > > >
> >>> > >> > > >> > > >> >> > > > Thanks,
> >>> > >> > > >> > > >> >> > > > Chinmay.
> >>> > >> > > >> > > >> >> > > >
> >>> > >> > > >> > > >> >> > > >
> >>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha
> >>> > >> Thombare <
> >>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> >>> > >> > > >> > > >> >> > > >
> >>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> >>> > >> > > >> > > >> >> > > > > We may need to iron out details about
> exact
> >>> > paths
> >>> > >> > etc.
> >>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR comments.
> >>> > >> > > >> > > >> >> > > > > Is that right @dev?
> >>> > >> > > >> > > >> >> > > > >
> >>> > >> > > >> > > >> >> > > > >
> >>> > >> > > >> > > >> >> > > > >
> >>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A.
> >>> Dalvi <
> >>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> >>> > >> > > >> > > >> >> > > > > wrote:
> >>> > >> > > >> > > >> >> > > > >
> >>> > >> > > >> > > >> >> > > > > > Inline comments...
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM,
> Chinmay
> >>> > >> > Kolhatkar <
> >>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> >>> > >> > > >> > > >> >> > > > > > wrote:
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present in
> DT
> >>> > >> community
> >>> > >> > > >> > edition
> >>> > >> > > >> > > >> will
> >>> > >> > > >> > > >> >> be
> >>> > >> > > >> > > >> >> > > > > included
> >>> > >> > > >> > > >> >> > > > > > > there.
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I found from
> >>> > common
> >>> > >> > > >> > dependencies
> >>> > >> > > >> > > >> >> between
> >>> > >> > > >> > > >> >> > > of
> >>> > >> > > >> > > >> >> > > > > apex
> >>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> >>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> >>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> >>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> >>> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> >>> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> >>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> >>> > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> >>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> >>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> >>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> >>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> >>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> >>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> >>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> >>> > >> > > >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> >>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> >>> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> >>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> >>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> >>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> >>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> >>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> >>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> >>> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some tests
> do
> >>> > check
> >>> > >> > that
> >>> > >> > > >> > dtcli
> >>> > >> > > >> > > >> works
> >>> > >> > > >> > > >> >> > > > properly
> >>> > >> > > >> > > >> >> > > > > > for
> >>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with
> only
> >>> > these
> >>> > >> > > >> libraries
> >>> > >> > > >> > > >> >> present in
> >>> > >> > > >> > > >> >> > > > > > isolation
> >>> > >> > > >> > > >> >> > > > > > > without dependency on local m2.
> >>> > >> > > >> > > >> >> > > > > > > I'm believe that there are unwanted
> >>> jars
> >>> > >> which
> >>> > >> > are
> >>> > >> > > >> used
> >>> > >> > > >> > > for
> >>> > >> > > >> > > >> >> > compile
> >>> > >> > > >> > > >> >> > > > > time
> >>> > >> > > >> > > >> >> > > > > > > dependency and not runtime in above
> >>> list
> >>> > >> which I
> >>> > >> > > can
> >>> > >> > > >> > drop
> >>> > >> > > >> > > >> to
> >>> > >> > > >> > > >> >> keep
> >>> > >> > > >> > > >> >> > > > > package
> >>> > >> > > >> > > >> >> > > > > > > size minimal.
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> >>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all
> of
> >>> the
> >>> > >> > > >> dt-env.sh is
> >>> > >> > > >> > > >> >> required.
> >>> > >> > > >> > > >> >> > > > Only
> >>> > >> > > >> > > >> >> > > > > > > required values I see in it are
> >>> classpath.
> >>> > >> and
> >>> > >> > > >> dtcli is
> >>> > >> > > >> > > >> >> capable
> >>> > >> > > >> > > >> >> > of
> >>> > >> > > >> > > >> >> > > > > > building
> >>> > >> > > >> > > >> >> > > > > > > it on the fly.
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to
> >>> see
> >>> > how
> >>> > >> > dtcli
> >>> > >> > > >> can
> >>> > >> > > >> > > find
> >>> > >> > > >> > > >> >> the
> >>> > >> > > >> > > >> >> > jar
> >>> > >> > > >> > > >> >> > > > > path,
> >>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf files. If
> >>> we
> >>> > >> need a
> >>> > >> > > >> change
> >>> > >> > > >> > in
> >>> > >> > > >> > > >> >> dtcli,
> >>> > >> > > >> > > >> >> > > then
> >>> > >> > > >> > > >> >> > > > > > > community's opinion is required for
> >>> whether
> >>> > >> > dtcli
> >>> > >> > > >> > should
> >>> > >> > > >> > > >> >> change
> >>> > >> > > >> > > >> >> > in
> >>> > >> > > >> > > >> >> > > > our
> >>> > >> > > >> > > >> >> > > > > > repo
> >>> > >> > > >> > > >> >> > > > > > > or a copy of that with required
> changes
> >>> > >> exist in
> >>> > >> > > >> bigtop
> >>> > >> > > >> > > >> repo
> >>> > >> > > >> > > >> >> > until
> >>> > >> > > >> > > >> >> > > we
> >>> > >> > > >> > > >> >> > > > > > make
> >>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > > Can we have these file paths set using
> >>> > >> environment
> >>> > >> > > >> > > variables
> >>> > >> > > >> > > >> >> with
> >>> > >> > > >> > > >> >> > > some
> >>> > >> > > >> > > >> >> > > > > > default values in dtcli?
> >>> > >> > > >> > > >> >> > > > > > Then we can set such params in
> dt-env.sh.
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file
> >>> > different
> >>> > >> > from
> >>> > >> > > >> > > >> >> dt-site.xml?
> >>> > >> > > >> > > >> >> > Not
> >>> > >> > > >> > > >> >> > > > > sure
> >>> > >> > > >> > > >> >> > > > > > if
> >>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help.
> Files &
> >>> > Dirs
> >>> > >> > from
> >>> > >> > > >> > > /etc/skel
> >>> > >> > > >> > > >> >> are
> >>> > >> > > >> > > >> >> > > > copied
> >>> > >> > > >> > > >> >> > > > > to
> >>> > >> > > >> > > >> >> > > > > > > home of new user when useradd program
> >>> is
> >>> > >> called.
> >>> > >> > > But
> >>> > >> > > >> > for
> >>> > >> > > >> > > >> >> existing
> >>> > >> > > >> > > >> >> > > > users
> >>> > >> > > >> > > >> >> > > > > > > this won't be of any use. Moreover,
> >>> dtcli
> >>> > >> > creates
> >>> > >> > > >> > ~/.dt/
> >>> > >> > > >> > > on
> >>> > >> > > >> > > >> >> the
> >>> > >> > > >> > > >> >> > fly
> >>> > >> > > >> > > >> >> > > > if
> >>> > >> > > >> > > >> >> > > > > > not
> >>> > >> > > >> > > >> >> > > > > > > encountered for the first time. Again
> >>> > >> correct me
> >>> > >> > > if
> >>> > >> > > >> I'm
> >>> > >> > > >> > > >> wrong.
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > > Yes, you are right. However this would
> >>> > >> probably be
> >>> > >> > > >> > > necessary
> >>> > >> > > >> > > >> >> step
> >>> > >> > > >> > > >> >> > for
> >>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get installed
> >>> > during
> >>> > >> OS
> >>> > >> > > >> install
> >>> > >> > > >> > > and
> >>> > >> > > >> > > >> >> > before
> >>> > >> > > >> > > >> >> > > > user
> >>> > >> > > >> > > >> >> > > > > > accounts were created.
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile ->
> >>> I'm
> >>> > not
> >>> > >> > sure
> >>> > >> > > >> its
> >>> > >> > > >> > the
> >>> > >> > > >> > > >> best
> >>> > >> > > >> > > >> >> > > idea.
> >>> > >> > > >> > > >> >> > > > > > We'll
> >>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location which is
> >>> by
> >>> > >> default
> >>> > >> > > >> > present
> >>> > >> > > >> > > in
> >>> > >> > > >> > > >> >> path
> >>> > >> > > >> > > >> >> > > i.e.
> >>> > >> > > >> > > >> >> > > > > > > /usr/bin.
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > > For other env variables specific to
> apex,
> >>> > I'll
> >>> > >> > > prefer
> >>> > >> > > >> to
> >>> > >> > > >> > > use
> >>> > >> > > >> > > >> >> > > dt-env.sh
> >>> > >> > > >> > > >> >> > > > > and
> >>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather than
> changing
> >>> > >> bashrc
> >>> > >> > > >> etc...
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
> >>> /etc/profile.d/
> >>> > >> shall
> >>> > >> > > be
> >>> > >> > > >> the
> >>> > >> > > >> > > >> >> cleaner
> >>> > >> > > >> > > >> >> > way
> >>> > >> > > >> > > >> >> > > > to
> >>> > >> > > >> > > >> >> > > > > > achieve the same.
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > > Thanks,
> >>> > >> > > >> > > >> >> > > > > > > Chinmay.
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM,
> >>> Aniruddha
> >>> > >> > > Thombare
> >>> > >> > > >> <
> >>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > Hi,
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > We need to consider following:
> >>> > >> > > >> > > >> >> > > > > > > > System wide default config files
> can
> >>> be
> >>> > >> > located
> >>> > >> > > at
> >>> > >> > > >> > > >> following
> >>> > >> > > >> > > >> >> > > > > locations:
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have
> to
> >>> > change
> >>> > >> > > dtcli
> >>> > >> > > >> > > >> behaviour
> >>> > >> > > >> > > >> >> on
> >>> > >> > > >> > > >> >> > > how
> >>> > >> > > >> > > >> >> > > > it
> >>> > >> > > >> > > >> >> > > > > > > finds
> >>> > >> > > >> > > >> >> > > > > > > > those locations)
> >>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have
> to
> >>> > >> change
> >>> > >> > > dtcli
> >>> > >> > > >> > > >> >> behaviour on
> >>> > >> > > >> > > >> >> > > how
> >>> > >> > > >> > > >> >> > > > > it
> >>> > >> > > >> > > >> >> > > > > > > > finds those locations)
> >>> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and
> other
> >>> > files
> >>> > >> > (for
> >>> > >> > > >> new
> >>> > >> > > >> > > users
> >>> > >> > > >> > > >> >> that
> >>> > >> > > >> > > >> >> > > will
> >>> > >> > > >> > > >> >> > > > > be
> >>> > >> > > >> > > >> >> > > > > > > > created in system in future.)
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > We may also have to modify bashrc /
> >>> > >> > bashprofile
> >>> > >> > > >> for
> >>> > >> > > >> > > >> >> population
> >>> > >> > > >> > > >> >> > > any
> >>> > >> > > >> > > >> >> > > > > > > > variables that are required.
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your comments /
> >>> > >> > suggestion.
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > Thanks,
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > Aniruddha
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM,
> >>> Thomas
> >>> > >> Weise
> >>> > >> > <
> >>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> >>> > >> > > >> > > >> >> > > > > > > > wrote:
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > -->
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM,
> >>> Chinmay
> >>> > >> > > >> Kolhatkar <
> >>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > wrote:
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging
> effort
> >>> > >> (rpm+deb)
> >>> > >> > > of
> >>> > >> > > >> > apex,
> >>> > >> > > >> > > >> here
> >>> > >> > > >> > > >> >> > are
> >>> > >> > > >> > > >> >> > > > some
> >>> > >> > > >> > > >> >> > > > > > > > > proposals
> >>> > >> > > >> > > >> >> > > > > > > > > > about package structure etc..
> >>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on bbigtop
> >>> mailing
> >>> > >> > list, I
> >>> > >> > > >> have
> >>> > >> > > >> > > >> some
> >>> > >> > > >> > > >> >> > > question
> >>> > >> > > >> > > >> >> > > > > for
> >>> > >> > > >> > > >> >> > > > > > > > apex
> >>> > >> > > >> > > >> >> > > > > > > > > > community.
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory structure of
> >>> apex
> >>> > >> > package
> >>> > >> > > >> for
> >>> > >> > > >> > > both
> >>> > >> > > >> > > >> >> deb &
> >>> > >> > > >> > > >> >> > > > rpm:
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> >>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other
> >>> dependent
> >>> > >> jars>
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> >>> > /usr/lib/apex/bin/dtcli
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> /usr/share/doc/man/man1/dtcli.1.gz
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > /usr/share/doc/apex/license/LICENSE.txt
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> >>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> >>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> >>> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> = All
> the
> >>> 3rd
> >>> > >> party
> >>> > >> > > >> jars
> >>> > >> > > >> > > which
> >>> > >> > > >> > > >> >> are
> >>> > >> > > >> > > >> >> > > > > required
> >>> > >> > > >> > > >> >> > > > > > > for
> >>> > >> > > >> > > >> >> > > > > > > > > apex
> >>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the
> dependencies
> >>> are
> >>> > >> > > packaged
> >>> > >> > > >> as
> >>> > >> > > >> > > >> part of
> >>> > >> > > >> > > >> >> > > > rpm/deb
> >>> > >> > > >> > > >> >> > > > > by
> >>> > >> > > >> > > >> >> > > > > > > any
> >>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > Can you specify what those jars
> >>> are in
> >>> > >> > Bigtop
> >>> > >> > > >> > > context?
> >>> > >> > > >> > > >> >> Same
> >>> > >> > > >> > > >> >> > as
> >>> > >> > > >> > > >> >> > > > > > shipped
> >>> > >> > > >> > > >> >> > > > > > > > with
> >>> > >> > > >> > > >> >> > > > > > > > > DT community addition under lib/
> ?
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of
> >>> > >> > > corresponding
> >>> > >> > > >> of
> >>> > >> > > >> > > 3rd
> >>> > >> > > >> > > >> >> party
> >>> > >> > > >> > > >> >> > > jars
> >>> > >> > > >> > > >> >> > > > > > which
> >>> > >> > > >> > > >> >> > > > > > > > > needs
> >>> > >> > > >> > > >> >> > > > > > > > > > to included while packaging.
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > Questions related to this:
> >>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli of
> >>> apex as
> >>> > >> > "apex"
> >>> > >> > > >> > instead
> >>> > >> > > >> > > >> of
> >>> > >> > > >> > > >> >> > > "dtcli"
> >>> > >> > > >> > > >> >> > > > in
> >>> > >> > > >> > > >> >> > > > > > > > bigtop
> >>> > >> > > >> > > >> >> > > > > > > > > > package?
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > I think we should keep the name
> >>> until
> >>> > we
> >>> > >> are
> >>> > >> > > >> able
> >>> > >> > > >> > to
> >>> > >> > > >> > > >> >> change
> >>> > >> > > >> > > >> >> > it
> >>> > >> > > >> > > >> >> > > in
> >>> > >> > > >> > > >> >> > > > > > Apex.
> >>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get confusing.
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares in
> >>> bigtop
> >>> > >> have
> >>> > >> > > man
> >>> > >> > > >> > page
> >>> > >> > > >> > > >> for
> >>> > >> > > >> > > >> >> > their
> >>> > >> > > >> > > >> >> > > > > > > > > executables.
> >>> > >> > > >> > > >> >> > > > > > > > > > I think we should have it too
> for
> >>> > >> dtcli.
> >>> > >> > Is
> >>> > >> > > >> there
> >>> > >> > > >> > > any
> >>> > >> > > >> > > >> >> > > > > documentation
> >>> > >> > > >> > > >> >> > > > > > > > > which I
> >>> > >> > > >> > > >> >> > > > > > > > > > can convert to man page? or can
> >>> I use
> >>> > >> > output
> >>> > >> > > >> of
> >>> > >> > > >> > > >> "dtcli
> >>> > >> > > >> > > >> >> > > --help"?
> >>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call version
> of
> >>> apex
> >>> > >> in
> >>> > >> > > >> Bigtop
> >>> > >> > > >> > as
> >>> > >> > > >> > > >> >> 3.3.0 OR
> >>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of
> >>> > bigtop
> >>> > >> to
> >>> > >> > > >> depend
> >>> > >> > > >> > on
> >>> > >> > > >> > > >> >> 2.7.1
> >>> > >> > > >> > > >> >> > > > > version
> >>> > >> > > >> > > >> >> > > > > > of
> >>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems
> that
> >>> we
> >>> > see
> >>> > >> > with
> >>> > >> > > >> this
> >>> > >> > > >> > > >> >> > dependency?
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > Should work without changed. I
> >>> thought
> >>> > we
> >>> > >> > > >> certified
> >>> > >> > > >> > > >> >> against
> >>> > >> > > >> > > >> >> > > > 2.7.0?
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache
> mirror
> >>> > from
> >>> > >> > which
> >>> > >> > > >> > bigtop
> >>> > >> > > >> > > >> will
> >>> > >> > > >> > > >> >> > pick
> >>> > >> > > >> > > >> >> > > > the
> >>> > >> > > >> > > >> >> > > > > > > apex
> >>> > >> > > >> > > >> >> > > > > > > > > > source for compilation and
> >>> packaging.
> >>> > >> > Please
> >>> > >> > > >> > > correct
> >>> > >> > > >> > > >> if
> >>> > >> > > >> > > >> >> > > > > incorrect:
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > >
> >>> > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >>> > >> > > >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is
> >>> the
> >>> > >> mirror
> >>> > >> > > >> used
> >>> > >> > > >> > by
> >>> > >> > > >> > > >> all
> >>> > >> > > >> > > >> >> the
> >>> > >> > > >> > > >> >> > > > > > softwares
> >>> > >> > > >> > > >> >> > > > > > > in
> >>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > Please share your opinion.
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> >>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19
> AM,
> >>> > >> Chinmay
> >>> > >> > > >> > Kolhatkar
> >>> > >> > > >> > > <
> >>> > >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> >>> > >> > > >> > > >> >> > > > > > > > > > wrote:
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the
> >>> > >> packaging.
> >>> > >> > > >> > > >> >> > > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > > I've already started
> >>> discussion on
> >>> > >> > bigtop
> >>> > >> > > >> dev
> >>> > >> > > >> > > >> mailing
> >>> > >> > > >> > > >> >> > list
> >>> > >> > > >> > > >> >> > > > for
> >>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also created a
> >>> Jira
> >>> > for
> >>> > >> the
> >>> > >> > > >> same.
> >>> > >> > > >> > > For
> >>> > >> > > >> > > >> >> this
> >>> > >> > > >> > > >> >> > > > > > communities
> >>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is the bigtop
> >>> Jira:
> >>> > >> > > >> > BIGTOP-2313.
> >>> > >> > > >> > > >> >> > > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13
> AM,
> >>> > >> Thomas
> >>> > >> > > >> Weise <
> >>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> >>> > >> > > >> > > >> >> > > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything under
> >>> > >> > prerequisites
> >>> > >> > > >> that
> >>> > >> > > >> > > >> would
> >>> > >> > > >> > > >> >> be a
> >>> > >> > > >> > > >> >> > > > > > problem.
> >>> > >> > > >> > > >> >> > > > > > > We
> >>> > >> > > >> > > >> >> > > > > > > > > > >> looked
> >>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
> >>> compatibility
> >>> > >> as
> >>> > >> > > part
> >>> > >> > > >> of
> >>> > >> > > >> > > >> >> becoming
> >>> > >> > > >> > > >> >> > an
> >>> > >> > > >> > > >> >> > > > > > > incubator
> >>> > >> > > >> > > >> >> > > > > > > > > > >> project.
> >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the
> packaging
> >>> > during
> >>> > >> > the
> >>> > >> > > >> next
> >>> > >> > > >> > > >> weeks.
> >>> > >> > > >> > > >> >> > Since
> >>> > >> > > >> > > >> >> > > > the
> >>> > >> > > >> > > >> >> > > > > > > work
> >>> > >> > > >> > > >> >> > > > > > > > > will
> >>> > >> > > >> > > >> >> > > > > > > > > > >> be
> >>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related
> >>> > discussions
> >>> > >> and
> >>> > >> > > >> JIRAs
> >>> > >> > > >> > > >> should
> >>> > >> > > >> > > >> >> > also
> >>> > >> > > >> > > >> >> > > be
> >>> > >> > > >> > > >> >> > > > > > > there.
> >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have the
> >>> > packaging
> >>> > >> in
> >>> > >> > > >> place
> >>> > >> > > >> > by
> >>> > >> > > >> > > >> end
> >>> > >> > > >> > > >> >> Feb.
> >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> >>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14
> >>> AM,
> >>> > >> > Chinmay
> >>> > >> > > >> > > >> Kolhatkar <
> >>> > >> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> >>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work on
> >>> adding
> >>> > >> > Apache
> >>> > >> > > >> Apex
> >>> > >> > > >> > > as a
> >>> > >> > > >> > > >> >> > > component
> >>> > >> > > >> > > >> >> > > > > to
> >>> > >> > > >> > > >> >> > > > > > > > Apache
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging
> >>> system
> >>> > for
> >>> > >> > the
> >>> > >> > > >> > Apache
> >>> > >> > > >> > > >> big
> >>> > >> > > >> > > >> >> data
> >>> > >> > > >> > > >> >> > > > > > > ecosystem.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> Several
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it,
> most
> >>> > >> recently
> >>> > >> > > EMR.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira
> >>> task
> >>> > in
> >>> > >> > > >> APEXCORE
> >>> > >> > > >> > for
> >>> > >> > > >> > > >> the
> >>> > >> > > >> > > >> >> > same:
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of execution
> >>> is as
> >>> > >> > > follows:
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
> >>> prerequisites*
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some
> hard
> >>> and
> >>> > >> soft
> >>> > >> > > >> > > expectation
> >>> > >> > > >> > > >> >> for
> >>> > >> > > >> > > >> >> > new
> >>> > >> > > >> > > >> >> > > > > > > > components
> >>> > >> > > >> > > >> >> > > > > > > > > to
> >>> > >> > > >> > > >> >> > > > > > > > > > >> get
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of it:
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > >
> >>> > >> > > >> > > >> >> > > >
> >>> > >> > > >> > > >> >> > >
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >>
> >>> > >> > > >> > > >>
> >>> > >> > > >> > >
> >>> > >> > > >> >
> >>> > >> > > >>
> >>> > >> > >
> >>> > >> >
> >>> > >>
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to be
> >>> > standard
> >>> > >> ASF
> >>> > >> > > >> based
> >>> > >> > > >> > > >> >> > > requirements,
> >>> > >> > > >> > > >> >> > > > > but
> >>> > >> > > >> > > >> >> > > > > > > few
> >>> > >> > > >> > > >> >> > > > > > > > > need
> >>> > >> > > >> > > >> >> > > > > > > > > > >> to
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects are
> >>> > expected
> >>> > >> to
> >>> > >> > be
> >>> > >> > > >> > > Licensed
> >>> > >> > > >> > > >> >> under
> >>> > >> > > >> > > >> >> > > > > Apache
> >>> > >> > > >> > > >> >> > > > > > > > > License,
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
> >>> > >> dependencies
> >>> > >> > are
> >>> > >> > > >> > > expected
> >>> > >> > > >> > > >> >> to be
> >>> > >> > > >> > > >> >> > > > > > > compatible
> >>> > >> > > >> > > >> >> > > > > > > > > with
> >>> > >> > > >> > > >> >> > > > > > > > > > >> this
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL
> 2.0
> >>> but
> >>> > >> need
> >>> > >> > to
> >>> > >> > > >> > check
> >>> > >> > > >> > > if
> >>> > >> > > >> > > >> >> > > > > dependencies
> >>> > >> > > >> > > >> >> > > > > > of
> >>> > >> > > >> > > >> >> > > > > > > > > Apex
> >>> > >> > > >> > > >> >> > > > > > > > > > >> are
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0.
> >>> This I
> >>> > >> guess
> >>> > >> > > >> would
> >>> > >> > > >> > > be a
> >>> > >> > > >> > > >> >> > > > > verification
> >>> > >> > > >> > > >> >> > > > > > > > check.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects are
> >>> > expected
> >>> > >> to
> >>> > >> > be
> >>> > >> > > >> > > >> compatible
> >>> > >> > > >> > > >> >> > with
> >>> > >> > > >> > > >> >> > > > all
> >>> > >> > > >> > > >> >> > > > > of
> >>> > >> > > >> > > >> >> > > > > > > the
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms that
> >>> Bigtop
> >>> > >> > > >> distribution
> >>> > >> > > >> > > is
> >>> > >> > > >> > > >> >> > > targeting
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs verifying
> >>> > whether
> >>> > >> > our
> >>> > >> > > >> > > software
> >>> > >> > > >> > > >> >> runs
> >>> > >> > > >> > > >> >> > > fine
> >>> > >> > > >> > > >> >> > > > in
> >>> > >> > > >> > > >> >> > > > > > > > > centos-6
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
> >>> ubuntu-14.04
> >>> > >> > > debian-8
> >>> > >> > > >> > > >> >> > opensuse-13.2.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that
> >>> should
> >>> > be
> >>> > >> > > added
> >>> > >> > > >> for
> >>> > >> > > >> > > >> >> > deployment.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test
> >>> > artifacts
> >>> > >> > which
> >>> > >> > > >> goes
> >>> > >> > > >> > > >> beyond
> >>> > >> > > >> > > >> >> > > smoke
> >>> > >> > > >> > > >> >> > > > > test
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are basically
> >>> the
> >>> > >> > > integration
> >>> > >> > > >> > > tests
> >>> > >> > > >> > > >> for
> >>> > >> > > >> > > >> >> > > > > > verification
> >>> > >> > > >> > > >> >> > > > > > > > > after
> >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a soft
> >>> > >> > requirement,
> >>> > >> > > >> but
> >>> > >> > > >> > > aim
> >>> > >> > > >> > > >> is
> >>> > >> > > >> > > >> >> to
> >>> > >> > > >> > > >> >> > > > > achieve
> >>> > >> > > >> > > >> >> > > > > > > this
> >>> > >> > > >> > > >> >> > > > > > > > > as
> >>> > >> > > >> > > >> >> > > > > > > > > > >> well
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have
> >>> explanation why
> >>> > >> not
> >>> > >> > to
> >>> > >> > > >> > > include.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any from the
> >>> link
> >>> > >> which
> >>> > >> > > >> > > explicitly
> >>> > >> > > >> > > >> >> needs
> >>> > >> > > >> > > >> >> > to
> >>> > >> > > >> > > >> >> > > > be
> >>> > >> > > >> > > >> >> > > > > > > > checked
> >>> > >> > > >> > > >> >> > > > > > > > > > >> other
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please let
> us
> >>> > know.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as
> >>> > component
> >>> > >> to
> >>> > >> > > >> Bigtop*
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail
> >>> archive of
> >>> > >> > Bigtop,
> >>> > >> > > >> it
> >>> > >> > > >> > was
> >>> > >> > > >> > > >> >> learnt
> >>> > >> > > >> > > >> >> > > that
> >>> > >> > > >> > > >> >> > > > > the
> >>> > >> > > >> > > >> >> > > > > > > > > bigtop
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > community want to see the
> >>> > >> addition of
> >>> > >> > > new
> >>> > >> > > >> > > >> >> components
> >>> > >> > > >> > > >> >> > in
> >>> > >> > > >> > > >> >> > > > > > phases.
> >>> > >> > > >> > > >> >> > > > > > > > Here
> >>> > >> > > >> > > >> >> > > > > > > > > > are
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs creating
> of
> >>> > >> package
> >>> > >> > > i.e.
> >>> > >> > > >> > rpm
> >>> > >> > > >> > > &
> >>> > >> > > >> > > >> deb
> >>> > >> > > >> > > >> >> > > files.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
> >>> documentations/READMEs,
> >>> > >> > LICENSE,
> >>> > >> > > >> > > >> DISCLAMER,
> >>> > >> > > >> > > >> >> > NOTES
> >>> > >> > > >> > > >> >> > > > etc
> >>> > >> > > >> > > >> >> > > > > if
> >>> > >> > > >> > > >> >> > > > > > > any
> >>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any documentation
> that
> >>> > need
> >>> > >> to
> >>> > >> > be
> >>> > >> > > >> > added
> >>> > >> > > >> > > to
> >>> > >> > > >> > > >> >> > > > > distribution
> >>> > >> > > >> > > >> >> > > > > > of
> >>> > >> > > >> > > >> >> > > > > > > > our
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license
> >>> information of
> >>> > >> > > >> > dependencies
> >>> > >> > > >> > > >> >> required
> >>> > >> > > >> > > >> >> > > to
> >>> > >> > > >> > > >> >> > > > be
> >>> > >> > > >> > > >> >> > > > > > > added
> >>> > >> > > >> > > >> >> > > > > > > > > to
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > package
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very
> >>> least)
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke test
> for
> >>> > >> > packaging.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for
> >>> automatic
> >>> > >> > > >> deployment
> >>> > >> > > >> > and
> >>> > >> > > >> > > >> >> > > > configuration
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes
> for
> >>> > >> > automatic
> >>> > >> > > >> > > >> deployment
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For verification of
> >>> > >> > deployments.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to
> make
> >>> > sure
> >>> > >> all
> >>> > >> > > new
> >>> > >> > > >> > > stuff
> >>> > >> > > >> > > >> is
> >>> > >> > > >> > > >> >> > > > compliant
> >>> > >> > > >> > > >> >> > > > > > > with
> >>> > >> > > >> > > >> >> > > > > > > > > ASF
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > license requirements. If
> >>> you add
> >>> > >> code
> >>> > >> > > >> > licenses
> >>> > >> > > >> > > >> >> under
> >>> > >> > > >> > > >> >> > > > > different
> >>> > >> > > >> > > >> >> > > > > > > > > > licenses,
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need to be
> >>> listed in
> >>> > >> the
> >>> > >> > > >> NOTICE.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your thoughts
> >>> on
> >>> > the
> >>> > >> > > >> approach.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start corresponding
> >>> > >> > communication
> >>> > >> > > >> on
> >>> > >> > > >> > > >> bigtop
> >>> > >> > > >> > > >> >> > > mailing
> >>> > >> > > >> > > >> >> > > > > list
> >>> > >> > > >> > > >> >> > > > > > > as
> >>> > >> > > >> > > >> >> > > > > > > > > > well.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some specific
> >>> > >> > > >> questions/suggestions
> >>> > >> > > >> > > >> >> related to
> >>> > >> > > >> > > >> >> > > > what
> >>> > >> > > >> > > >> >> > > > > > > should
> >>> > >> > > >> > > >> >> > > > > > > > > be
> >>> > >> > > >> > > >> >> > > > > > > > > > >> the
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the package and
> >>> what
> >>> > >> > should
> >>> > >> > > be
> >>> > >> > > >> > the
> >>> > >> > > >> > > >> smoke
> >>> > >> > > >> > > >> >> > > tests,
> >>> > >> > > >> > > >> >> > > > > but
> >>> > >> > > >> > > >> >> > > > > > > in
> >>> > >> > > >> > > >> >> > > > > > > > > the
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not having too
> >>> much
> >>> > >> > content
> >>> > >> > > >> > here,
> >>> > >> > > >> > > >> we'll
> >>> > >> > > >> > > >> >> > put
> >>> > >> > > >> > > >> >> > > > the
> >>> > >> > > >> > > >> >> > > > > > > > > questions
> >>> > >> > > >> > > >> >> > > > > > > > > > >> as a
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this
> >>> > mailthread.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> >>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> >>> > >> > > >> > > >> >> > > > > > > > > > >> >
> >>> > >> > > >> > > >> >> > > > > > > > > > >>
> >>> > >> > > >> > > >> >> > > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > > >
> >>> > >> > > >> > > >> >> > > > > > >
> >>> > >> > > >> > > >> >> > > > > >
> >>> > >> > > >> > > >> >> > > > >
> >>> > >> > > >> > > >> >> > > >
> >>> > >> > > >> > > >> >> > >
> >>> > >> > > >> > > >> >> >
> >>> > >> > > >> > > >> >>
> >>> > >> > > >> > > >> >
> >>> > >> > > >> > > >> >
> >>> > >> > > >> > > >>
> >>> > >> > > >> > > >
> >>> > >> > > >> > > >
> >>> > >> > > >> > >
> >>> > >> > > >> >
> >>> > >> > > >>
> >>> > >> > > >
> >>> > >> > > >
> >>> > >> > >
> >>> > >> >
> >>> > >>
> >>> > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>

Re: Apache Apex as component in Apache bigtop

[Tushar Gosavi <tu...@datatorrent.com>]

Hi Chinmay,

I am not able to find the attachment, may be attachments are not allowed in
the mailing list. Can you attach the man page to the Jira?

- Tushar.


On Wed, Feb 17, 2016 at 2:52 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Hi All,
>
> I've attached man page file for the purpose of bigtop integration.
> Please have a look and let me know if there any any feedback about it.
>
> You can see the man page content of this file with:
> man -l apex.1
>
> Thanks,
> Chinmay.
>
>
>
> On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
>> Thanks. Will follow bigtop recommendation. i.e. remove "-incubating".
>>
>> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <th...@datatorrent.com>
>> wrote:
>>
>>> Please follow the recommendation and continue discussion on the Bigtop
>>> list.
>>>
>>> --
>>> sent from mobile
>>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <ch...@apache.org> wrote:
>>>
>>> > I got a recommendation from bigtop community to drop "-incubating" from
>>> > version.
>>> >
>>> > Here is the mail thread:
>>> >
>>> >
>>> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
>>> >
>>> > Please let me know if removing "-incubating" is fine.
>>> >
>>> > Thanks,
>>> > Chinmay.
>>> >
>>> >
>>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <chinmay@apache.org
>>> >
>>> > wrote:
>>> >
>>> > > I've already started communication there.
>>> > >
>>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <
>>> thomas@datatorrent.com>
>>> > > wrote:
>>> > >
>>> > >> This is a question you should ask on the Bigtop mailing list.
>>> > >>
>>> > >> --
>>> > >> sent from mobile
>>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org>
>>> wrote:
>>> > >>
>>> > >> > Yes. In case of RPM, the only restriction on version is that it
>>> can
>>> > not
>>> > >> > contain '-'.
>>> > >> > However, as a workaround, one can use 'incubating' as a part of
>>> > >> %{release}.
>>> > >> >
>>> > >> > Version: 3.3.0
>>> > >> > Release: incubating
>>> > >> >
>>> > >> > apex-3.3.0-incubating.noarch.rpm
>>> > >> >
>>> > >> > It is the same practice followed for pre-release packages e.g.
>>> with
>>> > rcX,
>>> > >> > alphaX, betaX being used in %{release}.
>>> > >> >
>>> > >> > Regards,
>>> > >> > Pradeep A. Dalvi
>>> > >> >
>>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
>>> > chinmay@apache.org>
>>> > >> > wrote:
>>> > >> >
>>> > >> > > Hi Everyone,
>>> > >> > >
>>> > >> > > In one of the previous mails in this thread, we decided that
>>> version
>>> > >> of
>>> > >> > the
>>> > >> > > package will be "3.3.0-incubating".
>>> > >> > > The problem that I'm facing is in RPM base version "-" is not
>>> > allowed.
>>> > >> > >
>>> > >> > > Can we have package version to "3.3.0"?
>>> > >> > > i.e. the packages created will be as follows:
>>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
>>> > >> > > For DEB: apex_3.3.0-1_all.deb
>>> > >> > >
>>> > >> > > There is another integration for Zepplin in Bigtop which is
>>> > incubating
>>> > >> > and
>>> > >> > > is having base version as "0.5.6".
>>> > >> > >
>>> > >> > > Please let me know your thoughts.
>>> > >> > >
>>> > >> > > Thanks,
>>> > >> > > Chinmay.
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
>>> > >> chinmay@apache.org>
>>> > >> > > wrote:
>>> > >> > >
>>> > >> > > > Thanks everyone for the feedback.
>>> > >> > > >
>>> > >> > > > Also as Ram and David mentioned, I'll start creating the man
>>> page
>>> > as
>>> > >> > > well.
>>> > >> > > >
>>> > >> > > > Thanks,
>>> > >> > > > Chinmay.
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
>>> > >> thomas@datatorrent.com>
>>> > >> > > > wrote:
>>> > >> > > >
>>> > >> > > >> I think we already agreed that we will retain the current
>>> script
>>> > >> name
>>> > >> > in
>>> > >> > > >> this bigtop iteration. There is a ticket for renaming, add
>>> you
>>> > >> > > suggestions
>>> > >> > > >> there.
>>> > >> > > >>
>>> > >> > > >> --
>>> > >> > > >> sent from mobile
>>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
>>> > sandesh@datatorrent.com>
>>> > >> > > wrote:
>>> > >> > > >>
>>> > >> > > >> > +1 for Symlink from apex to dtcli or start the transition
>>> > >> process by
>>> > >> > > >> making
>>> > >> > > >> > dtcli as the symlink to apex cli.
>>> > >> > > >> >
>>> > >> > > >> > Renaming is a breaking change should be done during major
>>> > version
>>> > >> > > number
>>> > >> > > >> > change.
>>> > >> > > >> >
>>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
>>> > david@datatorrent.com
>>> > >> >
>>> > >> > > >> wrote:
>>> > >> > > >> >
>>> > >> > > >> > > Apologies for hitting the "Send" button too early.
>>> > >> > > >> > >
>>> > >> > > >> > > What is the scope of the name change?  If we simply have
>>> a
>>> > >> symlink
>>> > >> > > >> from
>>> > >> > > >> > > apex to dtcli, then it should have no backward
>>> compatibility
>>> > >> > problem
>>> > >> > > >> at
>>> > >> > > >> > > all.
>>> > >> > > >> > >
>>> > >> > > >> > > But the proper name change would involve:
>>> > >> > > >> > > - Change the name of DTCli class to ApexCli
>>> > >> > > >> > > - Change the prompt from dt> to apex>
>>> > >> > > >> > > - Change the initial greeting when running the CLI from
>>> > >> something
>>> > >> > > like
>>> > >> > > >> > "DT
>>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
>>> > >> > > >> > >
>>> > >> > > >> > > The above changes may break existing scripts that rely
>>> on the
>>> > >> the
>>> > >> > > >> string
>>> > >> > > >> > > "DT".
>>> > >> > > >> > > But if folks are okay with this potential problem, I am
>>> okay
>>> > >> with
>>> > >> > > the
>>> > >> > > >> > name
>>> > >> > > >> > > change.
>>> > >> > > >> > >
>>> > >> > > >> > > For the man page, we can start with the content of this
>>> page:
>>> > >> > > >> > >
>>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
>>> > >> > > >> > >
>>> > >> > > >> > > Plus the command line options when doing dtcli -h:
>>> > >> > > >> > >
>>> > >> > > >> > > usage: DTCli
>>> > >> > > >> > >  -e <arg>    Commands are read from the argument
>>> > >> > > >> > >  -f <arg>    Use the specified prompt at all time
>>> > >> > > >> > >  -h          Print this help
>>> > >> > > >> > >  -kp <arg>   Use the specified kerberos principal
>>> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
>>> > >> > > >> > >  -p <arg>    JSONP padding function
>>> > >> > > >> > >  -r          JSON Raw mode
>>> > >> > > >> > >  -v          Verbose mode level 1
>>> > >> > > >> > >  -vv         Verbose mode level 2
>>> > >> > > >> > >  -vvv        Verbose mode level 3
>>> > >> > > >> > >  -vvvv       Verbose mode level 4
>>> > >> > > >> > >
>>> > >> > > >> > > David
>>> > >> > > >> > >
>>> > >> > > >> > >
>>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
>>> > >> david@datatorrent.com
>>> > >> > >
>>> > >> > > >> > wrote:
>>> > >> > > >> > >
>>> > >> > > >> > > > What is the scope of the name change?  If we simply
>>> have a
>>> > >> > symlink
>>> > >> > > >> from
>>> > >> > > >> > > > apex to dtcli, then it should have no backward
>>> > compatibility
>>> > >> > > >> problem at
>>> > >> > > >> > > all.
>>> > >> > > >> > > >
>>> > >> > > >> > > > But the proper name change would involve:
>>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
>>> > >> > > >> > > > - Change the prompt from dt> to apex>
>>> > >> > > >> > > > - Change the initial greeting when running the CLI from
>>> > >> > something
>>> > >> > > >> like
>>> > >> > > >> > > "DT
>>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
>>> > >> > > >> > > >
>>> > >> > > >> > > > The above changes may break existing scripts that rely
>>> on
>>> > the
>>> > >> > the
>>> > >> > > >> > string
>>> > >> > > >> > > > "DT".
>>> > >> > > >> > > > But if folks are okay with this potential problem, I am
>>> > okay
>>> > >> > with
>>> > >> > > >> the
>>> > >> > > >> > > name
>>> > >> > > >> > > > change.
>>> > >> > > >> > > >
>>> > >> > > >> > > > For the man page, we can start with the content of this
>>> > page:
>>> > >> > > >> > > >
>>> > >> > > >> > > >
>>> > >> > > >> > > >
>>> > >> > > >> > > >
>>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
>>> > >> > > >> chinmay@apache.org
>>> > >> > > >> > >
>>> > >> > > >> > > > wrote:
>>> > >> > > >> > > >
>>> > >> > > >> > > >> Here is a suggestion I have related to name change:
>>> > >> > > >> > > >>
>>> > >> > > >> > > >> 1. While packaging we change the name of "dtcli"
>>> script to
>>> > >> > > "apex".
>>> > >> > > >> > This
>>> > >> > > >> > > is
>>> > >> > > >> > > >> done so that in future when we change the name to
>>> "apex",
>>> > >> users
>>> > >> > > of
>>> > >> > > >> > > bigtop
>>> > >> > > >> > > >> apex don't have to transition much.
>>> > >> > > >> > > >> 2. We also keep a symlink named dtcli which points to
>>> apex
>>> > >> > > script.
>>> > >> > > >> > This
>>> > >> > > >> > > is
>>> > >> > > >> > > >> for backward compatibility.
>>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first integration
>>> in
>>> > >> > bigtop.
>>> > >> > > >> This
>>> > >> > > >> > is
>>> > >> > > >> > > >> in
>>> > >> > > >> > > >> the interest of not changing the content of dtcli or
>>> apex
>>> > >> > script.
>>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these naming
>>> related
>>> > >> > > changes
>>> > >> > > >> and
>>> > >> > > >> > > >> update bigtop repository later to remove references to
>>> > dtcli
>>> > >> > all
>>> > >> > > >> > > together.
>>> > >> > > >> > > >>
>>> > >> > > >> > > >> Please share your thoughts on above approach.
>>> > >> > > >> > > >>
>>> > >> > > >> > > >> Also, please share what could be the man page content
>>> for
>>> > >> > "apex".
>>> > >> > > >> > > >>
>>> > >> > > >> > > >> Thanks,
>>> > >> > > >> > > >> Chinmay.
>>> > >> > > >> > > >>
>>> > >> > > >> > > >>
>>> > >> > > >> > > >>
>>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
>>> > >> > > >> > chinmay@apache.org>
>>> > >> > > >> > > >> wrote:
>>> > >> > > >> > > >>
>>> > >> > > >> > > >> > Yes.. It'll be build from source tar downloaded
>>> from one
>>> > >> of
>>> > >> > the
>>> > >> > > >> > apache
>>> > >> > > >> > > >> > mirror: apache.osuosl.org
>>> > >> > > >> > > >> >
>>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced from dtcli.
>>> > This
>>> > >> > > leaves
>>> > >> > > >> us
>>> > >> > > >> > 3
>>> > >> > > >> > > >> > options:
>>> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli would
>>> need a
>>> > >> > change
>>> > >> > > >> while
>>> > >> > > >> > > >> > packaging.
>>> > >> > > >> > > >> >
>>> > >> > > >> > > >> > But if we want to use dtcli as it is, then we would
>>> need
>>> > >> > > >> dt-env.sh
>>> > >> > > >> > > >> >
>>> > >> > > >> > > >> >
>>> > >> > > >> > > >> >
>>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
>>> > >> > > >> > thomas@datatorrent.com
>>> > >> > > >> > > >
>>> > >> > > >> > > >> > wrote:
>>> > >> > > >> > > >> >
>>> > >> > > >> > > >> >> Looks good overall, though there shouldn't be any
>>> > >> dt-env.sh
>>> > >> > > >> > > >> >>
>>> > >> > > >> > > >> >> What will the bigtop package be built from, the
>>> source
>>> > >> tar?
>>> > >> > > >> > > >> >>
>>> > >> > > >> > > >> >>
>>> > >> > > >> > > >> >>
>>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar
>>> <
>>> > >> > > >> > > chinmay@apache.org
>>> > >> > > >> > > >> >
>>> > >> > > >> > > >> >> wrote:
>>> > >> > > >> > > >> >>
>>> > >> > > >> > > >> >> > Hi Thomas,
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > Thanks for the feedback.
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > First, we're not changing the name anywhere.
>>> We'll
>>> > >> follow
>>> > >> > > what
>>> > >> > > >> > > >> >> currently is
>>> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating version.
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory structure
>>> below
>>> > >> which
>>> > >> > > is
>>> > >> > > >> > > inline
>>> > >> > > >> > > >> >> with
>>> > >> > > >> > > >> >> > other existing integrations with bigtop.
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > The need for each file is as follows:
>>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
>>> > >> > > >> engine/src/main/scripts/
>>> > >> > > >> > > of
>>> > >> > > >> > > >> >> source
>>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli.
>>> This
>>> > >> > > searches
>>> > >> > > >> for
>>> > >> > > >> > > >> hadoop
>>> > >> > > >> > > >> >> > binary path and exports an env variable for
>>> dtcli to
>>> > >> use
>>> > >> > it.
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is
>>> extracted
>>> > >> in a
>>> > >> > > >> similar
>>> > >> > > >> > > >> way to
>>> > >> > > >> > > >> >> > how DT Community Edition finds dependency jars.
>>> > >> > > >> > > >> >> > Please note that this is the first iteration
>>> list of
>>> > >> jars.
>>> > >> > > I'm
>>> > >> > > >> > > >> trying to
>>> > >> > > >> > > >> >> > narrow this down to only those which are really
>>> > >> required.
>>> > >> > > >> > > >> >> > The test that I'm running to check if dtcli runs
>>> fine
>>> > >> with
>>> > >> > > >> given
>>> > >> > > >> > > >> >> dependency
>>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi demo.
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > Above is the directory structure which is
>>> required
>>> > for
>>> > >> CLI
>>> > >> > > to
>>> > >> > > >> > work.
>>> > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned are some
>>> > >> > additional
>>> > >> > > >> > files
>>> > >> > > >> > > >> which
>>> > >> > > >> > > >> >> > makes the like of administrator easier. For eg.
>>> > >> /etc/skel
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > Please let me know if this is inline with what
>>> you're
>>> > >> > > >> thinking.
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > Thanks,
>>> > >> > > >> > > >> >> > Chinmay.
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > .
>>> > >> > > >> > > >> >> > |-- bin
>>> > >> > > >> > > >> >> > |   `-- dtcli
>>> > >> > > >> > > >> >> > |-- conf
>>> > >> > > >> > > >> >> > |   `-- dt-env.sh
>>> > >> > > >> > > >> >> > `-- lib
>>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
>>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
>>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
>>> > >> > > >> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
>>> > >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
>>> > >> > > >> > > >> >> >     |-- apex-engine.jar
>>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
>>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
>>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
>>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
>>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
>>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
>>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
>>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
>>> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
>>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
>>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
>>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
>>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
>>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
>>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
>>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
>>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
>>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
>>> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
>>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
>>> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
>>> > >> > > >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
>>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
>>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
>>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
>>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
>>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
>>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
>>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
>>> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
>>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
>>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > 3 directories, 40 files
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
>>> > >> > > >> > > >> thomas@datatorrent.com>
>>> > >> > > >> > > >> >> > wrote:
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >> > > Chinmay,
>>> > >> > > >> > > >> >> > >
>>> > >> > > >> > > >> >> > > Before discussing where to put the files, let's
>>> > make
>>> > >> > sure
>>> > >> > > >> they
>>> > >> > > >> > > are
>>> > >> > > >> > > >> >> really
>>> > >> > > >> > > >> >> > > needed for the operation of the CLI. As for
>>> names,
>>> > >> > > anything
>>> > >> > > >> new
>>> > >> > > >> > > >> needs
>>> > >> > > >> > > >> >> to
>>> > >> > > >> > > >> >> > > reflect Apex in the name and should follow
>>> common
>>> > >> > > >> conventions,
>>> > >> > > >> > > >> >> especially
>>> > >> > > >> > > >> >> > > in Bigtop where there are many existing
>>> > integrations
>>> > >> to
>>> > >> > > look
>>> > >> > > >> > at.
>>> > >> > > >> > > >> >> > >
>>> > >> > > >> > > >> >> > > Thanks,
>>> > >> > > >> > > >> >> > > Thomas
>>> > >> > > >> > > >> >> > >
>>> > >> > > >> > > >> >> > >
>>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay
>>> Kolhatkar
>>> > <
>>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
>>> > >> > > >> > > >> >> > > wrote:
>>> > >> > > >> > > >> >> > >
>>> > >> > > >> > > >> >> > > > Really good points Pradeep and Aniruddha.
>>> > >> > > >> > > >> >> > > >
>>> > >> > > >> > > >> >> > > > 1. I believe we won't need to change the
>>> dtcli
>>> > >> > > >> considering it
>>> > >> > > >> > > >> works
>>> > >> > > >> > > >> >> > with
>>> > >> > > >> > > >> >> > > DT
>>> > >> > > >> > > >> >> > > > Community edition. We can keep the directory
>>> > >> structure
>>> > >> > > >> > similar
>>> > >> > > >> > > to
>>> > >> > > >> > > >> >> that.
>>> > >> > > >> > > >> >> > > > dt-env.sh has variables which contains
>>> > information
>>> > >> > > >> required
>>> > >> > > >> > for
>>> > >> > > >> > > >> >> dtcli
>>> > >> > > >> > > >> >> > to
>>> > >> > > >> > > >> >> > > > launch.
>>> > >> > > >> > > >> >> > > >
>>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop community that
>>> > whether
>>> > >> > they
>>> > >> > > >> > > >> facilitate
>>> > >> > > >> > > >> >> the
>>> > >> > > >> > > >> >> > > > installation of rpms/debs before user is
>>> created.
>>> > >> In
>>> > >> > > >> either
>>> > >> > > >> > > case,
>>> > >> > > >> > > >> >> > current
>>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home directory.
>>> > Also
>>> > >> > > before
>>> > >> > > >> > > putting
>>> > >> > > >> > > >> >> > > anything
>>> > >> > > >> > > >> >> > > > in /etc/skel we need to define what are the
>>> > default
>>> > >> > > >> contents
>>> > >> > > >> > > that
>>> > >> > > >> > > >> >> > should
>>> > >> > > >> > > >> >> > > go
>>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults,
>>> > >> probably we
>>> > >> > > >> should
>>> > >> > > >> > > not
>>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
>>> > >> > > >> > > >> >> > > >
>>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This
>>> way
>>> > >> > > contents
>>> > >> > > >> of
>>> > >> > > >> > > >> >> dt-env.sh
>>> > >> > > >> > > >> >> > > are
>>> > >> > > >> > > >> >> > > > present as env variables. But I see a catch
>>> > there.
>>> > >> > > Adding
>>> > >> > > >> > > >> dt-env.sh
>>> > >> > > >> > > >> >> to
>>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the variables
>>> > >> available
>>> > >> > at
>>> > >> > > >> > > runtime
>>> > >> > > >> > > >> all
>>> > >> > > >> > > >> >> > the
>>> > >> > > >> > > >> >> > > > time. I feel a little skeptical about that.
>>> > Maybe a
>>> > >> > > >> possible
>>> > >> > > >> > > >> >> collision
>>> > >> > > >> > > >> >> > > can
>>> > >> > > >> > > >> >> > > > occur with other application vars.
>>> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
>>> > >> > "../conf/dt-env.sh"
>>> > >> > > >> and
>>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are
>>> anyway
>>> > >> > > >> available.
>>> > >> > > >> > > >> >> > > >
>>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is
>>> happening
>>> > at
>>> > >> > > Bigtop
>>> > >> > > >> > > >> >> > > Jira/repository.
>>> > >> > > >> > > >> >> > > > Packaging related code usually goes there.
>>> > (That's
>>> > >> > what
>>> > >> > > >> all
>>> > >> > > >> > the
>>> > >> > > >> > > >> >> > > components
>>> > >> > > >> > > >> >> > > > in bigtop does).
>>> > >> > > >> > > >> >> > > > Having said that, once a PR is created, I'll
>>> be
>>> > >> > sharing
>>> > >> > > >> the
>>> > >> > > >> > > link
>>> > >> > > >> > > >> of
>>> > >> > > >> > > >> >> PR
>>> > >> > > >> > > >> >> > > > here, so that apex community as well can
>>> review
>>> > it.
>>> > >> > > >> > > >> >> > > >
>>> > >> > > >> > > >> >> > > > Thanks,
>>> > >> > > >> > > >> >> > > > Chinmay.
>>> > >> > > >> > > >> >> > > >
>>> > >> > > >> > > >> >> > > >
>>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha
>>> > >> Thombare <
>>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
>>> > >> > > >> > > >> >> > > >
>>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
>>> > >> > > >> > > >> >> > > > > We may need to iron out details about exact
>>> > paths
>>> > >> > etc.
>>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR comments.
>>> > >> > > >> > > >> >> > > > > Is that right @dev?
>>> > >> > > >> > > >> >> > > > >
>>> > >> > > >> > > >> >> > > > >
>>> > >> > > >> > > >> >> > > > >
>>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A.
>>> Dalvi <
>>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
>>> > >> > > >> > > >> >> > > > > wrote:
>>> > >> > > >> > > >> >> > > > >
>>> > >> > > >> > > >> >> > > > > > Inline comments...
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
>>> > >> > Kolhatkar <
>>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
>>> > >> > > >> > > >> >> > > > > > wrote:
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present in DT
>>> > >> community
>>> > >> > > >> > edition
>>> > >> > > >> > > >> will
>>> > >> > > >> > > >> >> be
>>> > >> > > >> > > >> >> > > > > included
>>> > >> > > >> > > >> >> > > > > > > there.
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I found from
>>> > common
>>> > >> > > >> > dependencies
>>> > >> > > >> > > >> >> between
>>> > >> > > >> > > >> >> > > of
>>> > >> > > >> > > >> >> > > > > apex
>>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
>>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
>>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
>>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
>>> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
>>> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
>>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
>>> > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
>>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
>>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
>>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
>>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
>>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
>>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
>>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
>>> > >> > > >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
>>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
>>> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
>>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
>>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
>>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
>>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
>>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
>>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
>>> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some tests do
>>> > check
>>> > >> > that
>>> > >> > > >> > dtcli
>>> > >> > > >> > > >> works
>>> > >> > > >> > > >> >> > > > properly
>>> > >> > > >> > > >> >> > > > > > for
>>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with only
>>> > these
>>> > >> > > >> libraries
>>> > >> > > >> > > >> >> present in
>>> > >> > > >> > > >> >> > > > > > isolation
>>> > >> > > >> > > >> >> > > > > > > without dependency on local m2.
>>> > >> > > >> > > >> >> > > > > > > I'm believe that there are unwanted
>>> jars
>>> > >> which
>>> > >> > are
>>> > >> > > >> used
>>> > >> > > >> > > for
>>> > >> > > >> > > >> >> > compile
>>> > >> > > >> > > >> >> > > > > time
>>> > >> > > >> > > >> >> > > > > > > dependency and not runtime in above
>>> list
>>> > >> which I
>>> > >> > > can
>>> > >> > > >> > drop
>>> > >> > > >> > > >> to
>>> > >> > > >> > > >> >> keep
>>> > >> > > >> > > >> >> > > > > package
>>> > >> > > >> > > >> >> > > > > > > size minimal.
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
>>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of
>>> the
>>> > >> > > >> dt-env.sh is
>>> > >> > > >> > > >> >> required.
>>> > >> > > >> > > >> >> > > > Only
>>> > >> > > >> > > >> >> > > > > > > required values I see in it are
>>> classpath.
>>> > >> and
>>> > >> > > >> dtcli is
>>> > >> > > >> > > >> >> capable
>>> > >> > > >> > > >> >> > of
>>> > >> > > >> > > >> >> > > > > > building
>>> > >> > > >> > > >> >> > > > > > > it on the fly.
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to
>>> see
>>> > how
>>> > >> > dtcli
>>> > >> > > >> can
>>> > >> > > >> > > find
>>> > >> > > >> > > >> >> the
>>> > >> > > >> > > >> >> > jar
>>> > >> > > >> > > >> >> > > > > path,
>>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf files. If
>>> we
>>> > >> need a
>>> > >> > > >> change
>>> > >> > > >> > in
>>> > >> > > >> > > >> >> dtcli,
>>> > >> > > >> > > >> >> > > then
>>> > >> > > >> > > >> >> > > > > > > community's opinion is required for
>>> whether
>>> > >> > dtcli
>>> > >> > > >> > should
>>> > >> > > >> > > >> >> change
>>> > >> > > >> > > >> >> > in
>>> > >> > > >> > > >> >> > > > our
>>> > >> > > >> > > >> >> > > > > > repo
>>> > >> > > >> > > >> >> > > > > > > or a copy of that with required changes
>>> > >> exist in
>>> > >> > > >> bigtop
>>> > >> > > >> > > >> repo
>>> > >> > > >> > > >> >> > until
>>> > >> > > >> > > >> >> > > we
>>> > >> > > >> > > >> >> > > > > > make
>>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > > Can we have these file paths set using
>>> > >> environment
>>> > >> > > >> > > variables
>>> > >> > > >> > > >> >> with
>>> > >> > > >> > > >> >> > > some
>>> > >> > > >> > > >> >> > > > > > default values in dtcli?
>>> > >> > > >> > > >> >> > > > > > Then we can set such params in dt-env.sh.
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file
>>> > different
>>> > >> > from
>>> > >> > > >> > > >> >> dt-site.xml?
>>> > >> > > >> > > >> >> > Not
>>> > >> > > >> > > >> >> > > > > sure
>>> > >> > > >> > > >> >> > > > > > if
>>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help. Files &
>>> > Dirs
>>> > >> > from
>>> > >> > > >> > > /etc/skel
>>> > >> > > >> > > >> >> are
>>> > >> > > >> > > >> >> > > > copied
>>> > >> > > >> > > >> >> > > > > to
>>> > >> > > >> > > >> >> > > > > > > home of new user when useradd program
>>> is
>>> > >> called.
>>> > >> > > But
>>> > >> > > >> > for
>>> > >> > > >> > > >> >> existing
>>> > >> > > >> > > >> >> > > > users
>>> > >> > > >> > > >> >> > > > > > > this won't be of any use. Moreover,
>>> dtcli
>>> > >> > creates
>>> > >> > > >> > ~/.dt/
>>> > >> > > >> > > on
>>> > >> > > >> > > >> >> the
>>> > >> > > >> > > >> >> > fly
>>> > >> > > >> > > >> >> > > > if
>>> > >> > > >> > > >> >> > > > > > not
>>> > >> > > >> > > >> >> > > > > > > encountered for the first time. Again
>>> > >> correct me
>>> > >> > > if
>>> > >> > > >> I'm
>>> > >> > > >> > > >> wrong.
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > > Yes, you are right. However this would
>>> > >> probably be
>>> > >> > > >> > > necessary
>>> > >> > > >> > > >> >> step
>>> > >> > > >> > > >> >> > for
>>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get installed
>>> > during
>>> > >> OS
>>> > >> > > >> install
>>> > >> > > >> > > and
>>> > >> > > >> > > >> >> > before
>>> > >> > > >> > > >> >> > > > user
>>> > >> > > >> > > >> >> > > > > > accounts were created.
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile ->
>>> I'm
>>> > not
>>> > >> > sure
>>> > >> > > >> its
>>> > >> > > >> > the
>>> > >> > > >> > > >> best
>>> > >> > > >> > > >> >> > > idea.
>>> > >> > > >> > > >> >> > > > > > We'll
>>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location which is
>>> by
>>> > >> default
>>> > >> > > >> > present
>>> > >> > > >> > > in
>>> > >> > > >> > > >> >> path
>>> > >> > > >> > > >> >> > > i.e.
>>> > >> > > >> > > >> >> > > > > > > /usr/bin.
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > > For other env variables specific to apex,
>>> > I'll
>>> > >> > > prefer
>>> > >> > > >> to
>>> > >> > > >> > > use
>>> > >> > > >> > > >> >> > > dt-env.sh
>>> > >> > > >> > > >> >> > > > > and
>>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather than changing
>>> > >> bashrc
>>> > >> > > >> etc...
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
>>> /etc/profile.d/
>>> > >> shall
>>> > >> > > be
>>> > >> > > >> the
>>> > >> > > >> > > >> >> cleaner
>>> > >> > > >> > > >> >> > way
>>> > >> > > >> > > >> >> > > > to
>>> > >> > > >> > > >> >> > > > > > achieve the same.
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > > Thanks,
>>> > >> > > >> > > >> >> > > > > > > Chinmay.
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM,
>>> Aniruddha
>>> > >> > > Thombare
>>> > >> > > >> <
>>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > Hi,
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > We need to consider following:
>>> > >> > > >> > > >> >> > > > > > > > System wide default config files can
>>> be
>>> > >> > located
>>> > >> > > at
>>> > >> > > >> > > >> following
>>> > >> > > >> > > >> >> > > > > locations:
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to
>>> > change
>>> > >> > > dtcli
>>> > >> > > >> > > >> behaviour
>>> > >> > > >> > > >> >> on
>>> > >> > > >> > > >> >> > > how
>>> > >> > > >> > > >> >> > > > it
>>> > >> > > >> > > >> >> > > > > > > finds
>>> > >> > > >> > > >> >> > > > > > > > those locations)
>>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to
>>> > >> change
>>> > >> > > dtcli
>>> > >> > > >> > > >> >> behaviour on
>>> > >> > > >> > > >> >> > > how
>>> > >> > > >> > > >> >> > > > > it
>>> > >> > > >> > > >> >> > > > > > > > finds those locations)
>>> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other
>>> > files
>>> > >> > (for
>>> > >> > > >> new
>>> > >> > > >> > > users
>>> > >> > > >> > > >> >> that
>>> > >> > > >> > > >> >> > > will
>>> > >> > > >> > > >> >> > > > > be
>>> > >> > > >> > > >> >> > > > > > > > created in system in future.)
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > We may also have to modify bashrc /
>>> > >> > bashprofile
>>> > >> > > >> for
>>> > >> > > >> > > >> >> population
>>> > >> > > >> > > >> >> > > any
>>> > >> > > >> > > >> >> > > > > > > > variables that are required.
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your comments /
>>> > >> > suggestion.
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > Thanks,
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > Aniruddha
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM,
>>> Thomas
>>> > >> Weise
>>> > >> > <
>>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
>>> > >> > > >> > > >> >> > > > > > > > wrote:
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > -->
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM,
>>> Chinmay
>>> > >> > > >> Kolhatkar <
>>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > wrote:
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging effort
>>> > >> (rpm+deb)
>>> > >> > > of
>>> > >> > > >> > apex,
>>> > >> > > >> > > >> here
>>> > >> > > >> > > >> >> > are
>>> > >> > > >> > > >> >> > > > some
>>> > >> > > >> > > >> >> > > > > > > > > proposals
>>> > >> > > >> > > >> >> > > > > > > > > > about package structure etc..
>>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on bbigtop
>>> mailing
>>> > >> > list, I
>>> > >> > > >> have
>>> > >> > > >> > > >> some
>>> > >> > > >> > > >> >> > > question
>>> > >> > > >> > > >> >> > > > > for
>>> > >> > > >> > > >> >> > > > > > > > apex
>>> > >> > > >> > > >> >> > > > > > > > > > community.
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory structure of
>>> apex
>>> > >> > package
>>> > >> > > >> for
>>> > >> > > >> > > both
>>> > >> > > >> > > >> >> deb &
>>> > >> > > >> > > >> >> > > > rpm:
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
>>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other
>>> dependent
>>> > >> jars>
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
>>> > /usr/lib/apex/bin/dtcli
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> /usr/share/doc/man/man1/dtcli.1.gz
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > /usr/share/doc/apex/license/LICENSE.txt
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
>>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
>>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
>>> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> = All the
>>> 3rd
>>> > >> party
>>> > >> > > >> jars
>>> > >> > > >> > > which
>>> > >> > > >> > > >> >> are
>>> > >> > > >> > > >> >> > > > > required
>>> > >> > > >> > > >> >> > > > > > > for
>>> > >> > > >> > > >> >> > > > > > > > > apex
>>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the dependencies
>>> are
>>> > >> > > packaged
>>> > >> > > >> as
>>> > >> > > >> > > >> part of
>>> > >> > > >> > > >> >> > > > rpm/deb
>>> > >> > > >> > > >> >> > > > > by
>>> > >> > > >> > > >> >> > > > > > > any
>>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > Can you specify what those jars
>>> are in
>>> > >> > Bigtop
>>> > >> > > >> > > context?
>>> > >> > > >> > > >> >> Same
>>> > >> > > >> > > >> >> > as
>>> > >> > > >> > > >> >> > > > > > shipped
>>> > >> > > >> > > >> >> > > > > > > > with
>>> > >> > > >> > > >> >> > > > > > > > > DT community addition under lib/ ?
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of
>>> > >> > > corresponding
>>> > >> > > >> of
>>> > >> > > >> > > 3rd
>>> > >> > > >> > > >> >> party
>>> > >> > > >> > > >> >> > > jars
>>> > >> > > >> > > >> >> > > > > > which
>>> > >> > > >> > > >> >> > > > > > > > > needs
>>> > >> > > >> > > >> >> > > > > > > > > > to included while packaging.
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > Questions related to this:
>>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli of
>>> apex as
>>> > >> > "apex"
>>> > >> > > >> > instead
>>> > >> > > >> > > >> of
>>> > >> > > >> > > >> >> > > "dtcli"
>>> > >> > > >> > > >> >> > > > in
>>> > >> > > >> > > >> >> > > > > > > > bigtop
>>> > >> > > >> > > >> >> > > > > > > > > > package?
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > I think we should keep the name
>>> until
>>> > we
>>> > >> are
>>> > >> > > >> able
>>> > >> > > >> > to
>>> > >> > > >> > > >> >> change
>>> > >> > > >> > > >> >> > it
>>> > >> > > >> > > >> >> > > in
>>> > >> > > >> > > >> >> > > > > > Apex.
>>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get confusing.
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares in
>>> bigtop
>>> > >> have
>>> > >> > > man
>>> > >> > > >> > page
>>> > >> > > >> > > >> for
>>> > >> > > >> > > >> >> > their
>>> > >> > > >> > > >> >> > > > > > > > > executables.
>>> > >> > > >> > > >> >> > > > > > > > > > I think we should have it too for
>>> > >> dtcli.
>>> > >> > Is
>>> > >> > > >> there
>>> > >> > > >> > > any
>>> > >> > > >> > > >> >> > > > > documentation
>>> > >> > > >> > > >> >> > > > > > > > > which I
>>> > >> > > >> > > >> >> > > > > > > > > > can convert to man page? or can
>>> I use
>>> > >> > output
>>> > >> > > >> of
>>> > >> > > >> > > >> "dtcli
>>> > >> > > >> > > >> >> > > --help"?
>>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call version of
>>> apex
>>> > >> in
>>> > >> > > >> Bigtop
>>> > >> > > >> > as
>>> > >> > > >> > > >> >> 3.3.0 OR
>>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of
>>> > bigtop
>>> > >> to
>>> > >> > > >> depend
>>> > >> > > >> > on
>>> > >> > > >> > > >> >> 2.7.1
>>> > >> > > >> > > >> >> > > > > version
>>> > >> > > >> > > >> >> > > > > > of
>>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that
>>> we
>>> > see
>>> > >> > with
>>> > >> > > >> this
>>> > >> > > >> > > >> >> > dependency?
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > Should work without changed. I
>>> thought
>>> > we
>>> > >> > > >> certified
>>> > >> > > >> > > >> >> against
>>> > >> > > >> > > >> >> > > > 2.7.0?
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache mirror
>>> > from
>>> > >> > which
>>> > >> > > >> > bigtop
>>> > >> > > >> > > >> will
>>> > >> > > >> > > >> >> > pick
>>> > >> > > >> > > >> >> > > > the
>>> > >> > > >> > > >> >> > > > > > > apex
>>> > >> > > >> > > >> >> > > > > > > > > > source for compilation and
>>> packaging.
>>> > >> > Please
>>> > >> > > >> > > correct
>>> > >> > > >> > > >> if
>>> > >> > > >> > > >> >> > > > > incorrect:
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > >
>>> > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>>> > >> > > >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is
>>> the
>>> > >> mirror
>>> > >> > > >> used
>>> > >> > > >> > by
>>> > >> > > >> > > >> all
>>> > >> > > >> > > >> >> the
>>> > >> > > >> > > >> >> > > > > > softwares
>>> > >> > > >> > > >> >> > > > > > > in
>>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > Please share your opinion.
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
>>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM,
>>> > >> Chinmay
>>> > >> > > >> > Kolhatkar
>>> > >> > > >> > > <
>>> > >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
>>> > >> > > >> > > >> >> > > > > > > > > > wrote:
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the
>>> > >> packaging.
>>> > >> > > >> > > >> >> > > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > > I've already started
>>> discussion on
>>> > >> > bigtop
>>> > >> > > >> dev
>>> > >> > > >> > > >> mailing
>>> > >> > > >> > > >> >> > list
>>> > >> > > >> > > >> >> > > > for
>>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also created a
>>> Jira
>>> > for
>>> > >> the
>>> > >> > > >> same.
>>> > >> > > >> > > For
>>> > >> > > >> > > >> >> this
>>> > >> > > >> > > >> >> > > > > > communities
>>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is the bigtop
>>> Jira:
>>> > >> > > >> > BIGTOP-2313.
>>> > >> > > >> > > >> >> > > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM,
>>> > >> Thomas
>>> > >> > > >> Weise <
>>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
>>> > >> > > >> > > >> >> > > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
>>> > >> > > >> > > >> >> > > > > > > > > > >>
>>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything under
>>> > >> > prerequisites
>>> > >> > > >> that
>>> > >> > > >> > > >> would
>>> > >> > > >> > > >> >> be a
>>> > >> > > >> > > >> >> > > > > > problem.
>>> > >> > > >> > > >> >> > > > > > > We
>>> > >> > > >> > > >> >> > > > > > > > > > >> looked
>>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
>>> compatibility
>>> > >> as
>>> > >> > > part
>>> > >> > > >> of
>>> > >> > > >> > > >> >> becoming
>>> > >> > > >> > > >> >> > an
>>> > >> > > >> > > >> >> > > > > > > incubator
>>> > >> > > >> > > >> >> > > > > > > > > > >> project.
>>> > >> > > >> > > >> >> > > > > > > > > > >>
>>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the packaging
>>> > during
>>> > >> > the
>>> > >> > > >> next
>>> > >> > > >> > > >> weeks.
>>> > >> > > >> > > >> >> > Since
>>> > >> > > >> > > >> >> > > > the
>>> > >> > > >> > > >> >> > > > > > > work
>>> > >> > > >> > > >> >> > > > > > > > > will
>>> > >> > > >> > > >> >> > > > > > > > > > >> be
>>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related
>>> > discussions
>>> > >> and
>>> > >> > > >> JIRAs
>>> > >> > > >> > > >> should
>>> > >> > > >> > > >> >> > also
>>> > >> > > >> > > >> >> > > be
>>> > >> > > >> > > >> >> > > > > > > there.
>>> > >> > > >> > > >> >> > > > > > > > > > >>
>>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have the
>>> > packaging
>>> > >> in
>>> > >> > > >> place
>>> > >> > > >> > by
>>> > >> > > >> > > >> end
>>> > >> > > >> > > >> >> Feb.
>>> > >> > > >> > > >> >> > > > > > > > > > >>
>>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
>>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
>>> > >> > > >> > > >> >> > > > > > > > > > >>
>>> > >> > > >> > > >> >> > > > > > > > > > >>
>>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14
>>> AM,
>>> > >> > Chinmay
>>> > >> > > >> > > >> Kolhatkar <
>>> > >> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
>>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
>>> > >> > > >> > > >> >> > > > > > > > > > >>
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work on
>>> adding
>>> > >> > Apache
>>> > >> > > >> Apex
>>> > >> > > >> > > as a
>>> > >> > > >> > > >> >> > > component
>>> > >> > > >> > > >> >> > > > > to
>>> > >> > > >> > > >> >> > > > > > > > Apache
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging
>>> system
>>> > for
>>> > >> > the
>>> > >> > > >> > Apache
>>> > >> > > >> > > >> big
>>> > >> > > >> > > >> >> data
>>> > >> > > >> > > >> >> > > > > > > ecosystem.
>>> > >> > > >> > > >> >> > > > > > > > > > >> Several
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most
>>> > >> recently
>>> > >> > > EMR.
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira
>>> task
>>> > in
>>> > >> > > >> APEXCORE
>>> > >> > > >> > for
>>> > >> > > >> > > >> the
>>> > >> > > >> > > >> >> > same:
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of execution
>>> is as
>>> > >> > > follows:
>>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
>>> prerequisites*
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard
>>> and
>>> > >> soft
>>> > >> > > >> > > expectation
>>> > >> > > >> > > >> >> for
>>> > >> > > >> > > >> >> > new
>>> > >> > > >> > > >> >> > > > > > > > components
>>> > >> > > >> > > >> >> > > > > > > > > to
>>> > >> > > >> > > >> >> > > > > > > > > > >> get
>>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of it:
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >>
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > >
>>> > >> > > >> > > >> >> > > >
>>> > >> > > >> > > >> >> > >
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >>
>>> > >> > > >> > > >>
>>> > >> > > >> > >
>>> > >> > > >> >
>>> > >> > > >>
>>> > >> > >
>>> > >> >
>>> > >>
>>> >
>>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to be
>>> > standard
>>> > >> ASF
>>> > >> > > >> based
>>> > >> > > >> > > >> >> > > requirements,
>>> > >> > > >> > > >> >> > > > > but
>>> > >> > > >> > > >> >> > > > > > > few
>>> > >> > > >> > > >> >> > > > > > > > > need
>>> > >> > > >> > > >> >> > > > > > > > > > >> to
>>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
>>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects are
>>> > expected
>>> > >> to
>>> > >> > be
>>> > >> > > >> > > Licensed
>>> > >> > > >> > > >> >> under
>>> > >> > > >> > > >> >> > > > > Apache
>>> > >> > > >> > > >> >> > > > > > > > > License,
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
>>> > >> dependencies
>>> > >> > are
>>> > >> > > >> > > expected
>>> > >> > > >> > > >> >> to be
>>> > >> > > >> > > >> >> > > > > > > compatible
>>> > >> > > >> > > >> >> > > > > > > > > with
>>> > >> > > >> > > >> >> > > > > > > > > > >> this
>>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0
>>> but
>>> > >> need
>>> > >> > to
>>> > >> > > >> > check
>>> > >> > > >> > > if
>>> > >> > > >> > > >> >> > > > > dependencies
>>> > >> > > >> > > >> >> > > > > > of
>>> > >> > > >> > > >> >> > > > > > > > > Apex
>>> > >> > > >> > > >> >> > > > > > > > > > >> are
>>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0.
>>> This I
>>> > >> guess
>>> > >> > > >> would
>>> > >> > > >> > > be a
>>> > >> > > >> > > >> >> > > > > verification
>>> > >> > > >> > > >> >> > > > > > > > check.
>>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects are
>>> > expected
>>> > >> to
>>> > >> > be
>>> > >> > > >> > > >> compatible
>>> > >> > > >> > > >> >> > with
>>> > >> > > >> > > >> >> > > > all
>>> > >> > > >> > > >> >> > > > > of
>>> > >> > > >> > > >> >> > > > > > > the
>>> > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms that
>>> Bigtop
>>> > >> > > >> distribution
>>> > >> > > >> > > is
>>> > >> > > >> > > >> >> > > targeting
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs verifying
>>> > whether
>>> > >> > our
>>> > >> > > >> > > software
>>> > >> > > >> > > >> >> runs
>>> > >> > > >> > > >> >> > > fine
>>> > >> > > >> > > >> >> > > > in
>>> > >> > > >> > > >> >> > > > > > > > > centos-6
>>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
>>> ubuntu-14.04
>>> > >> > > debian-8
>>> > >> > > >> > > >> >> > opensuse-13.2.
>>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that
>>> should
>>> > be
>>> > >> > > added
>>> > >> > > >> for
>>> > >> > > >> > > >> >> > deployment.
>>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test
>>> > artifacts
>>> > >> > which
>>> > >> > > >> goes
>>> > >> > > >> > > >> beyond
>>> > >> > > >> > > >> >> > > smoke
>>> > >> > > >> > > >> >> > > > > test
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are basically
>>> the
>>> > >> > > integration
>>> > >> > > >> > > tests
>>> > >> > > >> > > >> for
>>> > >> > > >> > > >> >> > > > > > verification
>>> > >> > > >> > > >> >> > > > > > > > > after
>>> > >> > > >> > > >> >> > > > > > > > > > >> the
>>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a soft
>>> > >> > requirement,
>>> > >> > > >> but
>>> > >> > > >> > > aim
>>> > >> > > >> > > >> is
>>> > >> > > >> > > >> >> to
>>> > >> > > >> > > >> >> > > > > achieve
>>> > >> > > >> > > >> >> > > > > > > this
>>> > >> > > >> > > >> >> > > > > > > > > as
>>> > >> > > >> > > >> >> > > > > > > > > > >> well
>>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have
>>> explanation why
>>> > >> not
>>> > >> > to
>>> > >> > > >> > > include.
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any from the
>>> link
>>> > >> which
>>> > >> > > >> > > explicitly
>>> > >> > > >> > > >> >> needs
>>> > >> > > >> > > >> >> > to
>>> > >> > > >> > > >> >> > > > be
>>> > >> > > >> > > >> >> > > > > > > > checked
>>> > >> > > >> > > >> >> > > > > > > > > > >> other
>>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please let us
>>> > know.
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as
>>> > component
>>> > >> to
>>> > >> > > >> Bigtop*
>>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail
>>> archive of
>>> > >> > Bigtop,
>>> > >> > > >> it
>>> > >> > > >> > was
>>> > >> > > >> > > >> >> learnt
>>> > >> > > >> > > >> >> > > that
>>> > >> > > >> > > >> >> > > > > the
>>> > >> > > >> > > >> >> > > > > > > > > bigtop
>>> > >> > > >> > > >> >> > > > > > > > > > >> > community want to see the
>>> > >> addition of
>>> > >> > > new
>>> > >> > > >> > > >> >> components
>>> > >> > > >> > > >> >> > in
>>> > >> > > >> > > >> >> > > > > > phases.
>>> > >> > > >> > > >> >> > > > > > > > Here
>>> > >> > > >> > > >> >> > > > > > > > > > are
>>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
>>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs creating of
>>> > >> package
>>> > >> > > i.e.
>>> > >> > > >> > rpm
>>> > >> > > >> > > &
>>> > >> > > >> > > >> deb
>>> > >> > > >> > > >> >> > > files.
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     -
>>> documentations/READMEs,
>>> > >> > LICENSE,
>>> > >> > > >> > > >> DISCLAMER,
>>> > >> > > >> > > >> >> > NOTES
>>> > >> > > >> > > >> >> > > > etc
>>> > >> > > >> > > >> >> > > > > if
>>> > >> > > >> > > >> >> > > > > > > any
>>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any documentation that
>>> > need
>>> > >> to
>>> > >> > be
>>> > >> > > >> > added
>>> > >> > > >> > > to
>>> > >> > > >> > > >> >> > > > > distribution
>>> > >> > > >> > > >> >> > > > > > of
>>> > >> > > >> > > >> >> > > > > > > > our
>>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license
>>> information of
>>> > >> > > >> > dependencies
>>> > >> > > >> > > >> >> required
>>> > >> > > >> > > >> >> > > to
>>> > >> > > >> > > >> >> > > > be
>>> > >> > > >> > > >> >> > > > > > > added
>>> > >> > > >> > > >> >> > > > > > > > > to
>>> > >> > > >> > > >> >> > > > > > > > > > >> > package
>>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very
>>> least)
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke test for
>>> > >> > packaging.
>>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for
>>> automatic
>>> > >> > > >> deployment
>>> > >> > > >> > and
>>> > >> > > >> > > >> >> > > > configuration
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for
>>> > >> > automatic
>>> > >> > > >> > > >> deployment
>>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For verification of
>>> > >> > deployments.
>>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
>>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make
>>> > sure
>>> > >> all
>>> > >> > > new
>>> > >> > > >> > > stuff
>>> > >> > > >> > > >> is
>>> > >> > > >> > > >> >> > > > compliant
>>> > >> > > >> > > >> >> > > > > > > with
>>> > >> > > >> > > >> >> > > > > > > > > ASF
>>> > >> > > >> > > >> >> > > > > > > > > > >> > license requirements. If
>>> you add
>>> > >> code
>>> > >> > > >> > licenses
>>> > >> > > >> > > >> >> under
>>> > >> > > >> > > >> >> > > > > different
>>> > >> > > >> > > >> >> > > > > > > > > > licenses,
>>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need to be
>>> listed in
>>> > >> the
>>> > >> > > >> NOTICE.
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your thoughts
>>> on
>>> > the
>>> > >> > > >> approach.
>>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start corresponding
>>> > >> > communication
>>> > >> > > >> on
>>> > >> > > >> > > >> bigtop
>>> > >> > > >> > > >> >> > > mailing
>>> > >> > > >> > > >> >> > > > > list
>>> > >> > > >> > > >> >> > > > > > > as
>>> > >> > > >> > > >> >> > > > > > > > > > well.
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some specific
>>> > >> > > >> questions/suggestions
>>> > >> > > >> > > >> >> related to
>>> > >> > > >> > > >> >> > > > what
>>> > >> > > >> > > >> >> > > > > > > should
>>> > >> > > >> > > >> >> > > > > > > > > be
>>> > >> > > >> > > >> >> > > > > > > > > > >> the
>>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the package and
>>> what
>>> > >> > should
>>> > >> > > be
>>> > >> > > >> > the
>>> > >> > > >> > > >> smoke
>>> > >> > > >> > > >> >> > > tests,
>>> > >> > > >> > > >> >> > > > > but
>>> > >> > > >> > > >> >> > > > > > > in
>>> > >> > > >> > > >> >> > > > > > > > > the
>>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not having too
>>> much
>>> > >> > content
>>> > >> > > >> > here,
>>> > >> > > >> > > >> we'll
>>> > >> > > >> > > >> >> > put
>>> > >> > > >> > > >> >> > > > the
>>> > >> > > >> > > >> >> > > > > > > > > questions
>>> > >> > > >> > > >> >> > > > > > > > > > >> as a
>>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this
>>> > mailthread.
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
>>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
>>> > >> > > >> > > >> >> > > > > > > > > > >> >
>>> > >> > > >> > > >> >> > > > > > > > > > >>
>>> > >> > > >> > > >> >> > > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > > >
>>> > >> > > >> > > >> >> > > > > > > >
>>> > >> > > >> > > >> >> > > > > > >
>>> > >> > > >> > > >> >> > > > > >
>>> > >> > > >> > > >> >> > > > >
>>> > >> > > >> > > >> >> > > >
>>> > >> > > >> > > >> >> > >
>>> > >> > > >> > > >> >> >
>>> > >> > > >> > > >> >>
>>> > >> > > >> > > >> >
>>> > >> > > >> > > >> >
>>> > >> > > >> > > >>
>>> > >> > > >> > > >
>>> > >> > > >> > > >
>>> > >> > > >> > >
>>> > >> > > >> >
>>> > >> > > >>
>>> > >> > > >
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>
>>
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Hi All,

I've attached man page file for the purpose of bigtop integration.
Please have a look and let me know if there any any feedback about it.

You can see the man page content of this file with:
man -l apex.1

Thanks,
Chinmay.



On Wed, Feb 17, 2016 at 11:04 AM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Thanks. Will follow bigtop recommendation. i.e. remove "-incubating".
>
> On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
>> Please follow the recommendation and continue discussion on the Bigtop
>> list.
>>
>> --
>> sent from mobile
>> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <ch...@apache.org> wrote:
>>
>> > I got a recommendation from bigtop community to drop "-incubating" from
>> > version.
>> >
>> > Here is the mail thread:
>> >
>> >
>> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
>> >
>> > Please let me know if removing "-incubating" is fine.
>> >
>> > Thanks,
>> > Chinmay.
>> >
>> >
>> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <ch...@apache.org>
>> > wrote:
>> >
>> > > I've already started communication there.
>> > >
>> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <thomas@datatorrent.com
>> >
>> > > wrote:
>> > >
>> > >> This is a question you should ask on the Bigtop mailing list.
>> > >>
>> > >> --
>> > >> sent from mobile
>> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org> wrote:
>> > >>
>> > >> > Yes. In case of RPM, the only restriction on version is that it can
>> > not
>> > >> > contain '-'.
>> > >> > However, as a workaround, one can use 'incubating' as a part of
>> > >> %{release}.
>> > >> >
>> > >> > Version: 3.3.0
>> > >> > Release: incubating
>> > >> >
>> > >> > apex-3.3.0-incubating.noarch.rpm
>> > >> >
>> > >> > It is the same practice followed for pre-release packages e.g. with
>> > rcX,
>> > >> > alphaX, betaX being used in %{release}.
>> > >> >
>> > >> > Regards,
>> > >> > Pradeep A. Dalvi
>> > >> >
>> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
>> > chinmay@apache.org>
>> > >> > wrote:
>> > >> >
>> > >> > > Hi Everyone,
>> > >> > >
>> > >> > > In one of the previous mails in this thread, we decided that
>> version
>> > >> of
>> > >> > the
>> > >> > > package will be "3.3.0-incubating".
>> > >> > > The problem that I'm facing is in RPM base version "-" is not
>> > allowed.
>> > >> > >
>> > >> > > Can we have package version to "3.3.0"?
>> > >> > > i.e. the packages created will be as follows:
>> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
>> > >> > > For DEB: apex_3.3.0-1_all.deb
>> > >> > >
>> > >> > > There is another integration for Zepplin in Bigtop which is
>> > incubating
>> > >> > and
>> > >> > > is having base version as "0.5.6".
>> > >> > >
>> > >> > > Please let me know your thoughts.
>> > >> > >
>> > >> > > Thanks,
>> > >> > > Chinmay.
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
>> > >> chinmay@apache.org>
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Thanks everyone for the feedback.
>> > >> > > >
>> > >> > > > Also as Ram and David mentioned, I'll start creating the man
>> page
>> > as
>> > >> > > well.
>> > >> > > >
>> > >> > > > Thanks,
>> > >> > > > Chinmay.
>> > >> > > >
>> > >> > > >
>> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
>> > >> thomas@datatorrent.com>
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > >> I think we already agreed that we will retain the current
>> script
>> > >> name
>> > >> > in
>> > >> > > >> this bigtop iteration. There is a ticket for renaming, add you
>> > >> > > suggestions
>> > >> > > >> there.
>> > >> > > >>
>> > >> > > >> --
>> > >> > > >> sent from mobile
>> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
>> > sandesh@datatorrent.com>
>> > >> > > wrote:
>> > >> > > >>
>> > >> > > >> > +1 for Symlink from apex to dtcli or start the transition
>> > >> process by
>> > >> > > >> making
>> > >> > > >> > dtcli as the symlink to apex cli.
>> > >> > > >> >
>> > >> > > >> > Renaming is a breaking change should be done during major
>> > version
>> > >> > > number
>> > >> > > >> > change.
>> > >> > > >> >
>> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
>> > david@datatorrent.com
>> > >> >
>> > >> > > >> wrote:
>> > >> > > >> >
>> > >> > > >> > > Apologies for hitting the "Send" button too early.
>> > >> > > >> > >
>> > >> > > >> > > What is the scope of the name change?  If we simply have a
>> > >> symlink
>> > >> > > >> from
>> > >> > > >> > > apex to dtcli, then it should have no backward
>> compatibility
>> > >> > problem
>> > >> > > >> at
>> > >> > > >> > > all.
>> > >> > > >> > >
>> > >> > > >> > > But the proper name change would involve:
>> > >> > > >> > > - Change the name of DTCli class to ApexCli
>> > >> > > >> > > - Change the prompt from dt> to apex>
>> > >> > > >> > > - Change the initial greeting when running the CLI from
>> > >> something
>> > >> > > like
>> > >> > > >> > "DT
>> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
>> > >> > > >> > >
>> > >> > > >> > > The above changes may break existing scripts that rely on
>> the
>> > >> the
>> > >> > > >> string
>> > >> > > >> > > "DT".
>> > >> > > >> > > But if folks are okay with this potential problem, I am
>> okay
>> > >> with
>> > >> > > the
>> > >> > > >> > name
>> > >> > > >> > > change.
>> > >> > > >> > >
>> > >> > > >> > > For the man page, we can start with the content of this
>> page:
>> > >> > > >> > >
>> > >> > > >> > > http://docs.datatorrent.com/dtcli/
>> > >> > > >> > >
>> > >> > > >> > > Plus the command line options when doing dtcli -h:
>> > >> > > >> > >
>> > >> > > >> > > usage: DTCli
>> > >> > > >> > >  -e <arg>    Commands are read from the argument
>> > >> > > >> > >  -f <arg>    Use the specified prompt at all time
>> > >> > > >> > >  -h          Print this help
>> > >> > > >> > >  -kp <arg>   Use the specified kerberos principal
>> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
>> > >> > > >> > >  -p <arg>    JSONP padding function
>> > >> > > >> > >  -r          JSON Raw mode
>> > >> > > >> > >  -v          Verbose mode level 1
>> > >> > > >> > >  -vv         Verbose mode level 2
>> > >> > > >> > >  -vvv        Verbose mode level 3
>> > >> > > >> > >  -vvvv       Verbose mode level 4
>> > >> > > >> > >
>> > >> > > >> > > David
>> > >> > > >> > >
>> > >> > > >> > >
>> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
>> > >> david@datatorrent.com
>> > >> > >
>> > >> > > >> > wrote:
>> > >> > > >> > >
>> > >> > > >> > > > What is the scope of the name change?  If we simply
>> have a
>> > >> > symlink
>> > >> > > >> from
>> > >> > > >> > > > apex to dtcli, then it should have no backward
>> > compatibility
>> > >> > > >> problem at
>> > >> > > >> > > all.
>> > >> > > >> > > >
>> > >> > > >> > > > But the proper name change would involve:
>> > >> > > >> > > > - Change the name of DTCli class to ApexCli
>> > >> > > >> > > > - Change the prompt from dt> to apex>
>> > >> > > >> > > > - Change the initial greeting when running the CLI from
>> > >> > something
>> > >> > > >> like
>> > >> > > >> > > "DT
>> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
>> > >> > > >> > > >
>> > >> > > >> > > > The above changes may break existing scripts that rely
>> on
>> > the
>> > >> > the
>> > >> > > >> > string
>> > >> > > >> > > > "DT".
>> > >> > > >> > > > But if folks are okay with this potential problem, I am
>> > okay
>> > >> > with
>> > >> > > >> the
>> > >> > > >> > > name
>> > >> > > >> > > > change.
>> > >> > > >> > > >
>> > >> > > >> > > > For the man page, we can start with the content of this
>> > page:
>> > >> > > >> > > >
>> > >> > > >> > > >
>> > >> > > >> > > >
>> > >> > > >> > > >
>> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
>> > >> > > >> chinmay@apache.org
>> > >> > > >> > >
>> > >> > > >> > > > wrote:
>> > >> > > >> > > >
>> > >> > > >> > > >> Here is a suggestion I have related to name change:
>> > >> > > >> > > >>
>> > >> > > >> > > >> 1. While packaging we change the name of "dtcli"
>> script to
>> > >> > > "apex".
>> > >> > > >> > This
>> > >> > > >> > > is
>> > >> > > >> > > >> done so that in future when we change the name to
>> "apex",
>> > >> users
>> > >> > > of
>> > >> > > >> > > bigtop
>> > >> > > >> > > >> apex don't have to transition much.
>> > >> > > >> > > >> 2. We also keep a symlink named dtcli which points to
>> apex
>> > >> > > script.
>> > >> > > >> > This
>> > >> > > >> > > is
>> > >> > > >> > > >> for backward compatibility.
>> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first integration
>> in
>> > >> > bigtop.
>> > >> > > >> This
>> > >> > > >> > is
>> > >> > > >> > > >> in
>> > >> > > >> > > >> the interest of not changing the content of dtcli or
>> apex
>> > >> > script.
>> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these naming
>> related
>> > >> > > changes
>> > >> > > >> and
>> > >> > > >> > > >> update bigtop repository later to remove references to
>> > dtcli
>> > >> > all
>> > >> > > >> > > together.
>> > >> > > >> > > >>
>> > >> > > >> > > >> Please share your thoughts on above approach.
>> > >> > > >> > > >>
>> > >> > > >> > > >> Also, please share what could be the man page content
>> for
>> > >> > "apex".
>> > >> > > >> > > >>
>> > >> > > >> > > >> Thanks,
>> > >> > > >> > > >> Chinmay.
>> > >> > > >> > > >>
>> > >> > > >> > > >>
>> > >> > > >> > > >>
>> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
>> > >> > > >> > chinmay@apache.org>
>> > >> > > >> > > >> wrote:
>> > >> > > >> > > >>
>> > >> > > >> > > >> > Yes.. It'll be build from source tar downloaded from
>> one
>> > >> of
>> > >> > the
>> > >> > > >> > apache
>> > >> > > >> > > >> > mirror: apache.osuosl.org
>> > >> > > >> > > >> >
>> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced from dtcli.
>> > This
>> > >> > > leaves
>> > >> > > >> us
>> > >> > > >> > 3
>> > >> > > >> > > >> > options:
>> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli would
>> need a
>> > >> > change
>> > >> > > >> while
>> > >> > > >> > > >> > packaging.
>> > >> > > >> > > >> >
>> > >> > > >> > > >> > But if we want to use dtcli as it is, then we would
>> need
>> > >> > > >> dt-env.sh
>> > >> > > >> > > >> >
>> > >> > > >> > > >> >
>> > >> > > >> > > >> >
>> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
>> > >> > > >> > thomas@datatorrent.com
>> > >> > > >> > > >
>> > >> > > >> > > >> > wrote:
>> > >> > > >> > > >> >
>> > >> > > >> > > >> >> Looks good overall, though there shouldn't be any
>> > >> dt-env.sh
>> > >> > > >> > > >> >>
>> > >> > > >> > > >> >> What will the bigtop package be built from, the
>> source
>> > >> tar?
>> > >> > > >> > > >> >>
>> > >> > > >> > > >> >>
>> > >> > > >> > > >> >>
>> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
>> > >> > > >> > > chinmay@apache.org
>> > >> > > >> > > >> >
>> > >> > > >> > > >> >> wrote:
>> > >> > > >> > > >> >>
>> > >> > > >> > > >> >> > Hi Thomas,
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > Thanks for the feedback.
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > First, we're not changing the name anywhere. We'll
>> > >> follow
>> > >> > > what
>> > >> > > >> > > >> >> currently is
>> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating version.
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > Secondly, I've mentioned a directory structure
>> below
>> > >> which
>> > >> > > is
>> > >> > > >> > > inline
>> > >> > > >> > > >> >> with
>> > >> > > >> > > >> >> > other existing integrations with bigtop.
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > The need for each file is as follows:
>> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
>> > >> > > >> engine/src/main/scripts/
>> > >> > > >> > > of
>> > >> > > >> > > >> >> source
>> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli.
>> This
>> > >> > > searches
>> > >> > > >> for
>> > >> > > >> > > >> hadoop
>> > >> > > >> > > >> >> > binary path and exports an env variable for dtcli
>> to
>> > >> use
>> > >> > it.
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is
>> extracted
>> > >> in a
>> > >> > > >> similar
>> > >> > > >> > > >> way to
>> > >> > > >> > > >> >> > how DT Community Edition finds dependency jars.
>> > >> > > >> > > >> >> > Please note that this is the first iteration list
>> of
>> > >> jars.
>> > >> > > I'm
>> > >> > > >> > > >> trying to
>> > >> > > >> > > >> >> > narrow this down to only those which are really
>> > >> required.
>> > >> > > >> > > >> >> > The test that I'm running to check if dtcli runs
>> fine
>> > >> with
>> > >> > > >> given
>> > >> > > >> > > >> >> dependency
>> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi demo.
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > Above is the directory structure which is required
>> > for
>> > >> CLI
>> > >> > > to
>> > >> > > >> > work.
>> > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned are some
>> > >> > additional
>> > >> > > >> > files
>> > >> > > >> > > >> which
>> > >> > > >> > > >> >> > makes the like of administrator easier. For eg.
>> > >> /etc/skel
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > Please let me know if this is inline with what
>> you're
>> > >> > > >> thinking.
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > Thanks,
>> > >> > > >> > > >> >> > Chinmay.
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > .
>> > >> > > >> > > >> >> > |-- bin
>> > >> > > >> > > >> >> > |   `-- dtcli
>> > >> > > >> > > >> >> > |-- conf
>> > >> > > >> > > >> >> > |   `-- dt-env.sh
>> > >> > > >> > > >> >> > `-- lib
>> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
>> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
>> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
>> > >> > > >> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
>> > >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
>> > >> > > >> > > >> >> >     |-- apex-engine.jar
>> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
>> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
>> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
>> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
>> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
>> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
>> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
>> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
>> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
>> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
>> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
>> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
>> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
>> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
>> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
>> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
>> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
>> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
>> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
>> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
>> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
>> > >> > > >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
>> > >> > > >> > > >> >> >     |-- jline-2.11.jar
>> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
>> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
>> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
>> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
>> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
>> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
>> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
>> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
>> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > 3 directories, 40 files
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
>> > >> > > >> > > >> thomas@datatorrent.com>
>> > >> > > >> > > >> >> > wrote:
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >> > > Chinmay,
>> > >> > > >> > > >> >> > >
>> > >> > > >> > > >> >> > > Before discussing where to put the files, let's
>> > make
>> > >> > sure
>> > >> > > >> they
>> > >> > > >> > > are
>> > >> > > >> > > >> >> really
>> > >> > > >> > > >> >> > > needed for the operation of the CLI. As for
>> names,
>> > >> > > anything
>> > >> > > >> new
>> > >> > > >> > > >> needs
>> > >> > > >> > > >> >> to
>> > >> > > >> > > >> >> > > reflect Apex in the name and should follow
>> common
>> > >> > > >> conventions,
>> > >> > > >> > > >> >> especially
>> > >> > > >> > > >> >> > > in Bigtop where there are many existing
>> > integrations
>> > >> to
>> > >> > > look
>> > >> > > >> > at.
>> > >> > > >> > > >> >> > >
>> > >> > > >> > > >> >> > > Thanks,
>> > >> > > >> > > >> >> > > Thomas
>> > >> > > >> > > >> >> > >
>> > >> > > >> > > >> >> > >
>> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay
>> Kolhatkar
>> > <
>> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
>> > >> > > >> > > >> >> > > wrote:
>> > >> > > >> > > >> >> > >
>> > >> > > >> > > >> >> > > > Really good points Pradeep and Aniruddha.
>> > >> > > >> > > >> >> > > >
>> > >> > > >> > > >> >> > > > 1. I believe we won't need to change the dtcli
>> > >> > > >> considering it
>> > >> > > >> > > >> works
>> > >> > > >> > > >> >> > with
>> > >> > > >> > > >> >> > > DT
>> > >> > > >> > > >> >> > > > Community edition. We can keep the directory
>> > >> structure
>> > >> > > >> > similar
>> > >> > > >> > > to
>> > >> > > >> > > >> >> that.
>> > >> > > >> > > >> >> > > > dt-env.sh has variables which contains
>> > information
>> > >> > > >> required
>> > >> > > >> > for
>> > >> > > >> > > >> >> dtcli
>> > >> > > >> > > >> >> > to
>> > >> > > >> > > >> >> > > > launch.
>> > >> > > >> > > >> >> > > >
>> > >> > > >> > > >> >> > > > 2. Let me check with bigtop community that
>> > whether
>> > >> > they
>> > >> > > >> > > >> facilitate
>> > >> > > >> > > >> >> the
>> > >> > > >> > > >> >> > > > installation of rpms/debs before user is
>> created.
>> > >> In
>> > >> > > >> either
>> > >> > > >> > > case,
>> > >> > > >> > > >> >> > current
>> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home directory.
>> > Also
>> > >> > > before
>> > >> > > >> > > putting
>> > >> > > >> > > >> >> > > anything
>> > >> > > >> > > >> >> > > > in /etc/skel we need to define what are the
>> > default
>> > >> > > >> contents
>> > >> > > >> > > that
>> > >> > > >> > > >> >> > should
>> > >> > > >> > > >> >> > > go
>> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults,
>> > >> probably we
>> > >> > > >> should
>> > >> > > >> > > not
>> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
>> > >> > > >> > > >> >> > > >
>> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This
>> way
>> > >> > > contents
>> > >> > > >> of
>> > >> > > >> > > >> >> dt-env.sh
>> > >> > > >> > > >> >> > > are
>> > >> > > >> > > >> >> > > > present as env variables. But I see a catch
>> > there.
>> > >> > > Adding
>> > >> > > >> > > >> dt-env.sh
>> > >> > > >> > > >> >> to
>> > >> > > >> > > >> >> > > > /etc/profile.d would make all the variables
>> > >> available
>> > >> > at
>> > >> > > >> > > runtime
>> > >> > > >> > > >> all
>> > >> > > >> > > >> >> > the
>> > >> > > >> > > >> >> > > > time. I feel a little skeptical about that.
>> > Maybe a
>> > >> > > >> possible
>> > >> > > >> > > >> >> collision
>> > >> > > >> > > >> >> > > can
>> > >> > > >> > > >> >> > > > occur with other application vars.
>> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
>> > >> > "../conf/dt-env.sh"
>> > >> > > >> and
>> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are
>> anyway
>> > >> > > >> available.
>> > >> > > >> > > >> >> > > >
>> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is
>> happening
>> > at
>> > >> > > Bigtop
>> > >> > > >> > > >> >> > > Jira/repository.
>> > >> > > >> > > >> >> > > > Packaging related code usually goes there.
>> > (That's
>> > >> > what
>> > >> > > >> all
>> > >> > > >> > the
>> > >> > > >> > > >> >> > > components
>> > >> > > >> > > >> >> > > > in bigtop does).
>> > >> > > >> > > >> >> > > > Having said that, once a PR is created, I'll
>> be
>> > >> > sharing
>> > >> > > >> the
>> > >> > > >> > > link
>> > >> > > >> > > >> of
>> > >> > > >> > > >> >> PR
>> > >> > > >> > > >> >> > > > here, so that apex community as well can
>> review
>> > it.
>> > >> > > >> > > >> >> > > >
>> > >> > > >> > > >> >> > > > Thanks,
>> > >> > > >> > > >> >> > > > Chinmay.
>> > >> > > >> > > >> >> > > >
>> > >> > > >> > > >> >> > > >
>> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha
>> > >> Thombare <
>> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
>> > >> > > >> > > >> >> > > >
>> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
>> > >> > > >> > > >> >> > > > > We may need to iron out details about exact
>> > paths
>> > >> > etc.
>> > >> > > >> > > >> >> > > > > Which can be done on jira / PR comments.
>> > >> > > >> > > >> >> > > > > Is that right @dev?
>> > >> > > >> > > >> >> > > > >
>> > >> > > >> > > >> >> > > > >
>> > >> > > >> > > >> >> > > > >
>> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A.
>> Dalvi <
>> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
>> > >> > > >> > > >> >> > > > > wrote:
>> > >> > > >> > > >> >> > > > >
>> > >> > > >> > > >> >> > > > > > Inline comments...
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
>> > >> > Kolhatkar <
>> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
>> > >> > > >> > > >> >> > > > > > wrote:
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present in DT
>> > >> community
>> > >> > > >> > edition
>> > >> > > >> > > >> will
>> > >> > > >> > > >> >> be
>> > >> > > >> > > >> >> > > > > included
>> > >> > > >> > > >> >> > > > > > > there.
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > > > Here is the list of jars I found from
>> > common
>> > >> > > >> > dependencies
>> > >> > > >> > > >> >> between
>> > >> > > >> > > >> >> > > of
>> > >> > > >> > > >> >> > > > > apex
>> > >> > > >> > > >> >> > > > > > > and DT Community edition:
>> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
>> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
>> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
>> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
>> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
>> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
>> > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
>> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
>> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
>> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
>> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
>> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
>> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
>> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
>> > >> > > >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
>> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
>> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
>> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
>> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
>> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
>> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
>> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
>> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
>> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some tests do
>> > check
>> > >> > that
>> > >> > > >> > dtcli
>> > >> > > >> > > >> works
>> > >> > > >> > > >> >> > > > properly
>> > >> > > >> > > >> >> > > > > > for
>> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with only
>> > these
>> > >> > > >> libraries
>> > >> > > >> > > >> >> present in
>> > >> > > >> > > >> >> > > > > > isolation
>> > >> > > >> > > >> >> > > > > > > without dependency on local m2.
>> > >> > > >> > > >> >> > > > > > > I'm believe that there are unwanted jars
>> > >> which
>> > >> > are
>> > >> > > >> used
>> > >> > > >> > > for
>> > >> > > >> > > >> >> > compile
>> > >> > > >> > > >> >> > > > > time
>> > >> > > >> > > >> >> > > > > > > dependency and not runtime in above list
>> > >> which I
>> > >> > > can
>> > >> > > >> > drop
>> > >> > > >> > > >> to
>> > >> > > >> > > >> >> keep
>> > >> > > >> > > >> >> > > > > package
>> > >> > > >> > > >> >> > > > > > > size minimal.
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > > > @Anniruddha,
>> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of
>> the
>> > >> > > >> dt-env.sh is
>> > >> > > >> > > >> >> required.
>> > >> > > >> > > >> >> > > > Only
>> > >> > > >> > > >> >> > > > > > > required values I see in it are
>> classpath.
>> > >> and
>> > >> > > >> dtcli is
>> > >> > > >> > > >> >> capable
>> > >> > > >> > > >> >> > of
>> > >> > > >> > > >> >> > > > > > building
>> > >> > > >> > > >> >> > > > > > > it on the fly.
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see
>> > how
>> > >> > dtcli
>> > >> > > >> can
>> > >> > > >> > > find
>> > >> > > >> > > >> >> the
>> > >> > > >> > > >> >> > jar
>> > >> > > >> > > >> >> > > > > path,
>> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf files. If we
>> > >> need a
>> > >> > > >> change
>> > >> > > >> > in
>> > >> > > >> > > >> >> dtcli,
>> > >> > > >> > > >> >> > > then
>> > >> > > >> > > >> >> > > > > > > community's opinion is required for
>> whether
>> > >> > dtcli
>> > >> > > >> > should
>> > >> > > >> > > >> >> change
>> > >> > > >> > > >> >> > in
>> > >> > > >> > > >> >> > > > our
>> > >> > > >> > > >> >> > > > > > repo
>> > >> > > >> > > >> >> > > > > > > or a copy of that with required changes
>> > >> exist in
>> > >> > > >> bigtop
>> > >> > > >> > > >> repo
>> > >> > > >> > > >> >> > until
>> > >> > > >> > > >> >> > > we
>> > >> > > >> > > >> >> > > > > > make
>> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > > Can we have these file paths set using
>> > >> environment
>> > >> > > >> > > variables
>> > >> > > >> > > >> >> with
>> > >> > > >> > > >> >> > > some
>> > >> > > >> > > >> >> > > > > > default values in dtcli?
>> > >> > > >> > > >> >> > > > > > Then we can set such params in dt-env.sh.
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file
>> > different
>> > >> > from
>> > >> > > >> > > >> >> dt-site.xml?
>> > >> > > >> > > >> >> > Not
>> > >> > > >> > > >> >> > > > > sure
>> > >> > > >> > > >> >> > > > > > if
>> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help. Files &
>> > Dirs
>> > >> > from
>> > >> > > >> > > /etc/skel
>> > >> > > >> > > >> >> are
>> > >> > > >> > > >> >> > > > copied
>> > >> > > >> > > >> >> > > > > to
>> > >> > > >> > > >> >> > > > > > > home of new user when useradd program is
>> > >> called.
>> > >> > > But
>> > >> > > >> > for
>> > >> > > >> > > >> >> existing
>> > >> > > >> > > >> >> > > > users
>> > >> > > >> > > >> >> > > > > > > this won't be of any use. Moreover,
>> dtcli
>> > >> > creates
>> > >> > > >> > ~/.dt/
>> > >> > > >> > > on
>> > >> > > >> > > >> >> the
>> > >> > > >> > > >> >> > fly
>> > >> > > >> > > >> >> > > > if
>> > >> > > >> > > >> >> > > > > > not
>> > >> > > >> > > >> >> > > > > > > encountered for the first time. Again
>> > >> correct me
>> > >> > > if
>> > >> > > >> I'm
>> > >> > > >> > > >> wrong.
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > > Yes, you are right. However this would
>> > >> probably be
>> > >> > > >> > > necessary
>> > >> > > >> > > >> >> step
>> > >> > > >> > > >> >> > for
>> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get installed
>> > during
>> > >> OS
>> > >> > > >> install
>> > >> > > >> > > and
>> > >> > > >> > > >> >> > before
>> > >> > > >> > > >> >> > > > user
>> > >> > > >> > > >> >> > > > > > accounts were created.
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm
>> > not
>> > >> > sure
>> > >> > > >> its
>> > >> > > >> > the
>> > >> > > >> > > >> best
>> > >> > > >> > > >> >> > > idea.
>> > >> > > >> > > >> >> > > > > > We'll
>> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location which is by
>> > >> default
>> > >> > > >> > present
>> > >> > > >> > > in
>> > >> > > >> > > >> >> path
>> > >> > > >> > > >> >> > > i.e.
>> > >> > > >> > > >> >> > > > > > > /usr/bin.
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > > For other env variables specific to apex,
>> > I'll
>> > >> > > prefer
>> > >> > > >> to
>> > >> > > >> > > use
>> > >> > > >> > > >> >> > > dt-env.sh
>> > >> > > >> > > >> >> > > > > and
>> > >> > > >> > > >> >> > > > > > > source it in dtcli rather than changing
>> > >> bashrc
>> > >> > > >> etc...
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
>> /etc/profile.d/
>> > >> shall
>> > >> > > be
>> > >> > > >> the
>> > >> > > >> > > >> >> cleaner
>> > >> > > >> > > >> >> > way
>> > >> > > >> > > >> >> > > > to
>> > >> > > >> > > >> >> > > > > > achieve the same.
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > > Thanks,
>> > >> > > >> > > >> >> > > > > > > Chinmay.
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM,
>> Aniruddha
>> > >> > > Thombare
>> > >> > > >> <
>> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > > > > Hi,
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > @Chinmay,
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > We need to consider following:
>> > >> > > >> > > >> >> > > > > > > > System wide default config files can
>> be
>> > >> > located
>> > >> > > at
>> > >> > > >> > > >> following
>> > >> > > >> > > >> >> > > > > locations:
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to
>> > change
>> > >> > > dtcli
>> > >> > > >> > > >> behaviour
>> > >> > > >> > > >> >> on
>> > >> > > >> > > >> >> > > how
>> > >> > > >> > > >> >> > > > it
>> > >> > > >> > > >> >> > > > > > > finds
>> > >> > > >> > > >> >> > > > > > > > those locations)
>> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to
>> > >> change
>> > >> > > dtcli
>> > >> > > >> > > >> >> behaviour on
>> > >> > > >> > > >> >> > > how
>> > >> > > >> > > >> >> > > > > it
>> > >> > > >> > > >> >> > > > > > > > finds those locations)
>> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other
>> > files
>> > >> > (for
>> > >> > > >> new
>> > >> > > >> > > users
>> > >> > > >> > > >> >> that
>> > >> > > >> > > >> >> > > will
>> > >> > > >> > > >> >> > > > > be
>> > >> > > >> > > >> >> > > > > > > > created in system in future.)
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > We may also have to modify bashrc /
>> > >> > bashprofile
>> > >> > > >> for
>> > >> > > >> > > >> >> population
>> > >> > > >> > > >> >> > > any
>> > >> > > >> > > >> >> > > > > > > > variables that are required.
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > @dev, please put in your comments /
>> > >> > suggestion.
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > Thanks,
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > Aniruddha
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM,
>> Thomas
>> > >> Weise
>> > >> > <
>> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
>> > >> > > >> > > >> >> > > > > > > > wrote:
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > -->
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM,
>> Chinmay
>> > >> > > >> Kolhatkar <
>> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > wrote:
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging effort
>> > >> (rpm+deb)
>> > >> > > of
>> > >> > > >> > apex,
>> > >> > > >> > > >> here
>> > >> > > >> > > >> >> > are
>> > >> > > >> > > >> >> > > > some
>> > >> > > >> > > >> >> > > > > > > > > proposals
>> > >> > > >> > > >> >> > > > > > > > > > about package structure etc..
>> > >> > > >> > > >> >> > > > > > > > > > Before posting it on bbigtop
>> mailing
>> > >> > list, I
>> > >> > > >> have
>> > >> > > >> > > >> some
>> > >> > > >> > > >> >> > > question
>> > >> > > >> > > >> >> > > > > for
>> > >> > > >> > > >> >> > > > > > > > apex
>> > >> > > >> > > >> >> > > > > > > > > > community.
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory structure of
>> apex
>> > >> > package
>> > >> > > >> for
>> > >> > > >> > > both
>> > >> > > >> > > >> >> deb &
>> > >> > > >> > > >> >> > > > rpm:
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> /usr/lib/apex/lib/apex-api-<version>.jar
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
>> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent
>> > >> jars>
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
>> > /usr/lib/apex/bin/dtcli
>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
>> > >> > > >> > > >> >> > > > > > > > > >
>> > /usr/share/doc/apex/license/LICENSE.txt
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
>> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
>> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> = All the
>> 3rd
>> > >> party
>> > >> > > >> jars
>> > >> > > >> > > which
>> > >> > > >> > > >> >> are
>> > >> > > >> > > >> >> > > > > required
>> > >> > > >> > > >> >> > > > > > > for
>> > >> > > >> > > >> >> > > > > > > > > apex
>> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the dependencies
>> are
>> > >> > > packaged
>> > >> > > >> as
>> > >> > > >> > > >> part of
>> > >> > > >> > > >> >> > > > rpm/deb
>> > >> > > >> > > >> >> > > > > by
>> > >> > > >> > > >> >> > > > > > > any
>> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > Can you specify what those jars are
>> in
>> > >> > Bigtop
>> > >> > > >> > > context?
>> > >> > > >> > > >> >> Same
>> > >> > > >> > > >> >> > as
>> > >> > > >> > > >> >> > > > > > shipped
>> > >> > > >> > > >> >> > > > > > > > with
>> > >> > > >> > > >> >> > > > > > > > > DT community addition under lib/ ?
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of
>> > >> > > corresponding
>> > >> > > >> of
>> > >> > > >> > > 3rd
>> > >> > > >> > > >> >> party
>> > >> > > >> > > >> >> > > jars
>> > >> > > >> > > >> >> > > > > > which
>> > >> > > >> > > >> >> > > > > > > > > needs
>> > >> > > >> > > >> >> > > > > > > > > > to included while packaging.
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > Questions related to this:
>> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli of apex
>> as
>> > >> > "apex"
>> > >> > > >> > instead
>> > >> > > >> > > >> of
>> > >> > > >> > > >> >> > > "dtcli"
>> > >> > > >> > > >> >> > > > in
>> > >> > > >> > > >> >> > > > > > > > bigtop
>> > >> > > >> > > >> >> > > > > > > > > > package?
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > I think we should keep the name
>> until
>> > we
>> > >> are
>> > >> > > >> able
>> > >> > > >> > to
>> > >> > > >> > > >> >> change
>> > >> > > >> > > >> >> > it
>> > >> > > >> > > >> >> > > in
>> > >> > > >> > > >> >> > > > > > Apex.
>> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get confusing.
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares in
>> bigtop
>> > >> have
>> > >> > > man
>> > >> > > >> > page
>> > >> > > >> > > >> for
>> > >> > > >> > > >> >> > their
>> > >> > > >> > > >> >> > > > > > > > > executables.
>> > >> > > >> > > >> >> > > > > > > > > > I think we should have it too for
>> > >> dtcli.
>> > >> > Is
>> > >> > > >> there
>> > >> > > >> > > any
>> > >> > > >> > > >> >> > > > > documentation
>> > >> > > >> > > >> >> > > > > > > > > which I
>> > >> > > >> > > >> >> > > > > > > > > > can convert to man page? or can I
>> use
>> > >> > output
>> > >> > > >> of
>> > >> > > >> > > >> "dtcli
>> > >> > > >> > > >> >> > > --help"?
>> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call version of
>> apex
>> > >> in
>> > >> > > >> Bigtop
>> > >> > > >> > as
>> > >> > > >> > > >> >> 3.3.0 OR
>> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of
>> > bigtop
>> > >> to
>> > >> > > >> depend
>> > >> > > >> > on
>> > >> > > >> > > >> >> 2.7.1
>> > >> > > >> > > >> >> > > > > version
>> > >> > > >> > > >> >> > > > > > of
>> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that
>> we
>> > see
>> > >> > with
>> > >> > > >> this
>> > >> > > >> > > >> >> > dependency?
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > Should work without changed. I
>> thought
>> > we
>> > >> > > >> certified
>> > >> > > >> > > >> >> against
>> > >> > > >> > > >> >> > > > 2.7.0?
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache mirror
>> > from
>> > >> > which
>> > >> > > >> > bigtop
>> > >> > > >> > > >> will
>> > >> > > >> > > >> >> > pick
>> > >> > > >> > > >> >> > > > the
>> > >> > > >> > > >> >> > > > > > > apex
>> > >> > > >> > > >> >> > > > > > > > > > source for compilation and
>> packaging.
>> > >> > Please
>> > >> > > >> > > correct
>> > >> > > >> > > >> if
>> > >> > > >> > > >> >> > > > > incorrect:
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > >
>> > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>> > >> > > >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is
>> the
>> > >> mirror
>> > >> > > >> used
>> > >> > > >> > by
>> > >> > > >> > > >> all
>> > >> > > >> > > >> >> the
>> > >> > > >> > > >> >> > > > > > softwares
>> > >> > > >> > > >> >> > > > > > > in
>> > >> > > >> > > >> >> > > > > > > > > > bigtop.
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > Please share your opinion.
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > Thanks,
>> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM,
>> > >> Chinmay
>> > >> > > >> > Kolhatkar
>> > >> > > >> > > <
>> > >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
>> > >> > > >> > > >> >> > > > > > > > > > wrote:
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the
>> > >> packaging.
>> > >> > > >> > > >> >> > > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > > I've already started discussion
>> on
>> > >> > bigtop
>> > >> > > >> dev
>> > >> > > >> > > >> mailing
>> > >> > > >> > > >> >> > list
>> > >> > > >> > > >> >> > > > for
>> > >> > > >> > > >> >> > > > > > > > > > > integration. Also created a Jira
>> > for
>> > >> the
>> > >> > > >> same.
>> > >> > > >> > > For
>> > >> > > >> > > >> >> this
>> > >> > > >> > > >> >> > > > > > communities
>> > >> > > >> > > >> >> > > > > > > > > > > reference, here is the bigtop
>> Jira:
>> > >> > > >> > BIGTOP-2313.
>> > >> > > >> > > >> >> > > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM,
>> > >> Thomas
>> > >> > > >> Weise <
>> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > > wrote:
>> > >> > > >> > > >> >> > > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything under
>> > >> > prerequisites
>> > >> > > >> that
>> > >> > > >> > > >> would
>> > >> > > >> > > >> >> be a
>> > >> > > >> > > >> >> > > > > > problem.
>> > >> > > >> > > >> >> > > > > > > We
>> > >> > > >> > > >> >> > > > > > > > > > >> looked
>> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
>> compatibility
>> > >> as
>> > >> > > part
>> > >> > > >> of
>> > >> > > >> > > >> >> becoming
>> > >> > > >> > > >> >> > an
>> > >> > > >> > > >> >> > > > > > > incubator
>> > >> > > >> > > >> >> > > > > > > > > > >> project.
>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the packaging
>> > during
>> > >> > the
>> > >> > > >> next
>> > >> > > >> > > >> weeks.
>> > >> > > >> > > >> >> > Since
>> > >> > > >> > > >> >> > > > the
>> > >> > > >> > > >> >> > > > > > > work
>> > >> > > >> > > >> >> > > > > > > > > will
>> > >> > > >> > > >> >> > > > > > > > > > >> be
>> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related
>> > discussions
>> > >> and
>> > >> > > >> JIRAs
>> > >> > > >> > > >> should
>> > >> > > >> > > >> >> > also
>> > >> > > >> > > >> >> > > be
>> > >> > > >> > > >> >> > > > > > > there.
>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have the
>> > packaging
>> > >> in
>> > >> > > >> place
>> > >> > > >> > by
>> > >> > > >> > > >> end
>> > >> > > >> > > >> >> Feb.
>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
>> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14
>> AM,
>> > >> > Chinmay
>> > >> > > >> > > >> Kolhatkar <
>> > >> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
>> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work on
>> adding
>> > >> > Apache
>> > >> > > >> Apex
>> > >> > > >> > > as a
>> > >> > > >> > > >> >> > > component
>> > >> > > >> > > >> >> > > > > to
>> > >> > > >> > > >> >> > > > > > > > Apache
>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
>> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging
>> system
>> > for
>> > >> > the
>> > >> > > >> > Apache
>> > >> > > >> > > >> big
>> > >> > > >> > > >> >> data
>> > >> > > >> > > >> >> > > > > > > ecosystem.
>> > >> > > >> > > >> >> > > > > > > > > > >> Several
>> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most
>> > >> recently
>> > >> > > EMR.
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira
>> task
>> > in
>> > >> > > >> APEXCORE
>> > >> > > >> > for
>> > >> > > >> > > >> the
>> > >> > > >> > > >> >> > same:
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of execution
>> is as
>> > >> > > follows:
>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle
>> prerequisites*
>> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard
>> and
>> > >> soft
>> > >> > > >> > > expectation
>> > >> > > >> > > >> >> for
>> > >> > > >> > > >> >> > new
>> > >> > > >> > > >> >> > > > > > > > components
>> > >> > > >> > > >> >> > > > > > > > > to
>> > >> > > >> > > >> >> > > > > > > > > > >> get
>> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
>> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of it:
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > >
>> > >> > > >> > > >> >> > > >
>> > >> > > >> > > >> >> > >
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >>
>> > >> > > >> > > >>
>> > >> > > >> > >
>> > >> > > >> >
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to be
>> > standard
>> > >> ASF
>> > >> > > >> based
>> > >> > > >> > > >> >> > > requirements,
>> > >> > > >> > > >> >> > > > > but
>> > >> > > >> > > >> >> > > > > > > few
>> > >> > > >> > > >> >> > > > > > > > > need
>> > >> > > >> > > >> >> > > > > > > > > > >> to
>> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects are
>> > expected
>> > >> to
>> > >> > be
>> > >> > > >> > > Licensed
>> > >> > > >> > > >> >> under
>> > >> > > >> > > >> >> > > > > Apache
>> > >> > > >> > > >> >> > > > > > > > > License,
>> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
>> > >> dependencies
>> > >> > are
>> > >> > > >> > > expected
>> > >> > > >> > > >> >> to be
>> > >> > > >> > > >> >> > > > > > > compatible
>> > >> > > >> > > >> >> > > > > > > > > with
>> > >> > > >> > > >> >> > > > > > > > > > >> this
>> > >> > > >> > > >> >> > > > > > > > > > >> > license)
>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0
>> but
>> > >> need
>> > >> > to
>> > >> > > >> > check
>> > >> > > >> > > if
>> > >> > > >> > > >> >> > > > > dependencies
>> > >> > > >> > > >> >> > > > > > of
>> > >> > > >> > > >> >> > > > > > > > > Apex
>> > >> > > >> > > >> >> > > > > > > > > > >> are
>> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0.
>> This I
>> > >> guess
>> > >> > > >> would
>> > >> > > >> > > be a
>> > >> > > >> > > >> >> > > > > verification
>> > >> > > >> > > >> >> > > > > > > > check.
>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects are
>> > expected
>> > >> to
>> > >> > be
>> > >> > > >> > > >> compatible
>> > >> > > >> > > >> >> > with
>> > >> > > >> > > >> >> > > > all
>> > >> > > >> > > >> >> > > > > of
>> > >> > > >> > > >> >> > > > > > > the
>> > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms that
>> Bigtop
>> > >> > > >> distribution
>> > >> > > >> > > is
>> > >> > > >> > > >> >> > > targeting
>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs verifying
>> > whether
>> > >> > our
>> > >> > > >> > > software
>> > >> > > >> > > >> >> runs
>> > >> > > >> > > >> >> > > fine
>> > >> > > >> > > >> >> > > > in
>> > >> > > >> > > >> >> > > > > > > > > centos-6
>> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
>> ubuntu-14.04
>> > >> > > debian-8
>> > >> > > >> > > >> >> > opensuse-13.2.
>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that
>> should
>> > be
>> > >> > > added
>> > >> > > >> for
>> > >> > > >> > > >> >> > deployment.
>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test
>> > artifacts
>> > >> > which
>> > >> > > >> goes
>> > >> > > >> > > >> beyond
>> > >> > > >> > > >> >> > > smoke
>> > >> > > >> > > >> >> > > > > test
>> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are basically the
>> > >> > > integration
>> > >> > > >> > > tests
>> > >> > > >> > > >> for
>> > >> > > >> > > >> >> > > > > > verification
>> > >> > > >> > > >> >> > > > > > > > > after
>> > >> > > >> > > >> >> > > > > > > > > > >> the
>> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a soft
>> > >> > requirement,
>> > >> > > >> but
>> > >> > > >> > > aim
>> > >> > > >> > > >> is
>> > >> > > >> > > >> >> to
>> > >> > > >> > > >> >> > > > > achieve
>> > >> > > >> > > >> >> > > > > > > this
>> > >> > > >> > > >> >> > > > > > > > > as
>> > >> > > >> > > >> >> > > > > > > > > > >> well
>> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have explanation
>> why
>> > >> not
>> > >> > to
>> > >> > > >> > > include.
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any from the
>> link
>> > >> which
>> > >> > > >> > > explicitly
>> > >> > > >> > > >> >> needs
>> > >> > > >> > > >> >> > to
>> > >> > > >> > > >> >> > > > be
>> > >> > > >> > > >> >> > > > > > > > checked
>> > >> > > >> > > >> >> > > > > > > > > > >> other
>> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please let us
>> > know.
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as
>> > component
>> > >> to
>> > >> > > >> Bigtop*
>> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail archive
>> of
>> > >> > Bigtop,
>> > >> > > >> it
>> > >> > > >> > was
>> > >> > > >> > > >> >> learnt
>> > >> > > >> > > >> >> > > that
>> > >> > > >> > > >> >> > > > > the
>> > >> > > >> > > >> >> > > > > > > > > bigtop
>> > >> > > >> > > >> >> > > > > > > > > > >> > community want to see the
>> > >> addition of
>> > >> > > new
>> > >> > > >> > > >> >> components
>> > >> > > >> > > >> >> > in
>> > >> > > >> > > >> >> > > > > > phases.
>> > >> > > >> > > >> >> > > > > > > > Here
>> > >> > > >> > > >> >> > > > > > > > > > are
>> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
>> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
>> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs creating of
>> > >> package
>> > >> > > i.e.
>> > >> > > >> > rpm
>> > >> > > >> > > &
>> > >> > > >> > > >> deb
>> > >> > > >> > > >> >> > > files.
>> > >> > > >> > > >> >> > > > > > > > > > >> >     - documentations/READMEs,
>> > >> > LICENSE,
>> > >> > > >> > > >> DISCLAMER,
>> > >> > > >> > > >> >> > NOTES
>> > >> > > >> > > >> >> > > > etc
>> > >> > > >> > > >> >> > > > > if
>> > >> > > >> > > >> >> > > > > > > any
>> > >> > > >> > > >> >> > > > > > > > > > >> needed.
>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any documentation that
>> > need
>> > >> to
>> > >> > be
>> > >> > > >> > added
>> > >> > > >> > > to
>> > >> > > >> > > >> >> > > > > distribution
>> > >> > > >> > > >> >> > > > > > of
>> > >> > > >> > > >> >> > > > > > > > our
>> > >> > > >> > > >> >> > > > > > > > > > >> > software.
>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license
>> information of
>> > >> > > >> > dependencies
>> > >> > > >> > > >> >> required
>> > >> > > >> > > >> >> > > to
>> > >> > > >> > > >> >> > > > be
>> > >> > > >> > > >> >> > > > > > > added
>> > >> > > >> > > >> >> > > > > > > > > to
>> > >> > > >> > > >> >> > > > > > > > > > >> > package
>> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very
>> least)
>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke test for
>> > >> > packaging.
>> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for
>> automatic
>> > >> > > >> deployment
>> > >> > > >> > and
>> > >> > > >> > > >> >> > > > configuration
>> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for
>> > >> > automatic
>> > >> > > >> > > >> deployment
>> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
>> > >> > > >> > > >> >> > > > > > > > > > >> >     - For verification of
>> > >> > deployments.
>> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
>> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make
>> > sure
>> > >> all
>> > >> > > new
>> > >> > > >> > > stuff
>> > >> > > >> > > >> is
>> > >> > > >> > > >> >> > > > compliant
>> > >> > > >> > > >> >> > > > > > > with
>> > >> > > >> > > >> >> > > > > > > > > ASF
>> > >> > > >> > > >> >> > > > > > > > > > >> > license requirements. If you
>> add
>> > >> code
>> > >> > > >> > licenses
>> > >> > > >> > > >> >> under
>> > >> > > >> > > >> >> > > > > different
>> > >> > > >> > > >> >> > > > > > > > > > licenses,
>> > >> > > >> > > >> >> > > > > > > > > > >> > those would need to be
>> listed in
>> > >> the
>> > >> > > >> NOTICE.
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your thoughts on
>> > the
>> > >> > > >> approach.
>> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start corresponding
>> > >> > communication
>> > >> > > >> on
>> > >> > > >> > > >> bigtop
>> > >> > > >> > > >> >> > > mailing
>> > >> > > >> > > >> >> > > > > list
>> > >> > > >> > > >> >> > > > > > > as
>> > >> > > >> > > >> >> > > > > > > > > > well.
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >> > We have some specific
>> > >> > > >> questions/suggestions
>> > >> > > >> > > >> >> related to
>> > >> > > >> > > >> >> > > > what
>> > >> > > >> > > >> >> > > > > > > should
>> > >> > > >> > > >> >> > > > > > > > > be
>> > >> > > >> > > >> >> > > > > > > > > > >> the
>> > >> > > >> > > >> >> > > > > > > > > > >> > content of the package and
>> what
>> > >> > should
>> > >> > > be
>> > >> > > >> > the
>> > >> > > >> > > >> smoke
>> > >> > > >> > > >> >> > > tests,
>> > >> > > >> > > >> >> > > > > but
>> > >> > > >> > > >> >> > > > > > > in
>> > >> > > >> > > >> >> > > > > > > > > the
>> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not having too
>> much
>> > >> > content
>> > >> > > >> > here,
>> > >> > > >> > > >> we'll
>> > >> > > >> > > >> >> > put
>> > >> > > >> > > >> >> > > > the
>> > >> > > >> > > >> >> > > > > > > > > questions
>> > >> > > >> > > >> >> > > > > > > > > > >> as a
>> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this
>> > mailthread.
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
>> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
>> > >> > > >> > > >> >> > > > > > > > > > >> >
>> > >> > > >> > > >> >> > > > > > > > > > >>
>> > >> > > >> > > >> >> > > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > > >
>> > >> > > >> > > >> >> > > > > > > >
>> > >> > > >> > > >> >> > > > > > >
>> > >> > > >> > > >> >> > > > > >
>> > >> > > >> > > >> >> > > > >
>> > >> > > >> > > >> >> > > >
>> > >> > > >> > > >> >> > >
>> > >> > > >> > > >> >> >
>> > >> > > >> > > >> >>
>> > >> > > >> > > >> >
>> > >> > > >> > > >> >
>> > >> > > >> > > >>
>> > >> > > >> > > >
>> > >> > > >> > > >
>> > >> > > >> > >
>> > >> > > >> >
>> > >> > > >>
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Thanks. Will follow bigtop recommendation. i.e. remove "-incubating".

On Tue, Feb 16, 2016 at 9:52 PM, Thomas Weise <th...@datatorrent.com>
wrote:

> Please follow the recommendation and continue discussion on the Bigtop
> list.
>
> --
> sent from mobile
> On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <ch...@apache.org> wrote:
>
> > I got a recommendation from bigtop community to drop "-incubating" from
> > version.
> >
> > Here is the mail thread:
> >
> >
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
> >
> > Please let me know if removing "-incubating" is fine.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > I've already started communication there.
> > >
> > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <th...@datatorrent.com>
> > > wrote:
> > >
> > >> This is a question you should ask on the Bigtop mailing list.
> > >>
> > >> --
> > >> sent from mobile
> > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org> wrote:
> > >>
> > >> > Yes. In case of RPM, the only restriction on version is that it can
> > not
> > >> > contain '-'.
> > >> > However, as a workaround, one can use 'incubating' as a part of
> > >> %{release}.
> > >> >
> > >> > Version: 3.3.0
> > >> > Release: incubating
> > >> >
> > >> > apex-3.3.0-incubating.noarch.rpm
> > >> >
> > >> > It is the same practice followed for pre-release packages e.g. with
> > rcX,
> > >> > alphaX, betaX being used in %{release}.
> > >> >
> > >> > Regards,
> > >> > Pradeep A. Dalvi
> > >> >
> > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> > chinmay@apache.org>
> > >> > wrote:
> > >> >
> > >> > > Hi Everyone,
> > >> > >
> > >> > > In one of the previous mails in this thread, we decided that
> version
> > >> of
> > >> > the
> > >> > > package will be "3.3.0-incubating".
> > >> > > The problem that I'm facing is in RPM base version "-" is not
> > allowed.
> > >> > >
> > >> > > Can we have package version to "3.3.0"?
> > >> > > i.e. the packages created will be as follows:
> > >> > > For RPM: apex-3.3.0-1.noarch.rpm
> > >> > > For DEB: apex_3.3.0-1_all.deb
> > >> > >
> > >> > > There is another integration for Zepplin in Bigtop which is
> > incubating
> > >> > and
> > >> > > is having base version as "0.5.6".
> > >> > >
> > >> > > Please let me know your thoughts.
> > >> > >
> > >> > > Thanks,
> > >> > > Chinmay.
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> > >> chinmay@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > > > Thanks everyone for the feedback.
> > >> > > >
> > >> > > > Also as Ram and David mentioned, I'll start creating the man
> page
> > as
> > >> > > well.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Chinmay.
> > >> > > >
> > >> > > >
> > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> > >> thomas@datatorrent.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > >> I think we already agreed that we will retain the current
> script
> > >> name
> > >> > in
> > >> > > >> this bigtop iteration. There is a ticket for renaming, add you
> > >> > > suggestions
> > >> > > >> there.
> > >> > > >>
> > >> > > >> --
> > >> > > >> sent from mobile
> > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> > sandesh@datatorrent.com>
> > >> > > wrote:
> > >> > > >>
> > >> > > >> > +1 for Symlink from apex to dtcli or start the transition
> > >> process by
> > >> > > >> making
> > >> > > >> > dtcli as the symlink to apex cli.
> > >> > > >> >
> > >> > > >> > Renaming is a breaking change should be done during major
> > version
> > >> > > number
> > >> > > >> > change.
> > >> > > >> >
> > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> > david@datatorrent.com
> > >> >
> > >> > > >> wrote:
> > >> > > >> >
> > >> > > >> > > Apologies for hitting the "Send" button too early.
> > >> > > >> > >
> > >> > > >> > > What is the scope of the name change?  If we simply have a
> > >> symlink
> > >> > > >> from
> > >> > > >> > > apex to dtcli, then it should have no backward
> compatibility
> > >> > problem
> > >> > > >> at
> > >> > > >> > > all.
> > >> > > >> > >
> > >> > > >> > > But the proper name change would involve:
> > >> > > >> > > - Change the name of DTCli class to ApexCli
> > >> > > >> > > - Change the prompt from dt> to apex>
> > >> > > >> > > - Change the initial greeting when running the CLI from
> > >> something
> > >> > > like
> > >> > > >> > "DT
> > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > >> > > >> > >
> > >> > > >> > > The above changes may break existing scripts that rely on
> the
> > >> the
> > >> > > >> string
> > >> > > >> > > "DT".
> > >> > > >> > > But if folks are okay with this potential problem, I am
> okay
> > >> with
> > >> > > the
> > >> > > >> > name
> > >> > > >> > > change.
> > >> > > >> > >
> > >> > > >> > > For the man page, we can start with the content of this
> page:
> > >> > > >> > >
> > >> > > >> > > http://docs.datatorrent.com/dtcli/
> > >> > > >> > >
> > >> > > >> > > Plus the command line options when doing dtcli -h:
> > >> > > >> > >
> > >> > > >> > > usage: DTCli
> > >> > > >> > >  -e <arg>    Commands are read from the argument
> > >> > > >> > >  -f <arg>    Use the specified prompt at all time
> > >> > > >> > >  -h          Print this help
> > >> > > >> > >  -kp <arg>   Use the specified kerberos principal
> > >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> > >> > > >> > >  -p <arg>    JSONP padding function
> > >> > > >> > >  -r          JSON Raw mode
> > >> > > >> > >  -v          Verbose mode level 1
> > >> > > >> > >  -vv         Verbose mode level 2
> > >> > > >> > >  -vvv        Verbose mode level 3
> > >> > > >> > >  -vvvv       Verbose mode level 4
> > >> > > >> > >
> > >> > > >> > > David
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> > >> david@datatorrent.com
> > >> > >
> > >> > > >> > wrote:
> > >> > > >> > >
> > >> > > >> > > > What is the scope of the name change?  If we simply have
> a
> > >> > symlink
> > >> > > >> from
> > >> > > >> > > > apex to dtcli, then it should have no backward
> > compatibility
> > >> > > >> problem at
> > >> > > >> > > all.
> > >> > > >> > > >
> > >> > > >> > > > But the proper name change would involve:
> > >> > > >> > > > - Change the name of DTCli class to ApexCli
> > >> > > >> > > > - Change the prompt from dt> to apex>
> > >> > > >> > > > - Change the initial greeting when running the CLI from
> > >> > something
> > >> > > >> like
> > >> > > >> > > "DT
> > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > >> > > >> > > >
> > >> > > >> > > > The above changes may break existing scripts that rely on
> > the
> > >> > the
> > >> > > >> > string
> > >> > > >> > > > "DT".
> > >> > > >> > > > But if folks are okay with this potential problem, I am
> > okay
> > >> > with
> > >> > > >> the
> > >> > > >> > > name
> > >> > > >> > > > change.
> > >> > > >> > > >
> > >> > > >> > > > For the man page, we can start with the content of this
> > page:
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> > >> > > >> chinmay@apache.org
> > >> > > >> > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > >> Here is a suggestion I have related to name change:
> > >> > > >> > > >>
> > >> > > >> > > >> 1. While packaging we change the name of "dtcli" script
> to
> > >> > > "apex".
> > >> > > >> > This
> > >> > > >> > > is
> > >> > > >> > > >> done so that in future when we change the name to
> "apex",
> > >> users
> > >> > > of
> > >> > > >> > > bigtop
> > >> > > >> > > >> apex don't have to transition much.
> > >> > > >> > > >> 2. We also keep a symlink named dtcli which points to
> apex
> > >> > > script.
> > >> > > >> > This
> > >> > > >> > > is
> > >> > > >> > > >> for backward compatibility.
> > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first integration in
> > >> > bigtop.
> > >> > > >> This
> > >> > > >> > is
> > >> > > >> > > >> in
> > >> > > >> > > >> the interest of not changing the content of dtcli or
> apex
> > >> > script.
> > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these naming
> related
> > >> > > changes
> > >> > > >> and
> > >> > > >> > > >> update bigtop repository later to remove references to
> > dtcli
> > >> > all
> > >> > > >> > > together.
> > >> > > >> > > >>
> > >> > > >> > > >> Please share your thoughts on above approach.
> > >> > > >> > > >>
> > >> > > >> > > >> Also, please share what could be the man page content
> for
> > >> > "apex".
> > >> > > >> > > >>
> > >> > > >> > > >> Thanks,
> > >> > > >> > > >> Chinmay.
> > >> > > >> > > >>
> > >> > > >> > > >>
> > >> > > >> > > >>
> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> > >> > > >> > chinmay@apache.org>
> > >> > > >> > > >> wrote:
> > >> > > >> > > >>
> > >> > > >> > > >> > Yes.. It'll be build from source tar downloaded from
> one
> > >> of
> > >> > the
> > >> > > >> > apache
> > >> > > >> > > >> > mirror: apache.osuosl.org
> > >> > > >> > > >> >
> > >> > > >> > > >> > The reason for dt-env.sh is it's sourced from dtcli.
> > This
> > >> > > leaves
> > >> > > >> us
> > >> > > >> > 3
> > >> > > >> > > >> > options:
> > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli would need
> a
> > >> > change
> > >> > > >> while
> > >> > > >> > > >> > packaging.
> > >> > > >> > > >> >
> > >> > > >> > > >> > But if we want to use dtcli as it is, then we would
> need
> > >> > > >> dt-env.sh
> > >> > > >> > > >> >
> > >> > > >> > > >> >
> > >> > > >> > > >> >
> > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> > >> > > >> > thomas@datatorrent.com
> > >> > > >> > > >
> > >> > > >> > > >> > wrote:
> > >> > > >> > > >> >
> > >> > > >> > > >> >> Looks good overall, though there shouldn't be any
> > >> dt-env.sh
> > >> > > >> > > >> >>
> > >> > > >> > > >> >> What will the bigtop package be built from, the
> source
> > >> tar?
> > >> > > >> > > >> >>
> > >> > > >> > > >> >>
> > >> > > >> > > >> >>
> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > >> > > >> > > chinmay@apache.org
> > >> > > >> > > >> >
> > >> > > >> > > >> >> wrote:
> > >> > > >> > > >> >>
> > >> > > >> > > >> >> > Hi Thomas,
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > Thanks for the feedback.
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > First, we're not changing the name anywhere. We'll
> > >> follow
> > >> > > what
> > >> > > >> > > >> >> currently is
> > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating version.
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > Secondly, I've mentioned a directory structure
> below
> > >> which
> > >> > > is
> > >> > > >> > > inline
> > >> > > >> > > >> >> with
> > >> > > >> > > >> >> > other existing integrations with bigtop.
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > The need for each file is as follows:
> > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> > >> > > >> engine/src/main/scripts/
> > >> > > >> > > of
> > >> > > >> > > >> >> source
> > >> > > >> > > >> >> > code apex 3.3.0-incubating.
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli.
> This
> > >> > > searches
> > >> > > >> for
> > >> > > >> > > >> hadoop
> > >> > > >> > > >> >> > binary path and exports an env variable for dtcli
> to
> > >> use
> > >> > it.
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is
> extracted
> > >> in a
> > >> > > >> similar
> > >> > > >> > > >> way to
> > >> > > >> > > >> >> > how DT Community Edition finds dependency jars.
> > >> > > >> > > >> >> > Please note that this is the first iteration list
> of
> > >> jars.
> > >> > > I'm
> > >> > > >> > > >> trying to
> > >> > > >> > > >> >> > narrow this down to only those which are really
> > >> required.
> > >> > > >> > > >> >> > The test that I'm running to check if dtcli runs
> fine
> > >> with
> > >> > > >> given
> > >> > > >> > > >> >> dependency
> > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi demo.
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > Above is the directory structure which is required
> > for
> > >> CLI
> > >> > > to
> > >> > > >> > work.
> > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned are some
> > >> > additional
> > >> > > >> > files
> > >> > > >> > > >> which
> > >> > > >> > > >> >> > makes the like of administrator easier. For eg.
> > >> /etc/skel
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > Please let me know if this is inline with what
> you're
> > >> > > >> thinking.
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > Thanks,
> > >> > > >> > > >> >> > Chinmay.
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > .
> > >> > > >> > > >> >> > |-- bin
> > >> > > >> > > >> >> > |   `-- dtcli
> > >> > > >> > > >> >> > |-- conf
> > >> > > >> > > >> >> > |   `-- dt-env.sh
> > >> > > >> > > >> >> > `-- lib
> > >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > >> > > >> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > >> > > >> > > >> >> >     |-- apex-engine.jar
> > >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > >> > > >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > >> > > >> > > >> >> >     |-- jline-2.11.jar
> > >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > >> > > >> > > >> >> >     |-- minlog-1.2.jar
> > >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > 3 directories, 40 files
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > >> > > >> > > >> thomas@datatorrent.com>
> > >> > > >> > > >> >> > wrote:
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >> > > Chinmay,
> > >> > > >> > > >> >> > >
> > >> > > >> > > >> >> > > Before discussing where to put the files, let's
> > make
> > >> > sure
> > >> > > >> they
> > >> > > >> > > are
> > >> > > >> > > >> >> really
> > >> > > >> > > >> >> > > needed for the operation of the CLI. As for
> names,
> > >> > > anything
> > >> > > >> new
> > >> > > >> > > >> needs
> > >> > > >> > > >> >> to
> > >> > > >> > > >> >> > > reflect Apex in the name and should follow common
> > >> > > >> conventions,
> > >> > > >> > > >> >> especially
> > >> > > >> > > >> >> > > in Bigtop where there are many existing
> > integrations
> > >> to
> > >> > > look
> > >> > > >> > at.
> > >> > > >> > > >> >> > >
> > >> > > >> > > >> >> > > Thanks,
> > >> > > >> > > >> >> > > Thomas
> > >> > > >> > > >> >> > >
> > >> > > >> > > >> >> > >
> > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay
> Kolhatkar
> > <
> > >> > > >> > > >> >> > > chinmay@datatorrent.com>
> > >> > > >> > > >> >> > > wrote:
> > >> > > >> > > >> >> > >
> > >> > > >> > > >> >> > > > Really good points Pradeep and Aniruddha.
> > >> > > >> > > >> >> > > >
> > >> > > >> > > >> >> > > > 1. I believe we won't need to change the dtcli
> > >> > > >> considering it
> > >> > > >> > > >> works
> > >> > > >> > > >> >> > with
> > >> > > >> > > >> >> > > DT
> > >> > > >> > > >> >> > > > Community edition. We can keep the directory
> > >> structure
> > >> > > >> > similar
> > >> > > >> > > to
> > >> > > >> > > >> >> that.
> > >> > > >> > > >> >> > > > dt-env.sh has variables which contains
> > information
> > >> > > >> required
> > >> > > >> > for
> > >> > > >> > > >> >> dtcli
> > >> > > >> > > >> >> > to
> > >> > > >> > > >> >> > > > launch.
> > >> > > >> > > >> >> > > >
> > >> > > >> > > >> >> > > > 2. Let me check with bigtop community that
> > whether
> > >> > they
> > >> > > >> > > >> facilitate
> > >> > > >> > > >> >> the
> > >> > > >> > > >> >> > > > installation of rpms/debs before user is
> created.
> > >> In
> > >> > > >> either
> > >> > > >> > > case,
> > >> > > >> > > >> >> > current
> > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home directory.
> > Also
> > >> > > before
> > >> > > >> > > putting
> > >> > > >> > > >> >> > > anything
> > >> > > >> > > >> >> > > > in /etc/skel we need to define what are the
> > default
> > >> > > >> contents
> > >> > > >> > > that
> > >> > > >> > > >> >> > should
> > >> > > >> > > >> >> > > go
> > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults,
> > >> probably we
> > >> > > >> should
> > >> > > >> > > not
> > >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > >> > > >> > > >> >> > > >
> > >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This
> way
> > >> > > contents
> > >> > > >> of
> > >> > > >> > > >> >> dt-env.sh
> > >> > > >> > > >> >> > > are
> > >> > > >> > > >> >> > > > present as env variables. But I see a catch
> > there.
> > >> > > Adding
> > >> > > >> > > >> dt-env.sh
> > >> > > >> > > >> >> to
> > >> > > >> > > >> >> > > > /etc/profile.d would make all the variables
> > >> available
> > >> > at
> > >> > > >> > > runtime
> > >> > > >> > > >> all
> > >> > > >> > > >> >> > the
> > >> > > >> > > >> >> > > > time. I feel a little skeptical about that.
> > Maybe a
> > >> > > >> possible
> > >> > > >> > > >> >> collision
> > >> > > >> > > >> >> > > can
> > >> > > >> > > >> >> > > > occur with other application vars.
> > >> > > >> > > >> >> > > > Moreover, current dtcli does source
> > >> > "../conf/dt-env.sh"
> > >> > > >> and
> > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are
> anyway
> > >> > > >> available.
> > >> > > >> > > >> >> > > >
> > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is happening
> > at
> > >> > > Bigtop
> > >> > > >> > > >> >> > > Jira/repository.
> > >> > > >> > > >> >> > > > Packaging related code usually goes there.
> > (That's
> > >> > what
> > >> > > >> all
> > >> > > >> > the
> > >> > > >> > > >> >> > > components
> > >> > > >> > > >> >> > > > in bigtop does).
> > >> > > >> > > >> >> > > > Having said that, once a PR is created, I'll be
> > >> > sharing
> > >> > > >> the
> > >> > > >> > > link
> > >> > > >> > > >> of
> > >> > > >> > > >> >> PR
> > >> > > >> > > >> >> > > > here, so that apex community as well can review
> > it.
> > >> > > >> > > >> >> > > >
> > >> > > >> > > >> >> > > > Thanks,
> > >> > > >> > > >> >> > > > Chinmay.
> > >> > > >> > > >> >> > > >
> > >> > > >> > > >> >> > > >
> > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha
> > >> Thombare <
> > >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > >> > > >> > > >> >> > > >
> > >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> > >> > > >> > > >> >> > > > > We may need to iron out details about exact
> > paths
> > >> > etc.
> > >> > > >> > > >> >> > > > > Which can be done on jira / PR comments.
> > >> > > >> > > >> >> > > > > Is that right @dev?
> > >> > > >> > > >> >> > > > >
> > >> > > >> > > >> >> > > > >
> > >> > > >> > > >> >> > > > >
> > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi
> <
> > >> > > >> > > >> >> > apache@pradeepdalvi.com>
> > >> > > >> > > >> >> > > > > wrote:
> > >> > > >> > > >> >> > > > >
> > >> > > >> > > >> >> > > > > > Inline comments...
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
> > >> > Kolhatkar <
> > >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > >> > > >> > > >> >> > > > > > wrote:
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present in DT
> > >> community
> > >> > > >> > edition
> > >> > > >> > > >> will
> > >> > > >> > > >> >> be
> > >> > > >> > > >> >> > > > > included
> > >> > > >> > > >> >> > > > > > > there.
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > > > Here is the list of jars I found from
> > common
> > >> > > >> > dependencies
> > >> > > >> > > >> >> between
> > >> > > >> > > >> >> > > of
> > >> > > >> > > >> >> > > > > apex
> > >> > > >> > > >> >> > > > > > > and DT Community edition:
> > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > >> > > >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > >> > > >> > > >> >> > > > > > > jline-2.11.jar
> > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some tests do
> > check
> > >> > that
> > >> > > >> > dtcli
> > >> > > >> > > >> works
> > >> > > >> > > >> >> > > > properly
> > >> > > >> > > >> >> > > > > > for
> > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with only
> > these
> > >> > > >> libraries
> > >> > > >> > > >> >> present in
> > >> > > >> > > >> >> > > > > > isolation
> > >> > > >> > > >> >> > > > > > > without dependency on local m2.
> > >> > > >> > > >> >> > > > > > > I'm believe that there are unwanted jars
> > >> which
> > >> > are
> > >> > > >> used
> > >> > > >> > > for
> > >> > > >> > > >> >> > compile
> > >> > > >> > > >> >> > > > > time
> > >> > > >> > > >> >> > > > > > > dependency and not runtime in above list
> > >> which I
> > >> > > can
> > >> > > >> > drop
> > >> > > >> > > >> to
> > >> > > >> > > >> >> keep
> > >> > > >> > > >> >> > > > > package
> > >> > > >> > > >> >> > > > > > > size minimal.
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > > > @Anniruddha,
> > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of
> the
> > >> > > >> dt-env.sh is
> > >> > > >> > > >> >> required.
> > >> > > >> > > >> >> > > > Only
> > >> > > >> > > >> >> > > > > > > required values I see in it are
> classpath.
> > >> and
> > >> > > >> dtcli is
> > >> > > >> > > >> >> capable
> > >> > > >> > > >> >> > of
> > >> > > >> > > >> >> > > > > > building
> > >> > > >> > > >> >> > > > > > > it on the fly.
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see
> > how
> > >> > dtcli
> > >> > > >> can
> > >> > > >> > > find
> > >> > > >> > > >> >> the
> > >> > > >> > > >> >> > jar
> > >> > > >> > > >> >> > > > > path,
> > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf files. If we
> > >> need a
> > >> > > >> change
> > >> > > >> > in
> > >> > > >> > > >> >> dtcli,
> > >> > > >> > > >> >> > > then
> > >> > > >> > > >> >> > > > > > > community's opinion is required for
> whether
> > >> > dtcli
> > >> > > >> > should
> > >> > > >> > > >> >> change
> > >> > > >> > > >> >> > in
> > >> > > >> > > >> >> > > > our
> > >> > > >> > > >> >> > > > > > repo
> > >> > > >> > > >> >> > > > > > > or a copy of that with required changes
> > >> exist in
> > >> > > >> bigtop
> > >> > > >> > > >> repo
> > >> > > >> > > >> >> > until
> > >> > > >> > > >> >> > > we
> > >> > > >> > > >> >> > > > > > make
> > >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > > Can we have these file paths set using
> > >> environment
> > >> > > >> > > variables
> > >> > > >> > > >> >> with
> > >> > > >> > > >> >> > > some
> > >> > > >> > > >> >> > > > > > default values in dtcli?
> > >> > > >> > > >> >> > > > > > Then we can set such params in dt-env.sh.
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file
> > different
> > >> > from
> > >> > > >> > > >> >> dt-site.xml?
> > >> > > >> > > >> >> > Not
> > >> > > >> > > >> >> > > > > sure
> > >> > > >> > > >> >> > > > > > if
> > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help. Files &
> > Dirs
> > >> > from
> > >> > > >> > > /etc/skel
> > >> > > >> > > >> >> are
> > >> > > >> > > >> >> > > > copied
> > >> > > >> > > >> >> > > > > to
> > >> > > >> > > >> >> > > > > > > home of new user when useradd program is
> > >> called.
> > >> > > But
> > >> > > >> > for
> > >> > > >> > > >> >> existing
> > >> > > >> > > >> >> > > > users
> > >> > > >> > > >> >> > > > > > > this won't be of any use. Moreover, dtcli
> > >> > creates
> > >> > > >> > ~/.dt/
> > >> > > >> > > on
> > >> > > >> > > >> >> the
> > >> > > >> > > >> >> > fly
> > >> > > >> > > >> >> > > > if
> > >> > > >> > > >> >> > > > > > not
> > >> > > >> > > >> >> > > > > > > encountered for the first time. Again
> > >> correct me
> > >> > > if
> > >> > > >> I'm
> > >> > > >> > > >> wrong.
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > > Yes, you are right. However this would
> > >> probably be
> > >> > > >> > > necessary
> > >> > > >> > > >> >> step
> > >> > > >> > > >> >> > for
> > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get installed
> > during
> > >> OS
> > >> > > >> install
> > >> > > >> > > and
> > >> > > >> > > >> >> > before
> > >> > > >> > > >> >> > > > user
> > >> > > >> > > >> >> > > > > > accounts were created.
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm
> > not
> > >> > sure
> > >> > > >> its
> > >> > > >> > the
> > >> > > >> > > >> best
> > >> > > >> > > >> >> > > idea.
> > >> > > >> > > >> >> > > > > > We'll
> > >> > > >> > > >> >> > > > > > > anyway put dtcli in location which is by
> > >> default
> > >> > > >> > present
> > >> > > >> > > in
> > >> > > >> > > >> >> path
> > >> > > >> > > >> >> > > i.e.
> > >> > > >> > > >> >> > > > > > > /usr/bin.
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > > For other env variables specific to apex,
> > I'll
> > >> > > prefer
> > >> > > >> to
> > >> > > >> > > use
> > >> > > >> > > >> >> > > dt-env.sh
> > >> > > >> > > >> >> > > > > and
> > >> > > >> > > >> >> > > > > > > source it in dtcli rather than changing
> > >> bashrc
> > >> > > >> etc...
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to
> /etc/profile.d/
> > >> shall
> > >> > > be
> > >> > > >> the
> > >> > > >> > > >> >> cleaner
> > >> > > >> > > >> >> > way
> > >> > > >> > > >> >> > > > to
> > >> > > >> > > >> >> > > > > > achieve the same.
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > > Thanks,
> > >> > > >> > > >> >> > > > > > > Chinmay.
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM,
> Aniruddha
> > >> > > Thombare
> > >> > > >> <
> > >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > > > > Hi,
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > > @Chinmay,
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > > We need to consider following:
> > >> > > >> > > >> >> > > > > > > > System wide default config files can be
> > >> > located
> > >> > > at
> > >> > > >> > > >> following
> > >> > > >> > > >> >> > > > > locations:
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to
> > change
> > >> > > dtcli
> > >> > > >> > > >> behaviour
> > >> > > >> > > >> >> on
> > >> > > >> > > >> >> > > how
> > >> > > >> > > >> >> > > > it
> > >> > > >> > > >> >> > > > > > > finds
> > >> > > >> > > >> >> > > > > > > > those locations)
> > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to
> > >> change
> > >> > > dtcli
> > >> > > >> > > >> >> behaviour on
> > >> > > >> > > >> >> > > how
> > >> > > >> > > >> >> > > > > it
> > >> > > >> > > >> >> > > > > > > > finds those locations)
> > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other
> > files
> > >> > (for
> > >> > > >> new
> > >> > > >> > > users
> > >> > > >> > > >> >> that
> > >> > > >> > > >> >> > > will
> > >> > > >> > > >> >> > > > > be
> > >> > > >> > > >> >> > > > > > > > created in system in future.)
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > > We may also have to modify bashrc /
> > >> > bashprofile
> > >> > > >> for
> > >> > > >> > > >> >> population
> > >> > > >> > > >> >> > > any
> > >> > > >> > > >> >> > > > > > > > variables that are required.
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > > @dev, please put in your comments /
> > >> > suggestion.
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > > Thanks,
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > > Aniruddha
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas
> > >> Weise
> > >> > <
> > >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > >> > > >> > > >> >> > > > > > > > wrote:
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > -->
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM,
> Chinmay
> > >> > > >> Kolhatkar <
> > >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > wrote:
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging effort
> > >> (rpm+deb)
> > >> > > of
> > >> > > >> > apex,
> > >> > > >> > > >> here
> > >> > > >> > > >> >> > are
> > >> > > >> > > >> >> > > > some
> > >> > > >> > > >> >> > > > > > > > > proposals
> > >> > > >> > > >> >> > > > > > > > > > about package structure etc..
> > >> > > >> > > >> >> > > > > > > > > > Before posting it on bbigtop
> mailing
> > >> > list, I
> > >> > > >> have
> > >> > > >> > > >> some
> > >> > > >> > > >> >> > > question
> > >> > > >> > > >> >> > > > > for
> > >> > > >> > > >> >> > > > > > > > apex
> > >> > > >> > > >> >> > > > > > > > > > community.
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > Proposed Directory structure of
> apex
> > >> > package
> > >> > > >> for
> > >> > > >> > > both
> > >> > > >> > > >> >> deb &
> > >> > > >> > > >> >> > > > rpm:
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > >> > > >> > > >> >> > > > > > > > > >
> > >> /usr/lib/apex/lib/apex-api-<version>.jar
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > /usr/lib/apex/lib/apex-common-<version>.jar
> > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent
> > >> jars>
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> > /usr/lib/apex/bin/dtcli
> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > >> > > >> > > >> >> > > > > > > > > >
> > /usr/share/doc/apex/license/LICENSE.txt
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> = All the
> 3rd
> > >> party
> > >> > > >> jars
> > >> > > >> > > which
> > >> > > >> > > >> >> are
> > >> > > >> > > >> >> > > > > required
> > >> > > >> > > >> >> > > > > > > for
> > >> > > >> > > >> >> > > > > > > > > apex
> > >> > > >> > > >> >> > > > > > > > > > to run. Usually the dependencies
> are
> > >> > > packaged
> > >> > > >> as
> > >> > > >> > > >> part of
> > >> > > >> > > >> >> > > > rpm/deb
> > >> > > >> > > >> >> > > > > by
> > >> > > >> > > >> >> > > > > > > any
> > >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > Can you specify what those jars are
> in
> > >> > Bigtop
> > >> > > >> > > context?
> > >> > > >> > > >> >> Same
> > >> > > >> > > >> >> > as
> > >> > > >> > > >> >> > > > > > shipped
> > >> > > >> > > >> >> > > > > > > > with
> > >> > > >> > > >> >> > > > > > > > > DT community addition under lib/ ?
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of
> > >> > > corresponding
> > >> > > >> of
> > >> > > >> > > 3rd
> > >> > > >> > > >> >> party
> > >> > > >> > > >> >> > > jars
> > >> > > >> > > >> >> > > > > > which
> > >> > > >> > > >> >> > > > > > > > > needs
> > >> > > >> > > >> >> > > > > > > > > > to included while packaging.
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > Questions related to this:
> > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli of apex
> as
> > >> > "apex"
> > >> > > >> > instead
> > >> > > >> > > >> of
> > >> > > >> > > >> >> > > "dtcli"
> > >> > > >> > > >> >> > > > in
> > >> > > >> > > >> >> > > > > > > > bigtop
> > >> > > >> > > >> >> > > > > > > > > > package?
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > I think we should keep the name until
> > we
> > >> are
> > >> > > >> able
> > >> > > >> > to
> > >> > > >> > > >> >> change
> > >> > > >> > > >> >> > it
> > >> > > >> > > >> >> > > in
> > >> > > >> > > >> >> > > > > > Apex.
> > >> > > >> > > >> >> > > > > > > > > Otherwise this may get confusing.
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares in
> bigtop
> > >> have
> > >> > > man
> > >> > > >> > page
> > >> > > >> > > >> for
> > >> > > >> > > >> >> > their
> > >> > > >> > > >> >> > > > > > > > > executables.
> > >> > > >> > > >> >> > > > > > > > > > I think we should have it too for
> > >> dtcli.
> > >> > Is
> > >> > > >> there
> > >> > > >> > > any
> > >> > > >> > > >> >> > > > > documentation
> > >> > > >> > > >> >> > > > > > > > > which I
> > >> > > >> > > >> >> > > > > > > > > > can convert to man page? or can I
> use
> > >> > output
> > >> > > >> of
> > >> > > >> > > >> "dtcli
> > >> > > >> > > >> >> > > --help"?
> > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call version of
> apex
> > >> in
> > >> > > >> Bigtop
> > >> > > >> > as
> > >> > > >> > > >> >> 3.3.0 OR
> > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of
> > bigtop
> > >> to
> > >> > > >> depend
> > >> > > >> > on
> > >> > > >> > > >> >> 2.7.1
> > >> > > >> > > >> >> > > > > version
> > >> > > >> > > >> >> > > > > > of
> > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that we
> > see
> > >> > with
> > >> > > >> this
> > >> > > >> > > >> >> > dependency?
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > Should work without changed. I
> thought
> > we
> > >> > > >> certified
> > >> > > >> > > >> >> against
> > >> > > >> > > >> >> > > > 2.7.0?
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache mirror
> > from
> > >> > which
> > >> > > >> > bigtop
> > >> > > >> > > >> will
> > >> > > >> > > >> >> > pick
> > >> > > >> > > >> >> > > > the
> > >> > > >> > > >> >> > > > > > > apex
> > >> > > >> > > >> >> > > > > > > > > > source for compilation and
> packaging.
> > >> > Please
> > >> > > >> > > correct
> > >> > > >> > > >> if
> > >> > > >> > > >> >> > > > > incorrect:
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > >
> > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > >> > > >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the
> > >> mirror
> > >> > > >> used
> > >> > > >> > by
> > >> > > >> > > >> all
> > >> > > >> > > >> >> the
> > >> > > >> > > >> >> > > > > > softwares
> > >> > > >> > > >> >> > > > > > > in
> > >> > > >> > > >> >> > > > > > > > > > bigtop.
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > Please share your opinion.
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > Thanks,
> > >> > > >> > > >> >> > > > > > > > > > Chinmay.
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM,
> > >> Chinmay
> > >> > > >> > Kolhatkar
> > >> > > >> > > <
> > >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > >> > > >> > > >> >> > > > > > > > > > wrote:
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the
> > >> packaging.
> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > > I've already started discussion
> on
> > >> > bigtop
> > >> > > >> dev
> > >> > > >> > > >> mailing
> > >> > > >> > > >> >> > list
> > >> > > >> > > >> >> > > > for
> > >> > > >> > > >> >> > > > > > > > > > > integration. Also created a Jira
> > for
> > >> the
> > >> > > >> same.
> > >> > > >> > > For
> > >> > > >> > > >> >> this
> > >> > > >> > > >> >> > > > > > communities
> > >> > > >> > > >> >> > > > > > > > > > > reference, here is the bigtop
> Jira:
> > >> > > >> > BIGTOP-2313.
> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM,
> > >> Thomas
> > >> > > >> Weise <
> > >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > > wrote:
> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything under
> > >> > prerequisites
> > >> > > >> that
> > >> > > >> > > >> would
> > >> > > >> > > >> >> be a
> > >> > > >> > > >> >> > > > > > problem.
> > >> > > >> > > >> >> > > > > > > We
> > >> > > >> > > >> >> > > > > > > > > > >> looked
> > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing
> compatibility
> > >> as
> > >> > > part
> > >> > > >> of
> > >> > > >> > > >> >> becoming
> > >> > > >> > > >> >> > an
> > >> > > >> > > >> >> > > > > > > incubator
> > >> > > >> > > >> >> > > > > > > > > > >> project.
> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the packaging
> > during
> > >> > the
> > >> > > >> next
> > >> > > >> > > >> weeks.
> > >> > > >> > > >> >> > Since
> > >> > > >> > > >> >> > > > the
> > >> > > >> > > >> >> > > > > > > work
> > >> > > >> > > >> >> > > > > > > > > will
> > >> > > >> > > >> >> > > > > > > > > > >> be
> > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related
> > discussions
> > >> and
> > >> > > >> JIRAs
> > >> > > >> > > >> should
> > >> > > >> > > >> >> > also
> > >> > > >> > > >> >> > > be
> > >> > > >> > > >> >> > > > > > > there.
> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have the
> > packaging
> > >> in
> > >> > > >> place
> > >> > > >> > by
> > >> > > >> > > >> end
> > >> > > >> > > >> >> Feb.
> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > >> > > >> > > >> >> > > > > > > > > > >> Thomas
> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM,
> > >> > Chinmay
> > >> > > >> > > >> Kolhatkar <
> > >> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> > >> > > >> > > >> >> > > > > > > > > > >> wrote:
> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work on
> adding
> > >> > Apache
> > >> > > >> Apex
> > >> > > >> > > as a
> > >> > > >> > > >> >> > > component
> > >> > > >> > > >> >> > > > > to
> > >> > > >> > > >> >> > > > > > > > Apache
> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging system
> > for
> > >> > the
> > >> > > >> > Apache
> > >> > > >> > > >> big
> > >> > > >> > > >> >> data
> > >> > > >> > > >> >> > > > > > > ecosystem.
> > >> > > >> > > >> >> > > > > > > > > > >> Several
> > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most
> > >> recently
> > >> > > EMR.
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira task
> > in
> > >> > > >> APEXCORE
> > >> > > >> > for
> > >> > > >> > > >> the
> > >> > > >> > > >> >> > same:
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of execution is
> as
> > >> > > follows:
> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard
> and
> > >> soft
> > >> > > >> > > expectation
> > >> > > >> > > >> >> for
> > >> > > >> > > >> >> > new
> > >> > > >> > > >> >> > > > > > > > components
> > >> > > >> > > >> >> > > > > > > > > to
> > >> > > >> > > >> >> > > > > > > > > > >> get
> > >> > > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
> > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of it:
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > >
> > >> > > >> > > >> >> > > >
> > >> > > >> > > >> >> > >
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >>
> > >> > > >> > > >>
> > >> > > >> > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to be
> > standard
> > >> ASF
> > >> > > >> based
> > >> > > >> > > >> >> > > requirements,
> > >> > > >> > > >> >> > > > > but
> > >> > > >> > > >> >> > > > > > > few
> > >> > > >> > > >> >> > > > > > > > > need
> > >> > > >> > > >> >> > > > > > > > > > >> to
> > >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects are
> > expected
> > >> to
> > >> > be
> > >> > > >> > > Licensed
> > >> > > >> > > >> >> under
> > >> > > >> > > >> >> > > > > Apache
> > >> > > >> > > >> >> > > > > > > > > License,
> > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
> > >> dependencies
> > >> > are
> > >> > > >> > > expected
> > >> > > >> > > >> >> to be
> > >> > > >> > > >> >> > > > > > > compatible
> > >> > > >> > > >> >> > > > > > > > > with
> > >> > > >> > > >> >> > > > > > > > > > >> this
> > >> > > >> > > >> >> > > > > > > > > > >> > license)
> > >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0
> but
> > >> need
> > >> > to
> > >> > > >> > check
> > >> > > >> > > if
> > >> > > >> > > >> >> > > > > dependencies
> > >> > > >> > > >> >> > > > > > of
> > >> > > >> > > >> >> > > > > > > > > Apex
> > >> > > >> > > >> >> > > > > > > > > > >> are
> > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This
> I
> > >> guess
> > >> > > >> would
> > >> > > >> > > be a
> > >> > > >> > > >> >> > > > > verification
> > >> > > >> > > >> >> > > > > > > > check.
> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects are
> > expected
> > >> to
> > >> > be
> > >> > > >> > > >> compatible
> > >> > > >> > > >> >> > with
> > >> > > >> > > >> >> > > > all
> > >> > > >> > > >> >> > > > > of
> > >> > > >> > > >> >> > > > > > > the
> > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms that
> Bigtop
> > >> > > >> distribution
> > >> > > >> > > is
> > >> > > >> > > >> >> > > targeting
> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs verifying
> > whether
> > >> > our
> > >> > > >> > > software
> > >> > > >> > > >> >> runs
> > >> > > >> > > >> >> > > fine
> > >> > > >> > > >> >> > > > in
> > >> > > >> > > >> >> > > > > > > > > centos-6
> > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20
> ubuntu-14.04
> > >> > > debian-8
> > >> > > >> > > >> >> > opensuse-13.2.
> > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that
> should
> > be
> > >> > > added
> > >> > > >> for
> > >> > > >> > > >> >> > deployment.
> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test
> > artifacts
> > >> > which
> > >> > > >> goes
> > >> > > >> > > >> beyond
> > >> > > >> > > >> >> > > smoke
> > >> > > >> > > >> >> > > > > test
> > >> > > >> > > >> >> > > > > > > > > > >> >     - These are basically the
> > >> > > integration
> > >> > > >> > > tests
> > >> > > >> > > >> for
> > >> > > >> > > >> >> > > > > > verification
> > >> > > >> > > >> >> > > > > > > > > after
> > >> > > >> > > >> >> > > > > > > > > > >> the
> > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a soft
> > >> > requirement,
> > >> > > >> but
> > >> > > >> > > aim
> > >> > > >> > > >> is
> > >> > > >> > > >> >> to
> > >> > > >> > > >> >> > > > > achieve
> > >> > > >> > > >> >> > > > > > > this
> > >> > > >> > > >> >> > > > > > > > > as
> > >> > > >> > > >> >> > > > > > > > > > >> well
> > >> > > >> > > >> >> > > > > > > > > > >> > or at least have explanation
> why
> > >> not
> > >> > to
> > >> > > >> > > include.
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >> > If there are any from the link
> > >> which
> > >> > > >> > > explicitly
> > >> > > >> > > >> >> needs
> > >> > > >> > > >> >> > to
> > >> > > >> > > >> >> > > > be
> > >> > > >> > > >> >> > > > > > > > checked
> > >> > > >> > > >> >> > > > > > > > > > >> other
> > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please let us
> > know.
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as
> > component
> > >> to
> > >> > > >> Bigtop*
> > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail archive
> of
> > >> > Bigtop,
> > >> > > >> it
> > >> > > >> > was
> > >> > > >> > > >> >> learnt
> > >> > > >> > > >> >> > > that
> > >> > > >> > > >> >> > > > > the
> > >> > > >> > > >> >> > > > > > > > > bigtop
> > >> > > >> > > >> >> > > > > > > > > > >> > community want to see the
> > >> addition of
> > >> > > new
> > >> > > >> > > >> >> components
> > >> > > >> > > >> >> > in
> > >> > > >> > > >> >> > > > > > phases.
> > >> > > >> > > >> >> > > > > > > > Here
> > >> > > >> > > >> >> > > > > > > > > > are
> > >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > >> > > >> > > >> >> > > > > > > > > > >> >     - This needs creating of
> > >> package
> > >> > > i.e.
> > >> > > >> > rpm
> > >> > > >> > > &
> > >> > > >> > > >> deb
> > >> > > >> > > >> >> > > files.
> > >> > > >> > > >> >> > > > > > > > > > >> >     - documentations/READMEs,
> > >> > LICENSE,
> > >> > > >> > > >> DISCLAMER,
> > >> > > >> > > >> >> > NOTES
> > >> > > >> > > >> >> > > > etc
> > >> > > >> > > >> >> > > > > if
> > >> > > >> > > >> >> > > > > > > any
> > >> > > >> > > >> >> > > > > > > > > > >> needed.
> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any documentation that
> > need
> > >> to
> > >> > be
> > >> > > >> > added
> > >> > > >> > > to
> > >> > > >> > > >> >> > > > > distribution
> > >> > > >> > > >> >> > > > > > of
> > >> > > >> > > >> >> > > > > > > > our
> > >> > > >> > > >> >> > > > > > > > > > >> > software.
> > >> > > >> > > >> >> > > > > > > > > > >> >     - Any license information
> of
> > >> > > >> > dependencies
> > >> > > >> > > >> >> required
> > >> > > >> > > >> >> > > to
> > >> > > >> > > >> >> > > > be
> > >> > > >> > > >> >> > > > > > > added
> > >> > > >> > > >> >> > > > > > > > > to
> > >> > > >> > > >> >> > > > > > > > > > >> > package
> > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke test for
> > >> > packaging.
> > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for
> automatic
> > >> > > >> deployment
> > >> > > >> > and
> > >> > > >> > > >> >> > > > configuration
> > >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for
> > >> > automatic
> > >> > > >> > > >> deployment
> > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> > >> > > >> > > >> >> > > > > > > > > > >> >     - For verification of
> > >> > deployments.
> > >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> > >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make
> > sure
> > >> all
> > >> > > new
> > >> > > >> > > stuff
> > >> > > >> > > >> is
> > >> > > >> > > >> >> > > > compliant
> > >> > > >> > > >> >> > > > > > > with
> > >> > > >> > > >> >> > > > > > > > > ASF
> > >> > > >> > > >> >> > > > > > > > > > >> > license requirements. If you
> add
> > >> code
> > >> > > >> > licenses
> > >> > > >> > > >> >> under
> > >> > > >> > > >> >> > > > > different
> > >> > > >> > > >> >> > > > > > > > > > licenses,
> > >> > > >> > > >> >> > > > > > > > > > >> > those would need to be listed
> in
> > >> the
> > >> > > >> NOTICE.
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >> > Please share your thoughts on
> > the
> > >> > > >> approach.
> > >> > > >> > > >> >> > > > > > > > > > >> > We'll start corresponding
> > >> > communication
> > >> > > >> on
> > >> > > >> > > >> bigtop
> > >> > > >> > > >> >> > > mailing
> > >> > > >> > > >> >> > > > > list
> > >> > > >> > > >> >> > > > > > > as
> > >> > > >> > > >> >> > > > > > > > > > well.
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >> > We have some specific
> > >> > > >> questions/suggestions
> > >> > > >> > > >> >> related to
> > >> > > >> > > >> >> > > > what
> > >> > > >> > > >> >> > > > > > > should
> > >> > > >> > > >> >> > > > > > > > > be
> > >> > > >> > > >> >> > > > > > > > > > >> the
> > >> > > >> > > >> >> > > > > > > > > > >> > content of the package and
> what
> > >> > should
> > >> > > be
> > >> > > >> > the
> > >> > > >> > > >> smoke
> > >> > > >> > > >> >> > > tests,
> > >> > > >> > > >> >> > > > > but
> > >> > > >> > > >> >> > > > > > > in
> > >> > > >> > > >> >> > > > > > > > > the
> > >> > > >> > > >> >> > > > > > > > > > >> > interest of not having too
> much
> > >> > content
> > >> > > >> > here,
> > >> > > >> > > >> we'll
> > >> > > >> > > >> >> > put
> > >> > > >> > > >> >> > > > the
> > >> > > >> > > >> >> > > > > > > > > questions
> > >> > > >> > > >> >> > > > > > > > > > >> as a
> > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this
> > mailthread.
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > >> > > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> > > >> >> > > > > > > > > > >>
> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > > >
> > >> > > >> > > >> >> > > > > > > > >
> > >> > > >> > > >> >> > > > > > > >
> > >> > > >> > > >> >> > > > > > >
> > >> > > >> > > >> >> > > > > >
> > >> > > >> > > >> >> > > > >
> > >> > > >> > > >> >> > > >
> > >> > > >> > > >> >> > >
> > >> > > >> > > >> >> >
> > >> > > >> > > >> >>
> > >> > > >> > > >> >
> > >> > > >> > > >> >
> > >> > > >> > > >>
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> >
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

Please follow the recommendation and continue discussion on the Bigtop list.

--
sent from mobile
On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <ch...@apache.org> wrote:

> I got a recommendation from bigtop community to drop "-incubating" from
> version.
>
> Here is the mail thread:
>
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
>
> Please let me know if removing "-incubating" is fine.
>
> Thanks,
> Chinmay.
>
>
> On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > I've already started communication there.
> >
> > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> >> This is a question you should ask on the Bigtop mailing list.
> >>
> >> --
> >> sent from mobile
> >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org> wrote:
> >>
> >> > Yes. In case of RPM, the only restriction on version is that it can
> not
> >> > contain '-'.
> >> > However, as a workaround, one can use 'incubating' as a part of
> >> %{release}.
> >> >
> >> > Version: 3.3.0
> >> > Release: incubating
> >> >
> >> > apex-3.3.0-incubating.noarch.rpm
> >> >
> >> > It is the same practice followed for pre-release packages e.g. with
> rcX,
> >> > alphaX, betaX being used in %{release}.
> >> >
> >> > Regards,
> >> > Pradeep A. Dalvi
> >> >
> >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> chinmay@apache.org>
> >> > wrote:
> >> >
> >> > > Hi Everyone,
> >> > >
> >> > > In one of the previous mails in this thread, we decided that version
> >> of
> >> > the
> >> > > package will be "3.3.0-incubating".
> >> > > The problem that I'm facing is in RPM base version "-" is not
> allowed.
> >> > >
> >> > > Can we have package version to "3.3.0"?
> >> > > i.e. the packages created will be as follows:
> >> > > For RPM: apex-3.3.0-1.noarch.rpm
> >> > > For DEB: apex_3.3.0-1_all.deb
> >> > >
> >> > > There is another integration for Zepplin in Bigtop which is
> incubating
> >> > and
> >> > > is having base version as "0.5.6".
> >> > >
> >> > > Please let me know your thoughts.
> >> > >
> >> > > Thanks,
> >> > > Chinmay.
> >> > >
> >> > >
> >> > >
> >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> >> chinmay@apache.org>
> >> > > wrote:
> >> > >
> >> > > > Thanks everyone for the feedback.
> >> > > >
> >> > > > Also as Ram and David mentioned, I'll start creating the man page
> as
> >> > > well.
> >> > > >
> >> > > > Thanks,
> >> > > > Chinmay.
> >> > > >
> >> > > >
> >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> >> thomas@datatorrent.com>
> >> > > > wrote:
> >> > > >
> >> > > >> I think we already agreed that we will retain the current script
> >> name
> >> > in
> >> > > >> this bigtop iteration. There is a ticket for renaming, add you
> >> > > suggestions
> >> > > >> there.
> >> > > >>
> >> > > >> --
> >> > > >> sent from mobile
> >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> sandesh@datatorrent.com>
> >> > > wrote:
> >> > > >>
> >> > > >> > +1 for Symlink from apex to dtcli or start the transition
> >> process by
> >> > > >> making
> >> > > >> > dtcli as the symlink to apex cli.
> >> > > >> >
> >> > > >> > Renaming is a breaking change should be done during major
> version
> >> > > number
> >> > > >> > change.
> >> > > >> >
> >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> david@datatorrent.com
> >> >
> >> > > >> wrote:
> >> > > >> >
> >> > > >> > > Apologies for hitting the "Send" button too early.
> >> > > >> > >
> >> > > >> > > What is the scope of the name change?  If we simply have a
> >> symlink
> >> > > >> from
> >> > > >> > > apex to dtcli, then it should have no backward compatibility
> >> > problem
> >> > > >> at
> >> > > >> > > all.
> >> > > >> > >
> >> > > >> > > But the proper name change would involve:
> >> > > >> > > - Change the name of DTCli class to ApexCli
> >> > > >> > > - Change the prompt from dt> to apex>
> >> > > >> > > - Change the initial greeting when running the CLI from
> >> something
> >> > > like
> >> > > >> > "DT
> >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> >> > > >> > >
> >> > > >> > > The above changes may break existing scripts that rely on the
> >> the
> >> > > >> string
> >> > > >> > > "DT".
> >> > > >> > > But if folks are okay with this potential problem, I am okay
> >> with
> >> > > the
> >> > > >> > name
> >> > > >> > > change.
> >> > > >> > >
> >> > > >> > > For the man page, we can start with the content of this page:
> >> > > >> > >
> >> > > >> > > http://docs.datatorrent.com/dtcli/
> >> > > >> > >
> >> > > >> > > Plus the command line options when doing dtcli -h:
> >> > > >> > >
> >> > > >> > > usage: DTCli
> >> > > >> > >  -e <arg>    Commands are read from the argument
> >> > > >> > >  -f <arg>    Use the specified prompt at all time
> >> > > >> > >  -h          Print this help
> >> > > >> > >  -kp <arg>   Use the specified kerberos principal
> >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> >> > > >> > >  -p <arg>    JSONP padding function
> >> > > >> > >  -r          JSON Raw mode
> >> > > >> > >  -v          Verbose mode level 1
> >> > > >> > >  -vv         Verbose mode level 2
> >> > > >> > >  -vvv        Verbose mode level 3
> >> > > >> > >  -vvvv       Verbose mode level 4
> >> > > >> > >
> >> > > >> > > David
> >> > > >> > >
> >> > > >> > >
> >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> >> david@datatorrent.com
> >> > >
> >> > > >> > wrote:
> >> > > >> > >
> >> > > >> > > > What is the scope of the name change?  If we simply have a
> >> > symlink
> >> > > >> from
> >> > > >> > > > apex to dtcli, then it should have no backward
> compatibility
> >> > > >> problem at
> >> > > >> > > all.
> >> > > >> > > >
> >> > > >> > > > But the proper name change would involve:
> >> > > >> > > > - Change the name of DTCli class to ApexCli
> >> > > >> > > > - Change the prompt from dt> to apex>
> >> > > >> > > > - Change the initial greeting when running the CLI from
> >> > something
> >> > > >> like
> >> > > >> > > "DT
> >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> >> > > >> > > >
> >> > > >> > > > The above changes may break existing scripts that rely on
> the
> >> > the
> >> > > >> > string
> >> > > >> > > > "DT".
> >> > > >> > > > But if folks are okay with this potential problem, I am
> okay
> >> > with
> >> > > >> the
> >> > > >> > > name
> >> > > >> > > > change.
> >> > > >> > > >
> >> > > >> > > > For the man page, we can start with the content of this
> page:
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> >> > > >> chinmay@apache.org
> >> > > >> > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > >> Here is a suggestion I have related to name change:
> >> > > >> > > >>
> >> > > >> > > >> 1. While packaging we change the name of "dtcli" script to
> >> > > "apex".
> >> > > >> > This
> >> > > >> > > is
> >> > > >> > > >> done so that in future when we change the name to "apex",
> >> users
> >> > > of
> >> > > >> > > bigtop
> >> > > >> > > >> apex don't have to transition much.
> >> > > >> > > >> 2. We also keep a symlink named dtcli which points to apex
> >> > > script.
> >> > > >> > This
> >> > > >> > > is
> >> > > >> > > >> for backward compatibility.
> >> > > >> > > >> 3. We still keep conf/dt-env.sh for first integration in
> >> > bigtop.
> >> > > >> This
> >> > > >> > is
> >> > > >> > > >> in
> >> > > >> > > >> the interest of not changing the content of dtcli or apex
> >> > script.
> >> > > >> > > >> 4. In apex 3.4.0, we take care of all these naming related
> >> > > changes
> >> > > >> and
> >> > > >> > > >> update bigtop repository later to remove references to
> dtcli
> >> > all
> >> > > >> > > together.
> >> > > >> > > >>
> >> > > >> > > >> Please share your thoughts on above approach.
> >> > > >> > > >>
> >> > > >> > > >> Also, please share what could be the man page content for
> >> > "apex".
> >> > > >> > > >>
> >> > > >> > > >> Thanks,
> >> > > >> > > >> Chinmay.
> >> > > >> > > >>
> >> > > >> > > >>
> >> > > >> > > >>
> >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> >> > > >> > chinmay@apache.org>
> >> > > >> > > >> wrote:
> >> > > >> > > >>
> >> > > >> > > >> > Yes.. It'll be build from source tar downloaded from one
> >> of
> >> > the
> >> > > >> > apache
> >> > > >> > > >> > mirror: apache.osuosl.org
> >> > > >> > > >> >
> >> > > >> > > >> > The reason for dt-env.sh is it's sourced from dtcli.
> This
> >> > > leaves
> >> > > >> us
> >> > > >> > 3
> >> > > >> > > >> > options:
> >> > > >> > > >> > If we should not have any dt-env.sh, dtcli would need a
> >> > change
> >> > > >> while
> >> > > >> > > >> > packaging.
> >> > > >> > > >> >
> >> > > >> > > >> > But if we want to use dtcli as it is, then we would need
> >> > > >> dt-env.sh
> >> > > >> > > >> >
> >> > > >> > > >> >
> >> > > >> > > >> >
> >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> >> > > >> > thomas@datatorrent.com
> >> > > >> > > >
> >> > > >> > > >> > wrote:
> >> > > >> > > >> >
> >> > > >> > > >> >> Looks good overall, though there shouldn't be any
> >> dt-env.sh
> >> > > >> > > >> >>
> >> > > >> > > >> >> What will the bigtop package be built from, the source
> >> tar?
> >> > > >> > > >> >>
> >> > > >> > > >> >>
> >> > > >> > > >> >>
> >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> >> > > >> > > chinmay@apache.org
> >> > > >> > > >> >
> >> > > >> > > >> >> wrote:
> >> > > >> > > >> >>
> >> > > >> > > >> >> > Hi Thomas,
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > Thanks for the feedback.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > First, we're not changing the name anywhere. We'll
> >> follow
> >> > > what
> >> > > >> > > >> >> currently is
> >> > > >> > > >> >> > in source tarball for 3.3.0-incubating version.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > Secondly, I've mentioned a directory structure below
> >> which
> >> > > is
> >> > > >> > > inline
> >> > > >> > > >> >> with
> >> > > >> > > >> >> > other existing integrations with bigtop.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > The need for each file is as follows:
> >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> >> > > >> engine/src/main/scripts/
> >> > > >> > > of
> >> > > >> > > >> >> source
> >> > > >> > > >> >> > code apex 3.3.0-incubating.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This
> >> > > searches
> >> > > >> for
> >> > > >> > > >> hadoop
> >> > > >> > > >> >> > binary path and exports an env variable for dtcli to
> >> use
> >> > it.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is extracted
> >> in a
> >> > > >> similar
> >> > > >> > > >> way to
> >> > > >> > > >> >> > how DT Community Edition finds dependency jars.
> >> > > >> > > >> >> > Please note that this is the first iteration list of
> >> jars.
> >> > > I'm
> >> > > >> > > >> trying to
> >> > > >> > > >> >> > narrow this down to only those which are really
> >> required.
> >> > > >> > > >> >> > The test that I'm running to check if dtcli runs fine
> >> with
> >> > > >> given
> >> > > >> > > >> >> dependency
> >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi demo.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > Above is the directory structure which is required
> for
> >> CLI
> >> > > to
> >> > > >> > work.
> >> > > >> > > >> >> > What aniruddha and pradeep has mentioned are some
> >> > additional
> >> > > >> > files
> >> > > >> > > >> which
> >> > > >> > > >> >> > makes the like of administrator easier. For eg.
> >> /etc/skel
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > Please let me know if this is inline with what you're
> >> > > >> thinking.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > Thanks,
> >> > > >> > > >> >> > Chinmay.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > .
> >> > > >> > > >> >> > |-- bin
> >> > > >> > > >> >> > |   `-- dtcli
> >> > > >> > > >> >> > |-- conf
> >> > > >> > > >> >> > |   `-- dt-env.sh
> >> > > >> > > >> >> > `-- lib
> >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> >> > > >> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> >> > > >> > > >> >> >     |-- apex-engine.jar
> >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> >> > > >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> >> > > >> > > >> >> >     |-- jline-2.11.jar
> >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> >> > > >> > > >> >> >     |-- minlog-1.2.jar
> >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > 3 directories, 40 files
> >> > > >> > > >> >> >
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> >> > > >> > > >> thomas@datatorrent.com>
> >> > > >> > > >> >> > wrote:
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > > Chinmay,
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> > > Before discussing where to put the files, let's
> make
> >> > sure
> >> > > >> they
> >> > > >> > > are
> >> > > >> > > >> >> really
> >> > > >> > > >> >> > > needed for the operation of the CLI. As for names,
> >> > > anything
> >> > > >> new
> >> > > >> > > >> needs
> >> > > >> > > >> >> to
> >> > > >> > > >> >> > > reflect Apex in the name and should follow common
> >> > > >> conventions,
> >> > > >> > > >> >> especially
> >> > > >> > > >> >> > > in Bigtop where there are many existing
> integrations
> >> to
> >> > > look
> >> > > >> > at.
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> > > Thanks,
> >> > > >> > > >> >> > > Thomas
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar
> <
> >> > > >> > > >> >> > > chinmay@datatorrent.com>
> >> > > >> > > >> >> > > wrote:
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> > > > Really good points Pradeep and Aniruddha.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > 1. I believe we won't need to change the dtcli
> >> > > >> considering it
> >> > > >> > > >> works
> >> > > >> > > >> >> > with
> >> > > >> > > >> >> > > DT
> >> > > >> > > >> >> > > > Community edition. We can keep the directory
> >> structure
> >> > > >> > similar
> >> > > >> > > to
> >> > > >> > > >> >> that.
> >> > > >> > > >> >> > > > dt-env.sh has variables which contains
> information
> >> > > >> required
> >> > > >> > for
> >> > > >> > > >> >> dtcli
> >> > > >> > > >> >> > to
> >> > > >> > > >> >> > > > launch.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > 2. Let me check with bigtop community that
> whether
> >> > they
> >> > > >> > > >> facilitate
> >> > > >> > > >> >> the
> >> > > >> > > >> >> > > > installation of rpms/debs before user is created.
> >> In
> >> > > >> either
> >> > > >> > > case,
> >> > > >> > > >> >> > current
> >> > > >> > > >> >> > > > dtcli creates a .dt folder in home directory.
> Also
> >> > > before
> >> > > >> > > putting
> >> > > >> > > >> >> > > anything
> >> > > >> > > >> >> > > > in /etc/skel we need to define what are the
> default
> >> > > >> contents
> >> > > >> > > that
> >> > > >> > > >> >> > should
> >> > > >> > > >> >> > > go
> >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults,
> >> probably we
> >> > > >> should
> >> > > >> > > not
> >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This way
> >> > > contents
> >> > > >> of
> >> > > >> > > >> >> dt-env.sh
> >> > > >> > > >> >> > > are
> >> > > >> > > >> >> > > > present as env variables. But I see a catch
> there.
> >> > > Adding
> >> > > >> > > >> dt-env.sh
> >> > > >> > > >> >> to
> >> > > >> > > >> >> > > > /etc/profile.d would make all the variables
> >> available
> >> > at
> >> > > >> > > runtime
> >> > > >> > > >> all
> >> > > >> > > >> >> > the
> >> > > >> > > >> >> > > > time. I feel a little skeptical about that.
> Maybe a
> >> > > >> possible
> >> > > >> > > >> >> collision
> >> > > >> > > >> >> > > can
> >> > > >> > > >> >> > > > occur with other application vars.
> >> > > >> > > >> >> > > > Moreover, current dtcli does source
> >> > "../conf/dt-env.sh"
> >> > > >> and
> >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway
> >> > > >> available.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is happening
> at
> >> > > Bigtop
> >> > > >> > > >> >> > > Jira/repository.
> >> > > >> > > >> >> > > > Packaging related code usually goes there.
> (That's
> >> > what
> >> > > >> all
> >> > > >> > the
> >> > > >> > > >> >> > > components
> >> > > >> > > >> >> > > > in bigtop does).
> >> > > >> > > >> >> > > > Having said that, once a PR is created, I'll be
> >> > sharing
> >> > > >> the
> >> > > >> > > link
> >> > > >> > > >> of
> >> > > >> > > >> >> PR
> >> > > >> > > >> >> > > > here, so that apex community as well can review
> it.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > Thanks,
> >> > > >> > > >> >> > > > Chinmay.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha
> >> Thombare <
> >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> >> > > >> > > >> >> > > > > We may need to iron out details about exact
> paths
> >> > etc.
> >> > > >> > > >> >> > > > > Which can be done on jira / PR comments.
> >> > > >> > > >> >> > > > > Is that right @dev?
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> >> > > >> > > >> >> > apache@pradeepdalvi.com>
> >> > > >> > > >> >> > > > > wrote:
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > > > > Inline comments...
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
> >> > Kolhatkar <
> >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> >> > > >> > > >> >> > > > > > wrote:
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present in DT
> >> community
> >> > > >> > edition
> >> > > >> > > >> will
> >> > > >> > > >> >> be
> >> > > >> > > >> >> > > > > included
> >> > > >> > > >> >> > > > > > > there.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > Here is the list of jars I found from
> common
> >> > > >> > dependencies
> >> > > >> > > >> >> between
> >> > > >> > > >> >> > > of
> >> > > >> > > >> >> > > > > apex
> >> > > >> > > >> >> > > > > > > and DT Community edition:
> >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> >> > > >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> >> > > >> > > >> >> > > > > > > jline-2.11.jar
> >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some tests do
> check
> >> > that
> >> > > >> > dtcli
> >> > > >> > > >> works
> >> > > >> > > >> >> > > > properly
> >> > > >> > > >> >> > > > > > for
> >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with only
> these
> >> > > >> libraries
> >> > > >> > > >> >> present in
> >> > > >> > > >> >> > > > > > isolation
> >> > > >> > > >> >> > > > > > > without dependency on local m2.
> >> > > >> > > >> >> > > > > > > I'm believe that there are unwanted jars
> >> which
> >> > are
> >> > > >> used
> >> > > >> > > for
> >> > > >> > > >> >> > compile
> >> > > >> > > >> >> > > > > time
> >> > > >> > > >> >> > > > > > > dependency and not runtime in above list
> >> which I
> >> > > can
> >> > > >> > drop
> >> > > >> > > >> to
> >> > > >> > > >> >> keep
> >> > > >> > > >> >> > > > > package
> >> > > >> > > >> >> > > > > > > size minimal.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > @Anniruddha,
> >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
> >> > > >> dt-env.sh is
> >> > > >> > > >> >> required.
> >> > > >> > > >> >> > > > Only
> >> > > >> > > >> >> > > > > > > required values I see in it are classpath.
> >> and
> >> > > >> dtcli is
> >> > > >> > > >> >> capable
> >> > > >> > > >> >> > of
> >> > > >> > > >> >> > > > > > building
> >> > > >> > > >> >> > > > > > > it on the fly.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see
> how
> >> > dtcli
> >> > > >> can
> >> > > >> > > find
> >> > > >> > > >> >> the
> >> > > >> > > >> >> > jar
> >> > > >> > > >> >> > > > > path,
> >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf files. If we
> >> need a
> >> > > >> change
> >> > > >> > in
> >> > > >> > > >> >> dtcli,
> >> > > >> > > >> >> > > then
> >> > > >> > > >> >> > > > > > > community's opinion is required for whether
> >> > dtcli
> >> > > >> > should
> >> > > >> > > >> >> change
> >> > > >> > > >> >> > in
> >> > > >> > > >> >> > > > our
> >> > > >> > > >> >> > > > > > repo
> >> > > >> > > >> >> > > > > > > or a copy of that with required changes
> >> exist in
> >> > > >> bigtop
> >> > > >> > > >> repo
> >> > > >> > > >> >> > until
> >> > > >> > > >> >> > > we
> >> > > >> > > >> >> > > > > > make
> >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > Can we have these file paths set using
> >> environment
> >> > > >> > > variables
> >> > > >> > > >> >> with
> >> > > >> > > >> >> > > some
> >> > > >> > > >> >> > > > > > default values in dtcli?
> >> > > >> > > >> >> > > > > > Then we can set such params in dt-env.sh.
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file
> different
> >> > from
> >> > > >> > > >> >> dt-site.xml?
> >> > > >> > > >> >> > Not
> >> > > >> > > >> >> > > > > sure
> >> > > >> > > >> >> > > > > > if
> >> > > >> > > >> >> > > > > > > adding in /etc/skel would help. Files &
> Dirs
> >> > from
> >> > > >> > > /etc/skel
> >> > > >> > > >> >> are
> >> > > >> > > >> >> > > > copied
> >> > > >> > > >> >> > > > > to
> >> > > >> > > >> >> > > > > > > home of new user when useradd program is
> >> called.
> >> > > But
> >> > > >> > for
> >> > > >> > > >> >> existing
> >> > > >> > > >> >> > > > users
> >> > > >> > > >> >> > > > > > > this won't be of any use. Moreover, dtcli
> >> > creates
> >> > > >> > ~/.dt/
> >> > > >> > > on
> >> > > >> > > >> >> the
> >> > > >> > > >> >> > fly
> >> > > >> > > >> >> > > > if
> >> > > >> > > >> >> > > > > > not
> >> > > >> > > >> >> > > > > > > encountered for the first time. Again
> >> correct me
> >> > > if
> >> > > >> I'm
> >> > > >> > > >> wrong.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > Yes, you are right. However this would
> >> probably be
> >> > > >> > > necessary
> >> > > >> > > >> >> step
> >> > > >> > > >> >> > for
> >> > > >> > > >> >> > > > > > rpm/deb, as they may also get installed
> during
> >> OS
> >> > > >> install
> >> > > >> > > and
> >> > > >> > > >> >> > before
> >> > > >> > > >> >> > > > user
> >> > > >> > > >> >> > > > > > accounts were created.
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm
> not
> >> > sure
> >> > > >> its
> >> > > >> > the
> >> > > >> > > >> best
> >> > > >> > > >> >> > > idea.
> >> > > >> > > >> >> > > > > > We'll
> >> > > >> > > >> >> > > > > > > anyway put dtcli in location which is by
> >> default
> >> > > >> > present
> >> > > >> > > in
> >> > > >> > > >> >> path
> >> > > >> > > >> >> > > i.e.
> >> > > >> > > >> >> > > > > > > /usr/bin.
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > For other env variables specific to apex,
> I'll
> >> > > prefer
> >> > > >> to
> >> > > >> > > use
> >> > > >> > > >> >> > > dt-env.sh
> >> > > >> > > >> >> > > > > and
> >> > > >> > > >> >> > > > > > > source it in dtcli rather than changing
> >> bashrc
> >> > > >> etc...
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/
> >> shall
> >> > > be
> >> > > >> the
> >> > > >> > > >> >> cleaner
> >> > > >> > > >> >> > way
> >> > > >> > > >> >> > > > to
> >> > > >> > > >> >> > > > > > achieve the same.
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > Thanks,
> >> > > >> > > >> >> > > > > > > Chinmay.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
> >> > > Thombare
> >> > > >> <
> >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > > Hi,
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > @Chinmay,
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > We need to consider following:
> >> > > >> > > >> >> > > > > > > > System wide default config files can be
> >> > located
> >> > > at
> >> > > >> > > >> following
> >> > > >> > > >> >> > > > > locations:
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to
> change
> >> > > dtcli
> >> > > >> > > >> behaviour
> >> > > >> > > >> >> on
> >> > > >> > > >> >> > > how
> >> > > >> > > >> >> > > > it
> >> > > >> > > >> >> > > > > > > finds
> >> > > >> > > >> >> > > > > > > > those locations)
> >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to
> >> change
> >> > > dtcli
> >> > > >> > > >> >> behaviour on
> >> > > >> > > >> >> > > how
> >> > > >> > > >> >> > > > > it
> >> > > >> > > >> >> > > > > > > > finds those locations)
> >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other
> files
> >> > (for
> >> > > >> new
> >> > > >> > > users
> >> > > >> > > >> >> that
> >> > > >> > > >> >> > > will
> >> > > >> > > >> >> > > > > be
> >> > > >> > > >> >> > > > > > > > created in system in future.)
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > We may also have to modify bashrc /
> >> > bashprofile
> >> > > >> for
> >> > > >> > > >> >> population
> >> > > >> > > >> >> > > any
> >> > > >> > > >> >> > > > > > > > variables that are required.
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > @dev, please put in your comments /
> >> > suggestion.
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > Thanks,
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > Aniruddha
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas
> >> Weise
> >> > <
> >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> >> > > >> > > >> >> > > > > > > > wrote:
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > > -->
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> >> > > >> Kolhatkar <
> >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > wrote:
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Continuing with packaging effort
> >> (rpm+deb)
> >> > > of
> >> > > >> > apex,
> >> > > >> > > >> here
> >> > > >> > > >> >> > are
> >> > > >> > > >> >> > > > some
> >> > > >> > > >> >> > > > > > > > > proposals
> >> > > >> > > >> >> > > > > > > > > > about package structure etc..
> >> > > >> > > >> >> > > > > > > > > > Before posting it on bbigtop mailing
> >> > list, I
> >> > > >> have
> >> > > >> > > >> some
> >> > > >> > > >> >> > > question
> >> > > >> > > >> >> > > > > for
> >> > > >> > > >> >> > > > > > > > apex
> >> > > >> > > >> >> > > > > > > > > > community.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Proposed Directory structure of apex
> >> > package
> >> > > >> for
> >> > > >> > > both
> >> > > >> > > >> >> deb &
> >> > > >> > > >> >> > > > rpm:
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> >> > > >> > > >> >> > > > > > > > > >
> >> /usr/lib/apex/lib/apex-api-<version>.jar
> >> > > >> > > >> >> > > > > > > > > >
> >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >> > > >> > > >> >> > > > > > > > > >
> >> > /usr/lib/apex/lib/apex-common-<version>.jar
> >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent
> >> jars>
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> /usr/lib/apex/bin/dtcli
> >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> >> > > >> > > >> >> > > > > > > > > >
> /usr/share/doc/apex/license/LICENSE.txt
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> >> > > >> > > >> >> > > > > > > > > > <other dependent jars> = All the 3rd
> >> party
> >> > > >> jars
> >> > > >> > > which
> >> > > >> > > >> >> are
> >> > > >> > > >> >> > > > > required
> >> > > >> > > >> >> > > > > > > for
> >> > > >> > > >> >> > > > > > > > > apex
> >> > > >> > > >> >> > > > > > > > > > to run. Usually the dependencies are
> >> > > packaged
> >> > > >> as
> >> > > >> > > >> part of
> >> > > >> > > >> >> > > > rpm/deb
> >> > > >> > > >> >> > > > > by
> >> > > >> > > >> >> > > > > > > any
> >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > Can you specify what those jars are in
> >> > Bigtop
> >> > > >> > > context?
> >> > > >> > > >> >> Same
> >> > > >> > > >> >> > as
> >> > > >> > > >> >> > > > > > shipped
> >> > > >> > > >> >> > > > > > > > with
> >> > > >> > > >> >> > > > > > > > > DT community addition under lib/ ?
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of
> >> > > corresponding
> >> > > >> of
> >> > > >> > > 3rd
> >> > > >> > > >> >> party
> >> > > >> > > >> >> > > jars
> >> > > >> > > >> >> > > > > > which
> >> > > >> > > >> >> > > > > > > > > needs
> >> > > >> > > >> >> > > > > > > > > > to included while packaging.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Questions related to this:
> >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli of apex as
> >> > "apex"
> >> > > >> > instead
> >> > > >> > > >> of
> >> > > >> > > >> >> > > "dtcli"
> >> > > >> > > >> >> > > > in
> >> > > >> > > >> >> > > > > > > > bigtop
> >> > > >> > > >> >> > > > > > > > > > package?
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > I think we should keep the name until
> we
> >> are
> >> > > >> able
> >> > > >> > to
> >> > > >> > > >> >> change
> >> > > >> > > >> >> > it
> >> > > >> > > >> >> > > in
> >> > > >> > > >> >> > > > > > Apex.
> >> > > >> > > >> >> > > > > > > > > Otherwise this may get confusing.
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares in bigtop
> >> have
> >> > > man
> >> > > >> > page
> >> > > >> > > >> for
> >> > > >> > > >> >> > their
> >> > > >> > > >> >> > > > > > > > > executables.
> >> > > >> > > >> >> > > > > > > > > > I think we should have it too for
> >> dtcli.
> >> > Is
> >> > > >> there
> >> > > >> > > any
> >> > > >> > > >> >> > > > > documentation
> >> > > >> > > >> >> > > > > > > > > which I
> >> > > >> > > >> >> > > > > > > > > > can convert to man page? or can I use
> >> > output
> >> > > >> of
> >> > > >> > > >> "dtcli
> >> > > >> > > >> >> > > --help"?
> >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call version of apex
> >> in
> >> > > >> Bigtop
> >> > > >> > as
> >> > > >> > > >> >> 3.3.0 OR
> >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of
> bigtop
> >> to
> >> > > >> depend
> >> > > >> > on
> >> > > >> > > >> >> 2.7.1
> >> > > >> > > >> >> > > > > version
> >> > > >> > > >> >> > > > > > of
> >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that we
> see
> >> > with
> >> > > >> this
> >> > > >> > > >> >> > dependency?
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > Should work without changed. I thought
> we
> >> > > >> certified
> >> > > >> > > >> >> against
> >> > > >> > > >> >> > > > 2.7.0?
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache mirror
> from
> >> > which
> >> > > >> > bigtop
> >> > > >> > > >> will
> >> > > >> > > >> >> > pick
> >> > > >> > > >> >> > > > the
> >> > > >> > > >> >> > > > > > > apex
> >> > > >> > > >> >> > > > > > > > > > source for compilation and packaging.
> >> > Please
> >> > > >> > > correct
> >> > > >> > > >> if
> >> > > >> > > >> >> > > > > incorrect:
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > >
> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >> > > >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the
> >> mirror
> >> > > >> used
> >> > > >> > by
> >> > > >> > > >> all
> >> > > >> > > >> >> the
> >> > > >> > > >> >> > > > > > softwares
> >> > > >> > > >> >> > > > > > > in
> >> > > >> > > >> >> > > > > > > > > > bigtop.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Please share your opinion.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Thanks,
> >> > > >> > > >> >> > > > > > > > > > Chinmay.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM,
> >> Chinmay
> >> > > >> > Kolhatkar
> >> > > >> > > <
> >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> >> > > >> > > >> >> > > > > > > > > > wrote:
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the
> >> packaging.
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > > I've already started discussion on
> >> > bigtop
> >> > > >> dev
> >> > > >> > > >> mailing
> >> > > >> > > >> >> > list
> >> > > >> > > >> >> > > > for
> >> > > >> > > >> >> > > > > > > > > > > integration. Also created a Jira
> for
> >> the
> >> > > >> same.
> >> > > >> > > For
> >> > > >> > > >> >> this
> >> > > >> > > >> >> > > > > > communities
> >> > > >> > > >> >> > > > > > > > > > > reference, here is the bigtop Jira:
> >> > > >> > BIGTOP-2313.
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM,
> >> Thomas
> >> > > >> Weise <
> >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > > wrote:
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> I don't see anything under
> >> > prerequisites
> >> > > >> that
> >> > > >> > > >> would
> >> > > >> > > >> >> be a
> >> > > >> > > >> >> > > > > > problem.
> >> > > >> > > >> >> > > > > > > We
> >> > > >> > > >> >> > > > > > > > > > >> looked
> >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing compatibility
> >> as
> >> > > part
> >> > > >> of
> >> > > >> > > >> >> becoming
> >> > > >> > > >> >> > an
> >> > > >> > > >> >> > > > > > > incubator
> >> > > >> > > >> >> > > > > > > > > > >> project.
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> Please focus on the packaging
> during
> >> > the
> >> > > >> next
> >> > > >> > > >> weeks.
> >> > > >> > > >> >> > Since
> >> > > >> > > >> >> > > > the
> >> > > >> > > >> >> > > > > > > work
> >> > > >> > > >> >> > > > > > > > > will
> >> > > >> > > >> >> > > > > > > > > > >> be
> >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related
> discussions
> >> and
> >> > > >> JIRAs
> >> > > >> > > >> should
> >> > > >> > > >> >> > also
> >> > > >> > > >> >> > > be
> >> > > >> > > >> >> > > > > > > there.
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> Would be good to have the
> packaging
> >> in
> >> > > >> place
> >> > > >> > by
> >> > > >> > > >> end
> >> > > >> > > >> >> Feb.
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> >> > > >> > > >> >> > > > > > > > > > >> Thomas
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM,
> >> > Chinmay
> >> > > >> > > >> Kolhatkar <
> >> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> >> > > >> > > >> >> > > > > > > > > > >> wrote:
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work on adding
> >> > Apache
> >> > > >> Apex
> >> > > >> > > as a
> >> > > >> > > >> >> > > component
> >> > > >> > > >> >> > > > > to
> >> > > >> > > >> >> > > > > > > > Apache
> >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging system
> for
> >> > the
> >> > > >> > Apache
> >> > > >> > > >> big
> >> > > >> > > >> >> data
> >> > > >> > > >> >> > > > > > > ecosystem.
> >> > > >> > > >> >> > > > > > > > > > >> Several
> >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most
> >> recently
> >> > > EMR.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira task
> in
> >> > > >> APEXCORE
> >> > > >> > for
> >> > > >> > > >> the
> >> > > >> > > >> >> > same:
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of execution is as
> >> > > follows:
> >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard and
> >> soft
> >> > > >> > > expectation
> >> > > >> > > >> >> for
> >> > > >> > > >> >> > new
> >> > > >> > > >> >> > > > > > > > components
> >> > > >> > > >> >> > > > > > > > > to
> >> > > >> > > >> >> > > > > > > > > > >> get
> >> > > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
> >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of it:
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> >
> >> > > >> > > >> >>
> >> > > >> > > >>
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to be
> standard
> >> ASF
> >> > > >> based
> >> > > >> > > >> >> > > requirements,
> >> > > >> > > >> >> > > > > but
> >> > > >> > > >> >> > > > > > > few
> >> > > >> > > >> >> > > > > > > > > need
> >> > > >> > > >> >> > > > > > > > > > >> to
> >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects are
> expected
> >> to
> >> > be
> >> > > >> > > Licensed
> >> > > >> > > >> >> under
> >> > > >> > > >> >> > > > > Apache
> >> > > >> > > >> >> > > > > > > > > License,
> >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
> >> dependencies
> >> > are
> >> > > >> > > expected
> >> > > >> > > >> >> to be
> >> > > >> > > >> >> > > > > > > compatible
> >> > > >> > > >> >> > > > > > > > > with
> >> > > >> > > >> >> > > > > > > > > > >> this
> >> > > >> > > >> >> > > > > > > > > > >> > license)
> >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but
> >> need
> >> > to
> >> > > >> > check
> >> > > >> > > if
> >> > > >> > > >> >> > > > > dependencies
> >> > > >> > > >> >> > > > > > of
> >> > > >> > > >> >> > > > > > > > > Apex
> >> > > >> > > >> >> > > > > > > > > > >> are
> >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I
> >> guess
> >> > > >> would
> >> > > >> > > be a
> >> > > >> > > >> >> > > > > verification
> >> > > >> > > >> >> > > > > > > > check.
> >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects are
> expected
> >> to
> >> > be
> >> > > >> > > >> compatible
> >> > > >> > > >> >> > with
> >> > > >> > > >> >> > > > all
> >> > > >> > > >> >> > > > > of
> >> > > >> > > >> >> > > > > > > the
> >> > > >> > > >> >> > > > > > > > > > >> > supported platforms that Bigtop
> >> > > >> distribution
> >> > > >> > > is
> >> > > >> > > >> >> > > targeting
> >> > > >> > > >> >> > > > > > > > > > >> >     - This needs verifying
> whether
> >> > our
> >> > > >> > > software
> >> > > >> > > >> >> runs
> >> > > >> > > >> >> > > fine
> >> > > >> > > >> >> > > > in
> >> > > >> > > >> >> > > > > > > > > centos-6
> >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
> >> > > debian-8
> >> > > >> > > >> >> > opensuse-13.2.
> >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that should
> be
> >> > > added
> >> > > >> for
> >> > > >> > > >> >> > deployment.
> >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test
> artifacts
> >> > which
> >> > > >> goes
> >> > > >> > > >> beyond
> >> > > >> > > >> >> > > smoke
> >> > > >> > > >> >> > > > > test
> >> > > >> > > >> >> > > > > > > > > > >> >     - These are basically the
> >> > > integration
> >> > > >> > > tests
> >> > > >> > > >> for
> >> > > >> > > >> >> > > > > > verification
> >> > > >> > > >> >> > > > > > > > > after
> >> > > >> > > >> >> > > > > > > > > > >> the
> >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a soft
> >> > requirement,
> >> > > >> but
> >> > > >> > > aim
> >> > > >> > > >> is
> >> > > >> > > >> >> to
> >> > > >> > > >> >> > > > > achieve
> >> > > >> > > >> >> > > > > > > this
> >> > > >> > > >> >> > > > > > > > > as
> >> > > >> > > >> >> > > > > > > > > > >> well
> >> > > >> > > >> >> > > > > > > > > > >> > or at least have explanation why
> >> not
> >> > to
> >> > > >> > > include.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > If there are any from the link
> >> which
> >> > > >> > > explicitly
> >> > > >> > > >> >> needs
> >> > > >> > > >> >> > to
> >> > > >> > > >> >> > > > be
> >> > > >> > > >> >> > > > > > > > checked
> >> > > >> > > >> >> > > > > > > > > > >> other
> >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please let us
> know.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as
> component
> >> to
> >> > > >> Bigtop*
> >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail archive of
> >> > Bigtop,
> >> > > >> it
> >> > > >> > was
> >> > > >> > > >> >> learnt
> >> > > >> > > >> >> > > that
> >> > > >> > > >> >> > > > > the
> >> > > >> > > >> >> > > > > > > > > bigtop
> >> > > >> > > >> >> > > > > > > > > > >> > community want to see the
> >> addition of
> >> > > new
> >> > > >> > > >> >> components
> >> > > >> > > >> >> > in
> >> > > >> > > >> >> > > > > > phases.
> >> > > >> > > >> >> > > > > > > > Here
> >> > > >> > > >> >> > > > > > > > > > are
> >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> >> > > >> > > >> >> > > > > > > > > > >> >     - This needs creating of
> >> package
> >> > > i.e.
> >> > > >> > rpm
> >> > > >> > > &
> >> > > >> > > >> deb
> >> > > >> > > >> >> > > files.
> >> > > >> > > >> >> > > > > > > > > > >> >     - documentations/READMEs,
> >> > LICENSE,
> >> > > >> > > >> DISCLAMER,
> >> > > >> > > >> >> > NOTES
> >> > > >> > > >> >> > > > etc
> >> > > >> > > >> >> > > > > if
> >> > > >> > > >> >> > > > > > > any
> >> > > >> > > >> >> > > > > > > > > > >> needed.
> >> > > >> > > >> >> > > > > > > > > > >> >     - Any documentation that
> need
> >> to
> >> > be
> >> > > >> > added
> >> > > >> > > to
> >> > > >> > > >> >> > > > > distribution
> >> > > >> > > >> >> > > > > > of
> >> > > >> > > >> >> > > > > > > > our
> >> > > >> > > >> >> > > > > > > > > > >> > software.
> >> > > >> > > >> >> > > > > > > > > > >> >     - Any license information of
> >> > > >> > dependencies
> >> > > >> > > >> >> required
> >> > > >> > > >> >> > > to
> >> > > >> > > >> >> > > > be
> >> > > >> > > >> >> > > > > > > added
> >> > > >> > > >> >> > > > > > > > > to
> >> > > >> > > >> >> > > > > > > > > > >> > package
> >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke test for
> >> > packaging.
> >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic
> >> > > >> deployment
> >> > > >> > and
> >> > > >> > > >> >> > > > configuration
> >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for
> >> > automatic
> >> > > >> > > >> deployment
> >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> >> > > >> > > >> >> > > > > > > > > > >> >     - For verification of
> >> > deployments.
> >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make
> sure
> >> all
> >> > > new
> >> > > >> > > stuff
> >> > > >> > > >> is
> >> > > >> > > >> >> > > > compliant
> >> > > >> > > >> >> > > > > > > with
> >> > > >> > > >> >> > > > > > > > > ASF
> >> > > >> > > >> >> > > > > > > > > > >> > license requirements. If you add
> >> code
> >> > > >> > licenses
> >> > > >> > > >> >> under
> >> > > >> > > >> >> > > > > different
> >> > > >> > > >> >> > > > > > > > > > licenses,
> >> > > >> > > >> >> > > > > > > > > > >> > those would need to be listed in
> >> the
> >> > > >> NOTICE.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > Please share your thoughts on
> the
> >> > > >> approach.
> >> > > >> > > >> >> > > > > > > > > > >> > We'll start corresponding
> >> > communication
> >> > > >> on
> >> > > >> > > >> bigtop
> >> > > >> > > >> >> > > mailing
> >> > > >> > > >> >> > > > > list
> >> > > >> > > >> >> > > > > > > as
> >> > > >> > > >> >> > > > > > > > > > well.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > We have some specific
> >> > > >> questions/suggestions
> >> > > >> > > >> >> related to
> >> > > >> > > >> >> > > > what
> >> > > >> > > >> >> > > > > > > should
> >> > > >> > > >> >> > > > > > > > > be
> >> > > >> > > >> >> > > > > > > > > > >> the
> >> > > >> > > >> >> > > > > > > > > > >> > content of the package and what
> >> > should
> >> > > be
> >> > > >> > the
> >> > > >> > > >> smoke
> >> > > >> > > >> >> > > tests,
> >> > > >> > > >> >> > > > > but
> >> > > >> > > >> >> > > > > > > in
> >> > > >> > > >> >> > > > > > > > > the
> >> > > >> > > >> >> > > > > > > > > > >> > interest of not having too much
> >> > content
> >> > > >> > here,
> >> > > >> > > >> we'll
> >> > > >> > > >> >> > put
> >> > > >> > > >> >> > > > the
> >> > > >> > > >> >> > > > > > > > > questions
> >> > > >> > > >> >> > > > > > > > > > >> as a
> >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this
> mailthread.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> >
> >> > > >> > > >> >>
> >> > > >> > > >> >
> >> > > >> > > >> >
> >> > > >> > > >>
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

I got a recommendation from bigtop community to drop "-incubating" from
version.

Here is the mail thread:
http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E

Please let me know if removing "-incubating" is fine.

Thanks,
Chinmay.


On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> I've already started communication there.
>
> On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
>> This is a question you should ask on the Bigtop mailing list.
>>
>> --
>> sent from mobile
>> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org> wrote:
>>
>> > Yes. In case of RPM, the only restriction on version is that it can not
>> > contain '-'.
>> > However, as a workaround, one can use 'incubating' as a part of
>> %{release}.
>> >
>> > Version: 3.3.0
>> > Release: incubating
>> >
>> > apex-3.3.0-incubating.noarch.rpm
>> >
>> > It is the same practice followed for pre-release packages e.g. with rcX,
>> > alphaX, betaX being used in %{release}.
>> >
>> > Regards,
>> > Pradeep A. Dalvi
>> >
>> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <ch...@apache.org>
>> > wrote:
>> >
>> > > Hi Everyone,
>> > >
>> > > In one of the previous mails in this thread, we decided that version
>> of
>> > the
>> > > package will be "3.3.0-incubating".
>> > > The problem that I'm facing is in RPM base version "-" is not allowed.
>> > >
>> > > Can we have package version to "3.3.0"?
>> > > i.e. the packages created will be as follows:
>> > > For RPM: apex-3.3.0-1.noarch.rpm
>> > > For DEB: apex_3.3.0-1_all.deb
>> > >
>> > > There is another integration for Zepplin in Bigtop which is incubating
>> > and
>> > > is having base version as "0.5.6".
>> > >
>> > > Please let me know your thoughts.
>> > >
>> > > Thanks,
>> > > Chinmay.
>> > >
>> > >
>> > >
>> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
>> chinmay@apache.org>
>> > > wrote:
>> > >
>> > > > Thanks everyone for the feedback.
>> > > >
>> > > > Also as Ram and David mentioned, I'll start creating the man page as
>> > > well.
>> > > >
>> > > > Thanks,
>> > > > Chinmay.
>> > > >
>> > > >
>> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
>> thomas@datatorrent.com>
>> > > > wrote:
>> > > >
>> > > >> I think we already agreed that we will retain the current script
>> name
>> > in
>> > > >> this bigtop iteration. There is a ticket for renaming, add you
>> > > suggestions
>> > > >> there.
>> > > >>
>> > > >> --
>> > > >> sent from mobile
>> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <sa...@datatorrent.com>
>> > > wrote:
>> > > >>
>> > > >> > +1 for Symlink from apex to dtcli or start the transition
>> process by
>> > > >> making
>> > > >> > dtcli as the symlink to apex cli.
>> > > >> >
>> > > >> > Renaming is a breaking change should be done during major version
>> > > number
>> > > >> > change.
>> > > >> >
>> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <david@datatorrent.com
>> >
>> > > >> wrote:
>> > > >> >
>> > > >> > > Apologies for hitting the "Send" button too early.
>> > > >> > >
>> > > >> > > What is the scope of the name change?  If we simply have a
>> symlink
>> > > >> from
>> > > >> > > apex to dtcli, then it should have no backward compatibility
>> > problem
>> > > >> at
>> > > >> > > all.
>> > > >> > >
>> > > >> > > But the proper name change would involve:
>> > > >> > > - Change the name of DTCli class to ApexCli
>> > > >> > > - Change the prompt from dt> to apex>
>> > > >> > > - Change the initial greeting when running the CLI from
>> something
>> > > like
>> > > >> > "DT
>> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
>> > > >> > >
>> > > >> > > The above changes may break existing scripts that rely on the
>> the
>> > > >> string
>> > > >> > > "DT".
>> > > >> > > But if folks are okay with this potential problem, I am okay
>> with
>> > > the
>> > > >> > name
>> > > >> > > change.
>> > > >> > >
>> > > >> > > For the man page, we can start with the content of this page:
>> > > >> > >
>> > > >> > > http://docs.datatorrent.com/dtcli/
>> > > >> > >
>> > > >> > > Plus the command line options when doing dtcli -h:
>> > > >> > >
>> > > >> > > usage: DTCli
>> > > >> > >  -e <arg>    Commands are read from the argument
>> > > >> > >  -f <arg>    Use the specified prompt at all time
>> > > >> > >  -h          Print this help
>> > > >> > >  -kp <arg>   Use the specified kerberos principal
>> > > >> > >  -kt <arg>   Use the specified kerberos keytab
>> > > >> > >  -p <arg>    JSONP padding function
>> > > >> > >  -r          JSON Raw mode
>> > > >> > >  -v          Verbose mode level 1
>> > > >> > >  -vv         Verbose mode level 2
>> > > >> > >  -vvv        Verbose mode level 3
>> > > >> > >  -vvvv       Verbose mode level 4
>> > > >> > >
>> > > >> > > David
>> > > >> > >
>> > > >> > >
>> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
>> david@datatorrent.com
>> > >
>> > > >> > wrote:
>> > > >> > >
>> > > >> > > > What is the scope of the name change?  If we simply have a
>> > symlink
>> > > >> from
>> > > >> > > > apex to dtcli, then it should have no backward compatibility
>> > > >> problem at
>> > > >> > > all.
>> > > >> > > >
>> > > >> > > > But the proper name change would involve:
>> > > >> > > > - Change the name of DTCli class to ApexCli
>> > > >> > > > - Change the prompt from dt> to apex>
>> > > >> > > > - Change the initial greeting when running the CLI from
>> > something
>> > > >> like
>> > > >> > > "DT
>> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
>> > > >> > > >
>> > > >> > > > The above changes may break existing scripts that rely on the
>> > the
>> > > >> > string
>> > > >> > > > "DT".
>> > > >> > > > But if folks are okay with this potential problem, I am okay
>> > with
>> > > >> the
>> > > >> > > name
>> > > >> > > > change.
>> > > >> > > >
>> > > >> > > > For the man page, we can start with the content of this page:
>> > > >> > > >
>> > > >> > > >
>> > > >> > > >
>> > > >> > > >
>> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
>> > > >> chinmay@apache.org
>> > > >> > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > >> Here is a suggestion I have related to name change:
>> > > >> > > >>
>> > > >> > > >> 1. While packaging we change the name of "dtcli" script to
>> > > "apex".
>> > > >> > This
>> > > >> > > is
>> > > >> > > >> done so that in future when we change the name to "apex",
>> users
>> > > of
>> > > >> > > bigtop
>> > > >> > > >> apex don't have to transition much.
>> > > >> > > >> 2. We also keep a symlink named dtcli which points to apex
>> > > script.
>> > > >> > This
>> > > >> > > is
>> > > >> > > >> for backward compatibility.
>> > > >> > > >> 3. We still keep conf/dt-env.sh for first integration in
>> > bigtop.
>> > > >> This
>> > > >> > is
>> > > >> > > >> in
>> > > >> > > >> the interest of not changing the content of dtcli or apex
>> > script.
>> > > >> > > >> 4. In apex 3.4.0, we take care of all these naming related
>> > > changes
>> > > >> and
>> > > >> > > >> update bigtop repository later to remove references to dtcli
>> > all
>> > > >> > > together.
>> > > >> > > >>
>> > > >> > > >> Please share your thoughts on above approach.
>> > > >> > > >>
>> > > >> > > >> Also, please share what could be the man page content for
>> > "apex".
>> > > >> > > >>
>> > > >> > > >> Thanks,
>> > > >> > > >> Chinmay.
>> > > >> > > >>
>> > > >> > > >>
>> > > >> > > >>
>> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
>> > > >> > chinmay@apache.org>
>> > > >> > > >> wrote:
>> > > >> > > >>
>> > > >> > > >> > Yes.. It'll be build from source tar downloaded from one
>> of
>> > the
>> > > >> > apache
>> > > >> > > >> > mirror: apache.osuosl.org
>> > > >> > > >> >
>> > > >> > > >> > The reason for dt-env.sh is it's sourced from dtcli. This
>> > > leaves
>> > > >> us
>> > > >> > 3
>> > > >> > > >> > options:
>> > > >> > > >> > If we should not have any dt-env.sh, dtcli would need a
>> > change
>> > > >> while
>> > > >> > > >> > packaging.
>> > > >> > > >> >
>> > > >> > > >> > But if we want to use dtcli as it is, then we would need
>> > > >> dt-env.sh
>> > > >> > > >> >
>> > > >> > > >> >
>> > > >> > > >> >
>> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
>> > > >> > thomas@datatorrent.com
>> > > >> > > >
>> > > >> > > >> > wrote:
>> > > >> > > >> >
>> > > >> > > >> >> Looks good overall, though there shouldn't be any
>> dt-env.sh
>> > > >> > > >> >>
>> > > >> > > >> >> What will the bigtop package be built from, the source
>> tar?
>> > > >> > > >> >>
>> > > >> > > >> >>
>> > > >> > > >> >>
>> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
>> > > >> > > chinmay@apache.org
>> > > >> > > >> >
>> > > >> > > >> >> wrote:
>> > > >> > > >> >>
>> > > >> > > >> >> > Hi Thomas,
>> > > >> > > >> >> >
>> > > >> > > >> >> > Thanks for the feedback.
>> > > >> > > >> >> >
>> > > >> > > >> >> > First, we're not changing the name anywhere. We'll
>> follow
>> > > what
>> > > >> > > >> >> currently is
>> > > >> > > >> >> > in source tarball for 3.3.0-incubating version.
>> > > >> > > >> >> >
>> > > >> > > >> >> > Secondly, I've mentioned a directory structure below
>> which
>> > > is
>> > > >> > > inline
>> > > >> > > >> >> with
>> > > >> > > >> >> > other existing integrations with bigtop.
>> > > >> > > >> >> >
>> > > >> > > >> >> > The need for each file is as follows:
>> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
>> > > >> engine/src/main/scripts/
>> > > >> > > of
>> > > >> > > >> >> source
>> > > >> > > >> >> > code apex 3.3.0-incubating.
>> > > >> > > >> >> >
>> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This
>> > > searches
>> > > >> for
>> > > >> > > >> hadoop
>> > > >> > > >> >> > binary path and exports an env variable for dtcli to
>> use
>> > it.
>> > > >> > > >> >> >
>> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is extracted
>> in a
>> > > >> similar
>> > > >> > > >> way to
>> > > >> > > >> >> > how DT Community Edition finds dependency jars.
>> > > >> > > >> >> > Please note that this is the first iteration list of
>> jars.
>> > > I'm
>> > > >> > > >> trying to
>> > > >> > > >> >> > narrow this down to only those which are really
>> required.
>> > > >> > > >> >> > The test that I'm running to check if dtcli runs fine
>> with
>> > > >> given
>> > > >> > > >> >> dependency
>> > > >> > > >> >> > is to launch+shutdown+kill+status for pi demo.
>> > > >> > > >> >> >
>> > > >> > > >> >> > Above is the directory structure which is required for
>> CLI
>> > > to
>> > > >> > work.
>> > > >> > > >> >> > What aniruddha and pradeep has mentioned are some
>> > additional
>> > > >> > files
>> > > >> > > >> which
>> > > >> > > >> >> > makes the like of administrator easier. For eg.
>> /etc/skel
>> > > >> > > >> >> >
>> > > >> > > >> >> > Please let me know if this is inline with what you're
>> > > >> thinking.
>> > > >> > > >> >> >
>> > > >> > > >> >> > Thanks,
>> > > >> > > >> >> > Chinmay.
>> > > >> > > >> >> >
>> > > >> > > >> >> >
>> > > >> > > >> >> > .
>> > > >> > > >> >> > |-- bin
>> > > >> > > >> >> > |   `-- dtcli
>> > > >> > > >> >> > |-- conf
>> > > >> > > >> >> > |   `-- dt-env.sh
>> > > >> > > >> >> > `-- lib
>> > > >> > > >> >> >     |-- ant-1.9.2.jar
>> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
>> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
>> > > >> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
>> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
>> > > >> > > >> >> >     |-- apex-engine.jar
>> > > >> > > >> >> >     |-- apex-engine-tests.jar
>> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
>> > > >> > > >> >> >     |-- bval-core-0.5.jar
>> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
>> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
>> > > >> > > >> >> >     |-- commons-codec-1.10.jar
>> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
>> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
>> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
>> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
>> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
>> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
>> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
>> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
>> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
>> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
>> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
>> > > >> > > >> >> >     |-- jersey-client-1.9.jar
>> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
>> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
>> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
>> > > >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
>> > > >> > > >> >> >     |-- jline-2.11.jar
>> > > >> > > >> >> >     |-- kryo-2.24.0.jar
>> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
>> > > >> > > >> >> >     |-- minlog-1.2.jar
>> > > >> > > >> >> >     |-- netlet-1.2.0.jar
>> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
>> > > >> > > >> >> >     |-- objenesis-2.1.jar
>> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
>> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
>> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
>> > > >> > > >> >> >
>> > > >> > > >> >> > 3 directories, 40 files
>> > > >> > > >> >> >
>> > > >> > > >> >> >
>> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
>> > > >> > > >> thomas@datatorrent.com>
>> > > >> > > >> >> > wrote:
>> > > >> > > >> >> >
>> > > >> > > >> >> > > Chinmay,
>> > > >> > > >> >> > >
>> > > >> > > >> >> > > Before discussing where to put the files, let's make
>> > sure
>> > > >> they
>> > > >> > > are
>> > > >> > > >> >> really
>> > > >> > > >> >> > > needed for the operation of the CLI. As for names,
>> > > anything
>> > > >> new
>> > > >> > > >> needs
>> > > >> > > >> >> to
>> > > >> > > >> >> > > reflect Apex in the name and should follow common
>> > > >> conventions,
>> > > >> > > >> >> especially
>> > > >> > > >> >> > > in Bigtop where there are many existing integrations
>> to
>> > > look
>> > > >> > at.
>> > > >> > > >> >> > >
>> > > >> > > >> >> > > Thanks,
>> > > >> > > >> >> > > Thomas
>> > > >> > > >> >> > >
>> > > >> > > >> >> > >
>> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
>> > > >> > > >> >> > > chinmay@datatorrent.com>
>> > > >> > > >> >> > > wrote:
>> > > >> > > >> >> > >
>> > > >> > > >> >> > > > Really good points Pradeep and Aniruddha.
>> > > >> > > >> >> > > >
>> > > >> > > >> >> > > > 1. I believe we won't need to change the dtcli
>> > > >> considering it
>> > > >> > > >> works
>> > > >> > > >> >> > with
>> > > >> > > >> >> > > DT
>> > > >> > > >> >> > > > Community edition. We can keep the directory
>> structure
>> > > >> > similar
>> > > >> > > to
>> > > >> > > >> >> that.
>> > > >> > > >> >> > > > dt-env.sh has variables which contains information
>> > > >> required
>> > > >> > for
>> > > >> > > >> >> dtcli
>> > > >> > > >> >> > to
>> > > >> > > >> >> > > > launch.
>> > > >> > > >> >> > > >
>> > > >> > > >> >> > > > 2. Let me check with bigtop community that whether
>> > they
>> > > >> > > >> facilitate
>> > > >> > > >> >> the
>> > > >> > > >> >> > > > installation of rpms/debs before user is created.
>> In
>> > > >> either
>> > > >> > > case,
>> > > >> > > >> >> > current
>> > > >> > > >> >> > > > dtcli creates a .dt folder in home directory. Also
>> > > before
>> > > >> > > putting
>> > > >> > > >> >> > > anything
>> > > >> > > >> >> > > > in /etc/skel we need to define what are the default
>> > > >> contents
>> > > >> > > that
>> > > >> > > >> >> > should
>> > > >> > > >> >> > > go
>> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults,
>> probably we
>> > > >> should
>> > > >> > > not
>> > > >> > > >> >> > > > explicitly add it in /etc/skel.
>> > > >> > > >> >> > > >
>> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This way
>> > > contents
>> > > >> of
>> > > >> > > >> >> dt-env.sh
>> > > >> > > >> >> > > are
>> > > >> > > >> >> > > > present as env variables. But I see a catch there.
>> > > Adding
>> > > >> > > >> dt-env.sh
>> > > >> > > >> >> to
>> > > >> > > >> >> > > > /etc/profile.d would make all the variables
>> available
>> > at
>> > > >> > > runtime
>> > > >> > > >> all
>> > > >> > > >> >> > the
>> > > >> > > >> >> > > > time. I feel a little skeptical about that. Maybe a
>> > > >> possible
>> > > >> > > >> >> collision
>> > > >> > > >> >> > > can
>> > > >> > > >> >> > > > occur with other application vars.
>> > > >> > > >> >> > > > Moreover, current dtcli does source
>> > "../conf/dt-env.sh"
>> > > >> and
>> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway
>> > > >> available.
>> > > >> > > >> >> > > >
>> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is happening at
>> > > Bigtop
>> > > >> > > >> >> > > Jira/repository.
>> > > >> > > >> >> > > > Packaging related code usually goes there. (That's
>> > what
>> > > >> all
>> > > >> > the
>> > > >> > > >> >> > > components
>> > > >> > > >> >> > > > in bigtop does).
>> > > >> > > >> >> > > > Having said that, once a PR is created, I'll be
>> > sharing
>> > > >> the
>> > > >> > > link
>> > > >> > > >> of
>> > > >> > > >> >> PR
>> > > >> > > >> >> > > > here, so that apex community as well can review it.
>> > > >> > > >> >> > > >
>> > > >> > > >> >> > > > Thanks,
>> > > >> > > >> >> > > > Chinmay.
>> > > >> > > >> >> > > >
>> > > >> > > >> >> > > >
>> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha
>> Thombare <
>> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
>> > > >> > > >> >> > > >
>> > > >> > > >> >> > > > > +1 on suggestions and approach.
>> > > >> > > >> >> > > > > We may need to iron out details about exact paths
>> > etc.
>> > > >> > > >> >> > > > > Which can be done on jira / PR comments.
>> > > >> > > >> >> > > > > Is that right @dev?
>> > > >> > > >> >> > > > >
>> > > >> > > >> >> > > > >
>> > > >> > > >> >> > > > >
>> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
>> > > >> > > >> >> > apache@pradeepdalvi.com>
>> > > >> > > >> >> > > > > wrote:
>> > > >> > > >> >> > > > >
>> > > >> > > >> >> > > > > > Inline comments...
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
>> > Kolhatkar <
>> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
>> > > >> > > >> >> > > > > > wrote:
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present in DT
>> community
>> > > >> > edition
>> > > >> > > >> will
>> > > >> > > >> >> be
>> > > >> > > >> >> > > > > included
>> > > >> > > >> >> > > > > > > there.
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > > > Here is the list of jars I found from common
>> > > >> > dependencies
>> > > >> > > >> >> between
>> > > >> > > >> >> > > of
>> > > >> > > >> >> > > > > apex
>> > > >> > > >> >> > > > > > > and DT Community edition:
>> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
>> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
>> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
>> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
>> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
>> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
>> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
>> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
>> > > >> > > >> >> > > > > > > bval-core-0.5.jar
>> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
>> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
>> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
>> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
>> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
>> > > >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
>> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
>> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
>> > > >> > > >> >> > > > > > > jline-2.11.jar
>> > > >> > > >> >> > > > > > > ant-1.9.2.jar
>> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
>> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
>> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
>> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
>> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some tests do check
>> > that
>> > > >> > dtcli
>> > > >> > > >> works
>> > > >> > > >> >> > > > properly
>> > > >> > > >> >> > > > > > for
>> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with only these
>> > > >> libraries
>> > > >> > > >> >> present in
>> > > >> > > >> >> > > > > > isolation
>> > > >> > > >> >> > > > > > > without dependency on local m2.
>> > > >> > > >> >> > > > > > > I'm believe that there are unwanted jars
>> which
>> > are
>> > > >> used
>> > > >> > > for
>> > > >> > > >> >> > compile
>> > > >> > > >> >> > > > > time
>> > > >> > > >> >> > > > > > > dependency and not runtime in above list
>> which I
>> > > can
>> > > >> > drop
>> > > >> > > >> to
>> > > >> > > >> >> keep
>> > > >> > > >> >> > > > > package
>> > > >> > > >> >> > > > > > > size minimal.
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > > > @Anniruddha,
>> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
>> > > >> dt-env.sh is
>> > > >> > > >> >> required.
>> > > >> > > >> >> > > > Only
>> > > >> > > >> >> > > > > > > required values I see in it are classpath.
>> and
>> > > >> dtcli is
>> > > >> > > >> >> capable
>> > > >> > > >> >> > of
>> > > >> > > >> >> > > > > > building
>> > > >> > > >> >> > > > > > > it on the fly.
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how
>> > dtcli
>> > > >> can
>> > > >> > > find
>> > > >> > > >> >> the
>> > > >> > > >> >> > jar
>> > > >> > > >> >> > > > > path,
>> > > >> > > >> >> > > > > > > dt-env.sh and all such conf files. If we
>> need a
>> > > >> change
>> > > >> > in
>> > > >> > > >> >> dtcli,
>> > > >> > > >> >> > > then
>> > > >> > > >> >> > > > > > > community's opinion is required for whether
>> > dtcli
>> > > >> > should
>> > > >> > > >> >> change
>> > > >> > > >> >> > in
>> > > >> > > >> >> > > > our
>> > > >> > > >> >> > > > > > repo
>> > > >> > > >> >> > > > > > > or a copy of that with required changes
>> exist in
>> > > >> bigtop
>> > > >> > > >> repo
>> > > >> > > >> >> > until
>> > > >> > > >> >> > > we
>> > > >> > > >> >> > > > > > make
>> > > >> > > >> >> > > > > > > the dtcli generic enough.
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > > Can we have these file paths set using
>> environment
>> > > >> > > variables
>> > > >> > > >> >> with
>> > > >> > > >> >> > > some
>> > > >> > > >> >> > > > > > default values in dtcli?
>> > > >> > > >> >> > > > > > Then we can set such params in dt-env.sh.
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file different
>> > from
>> > > >> > > >> >> dt-site.xml?
>> > > >> > > >> >> > Not
>> > > >> > > >> >> > > > > sure
>> > > >> > > >> >> > > > > > if
>> > > >> > > >> >> > > > > > > adding in /etc/skel would help. Files & Dirs
>> > from
>> > > >> > > /etc/skel
>> > > >> > > >> >> are
>> > > >> > > >> >> > > > copied
>> > > >> > > >> >> > > > > to
>> > > >> > > >> >> > > > > > > home of new user when useradd program is
>> called.
>> > > But
>> > > >> > for
>> > > >> > > >> >> existing
>> > > >> > > >> >> > > > users
>> > > >> > > >> >> > > > > > > this won't be of any use. Moreover, dtcli
>> > creates
>> > > >> > ~/.dt/
>> > > >> > > on
>> > > >> > > >> >> the
>> > > >> > > >> >> > fly
>> > > >> > > >> >> > > > if
>> > > >> > > >> >> > > > > > not
>> > > >> > > >> >> > > > > > > encountered for the first time. Again
>> correct me
>> > > if
>> > > >> I'm
>> > > >> > > >> wrong.
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > > Yes, you are right. However this would
>> probably be
>> > > >> > > necessary
>> > > >> > > >> >> step
>> > > >> > > >> >> > for
>> > > >> > > >> >> > > > > > rpm/deb, as they may also get installed during
>> OS
>> > > >> install
>> > > >> > > and
>> > > >> > > >> >> > before
>> > > >> > > >> >> > > > user
>> > > >> > > >> >> > > > > > accounts were created.
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not
>> > sure
>> > > >> its
>> > > >> > the
>> > > >> > > >> best
>> > > >> > > >> >> > > idea.
>> > > >> > > >> >> > > > > > We'll
>> > > >> > > >> >> > > > > > > anyway put dtcli in location which is by
>> default
>> > > >> > present
>> > > >> > > in
>> > > >> > > >> >> path
>> > > >> > > >> >> > > i.e.
>> > > >> > > >> >> > > > > > > /usr/bin.
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > > For other env variables specific to apex, I'll
>> > > prefer
>> > > >> to
>> > > >> > > use
>> > > >> > > >> >> > > dt-env.sh
>> > > >> > > >> >> > > > > and
>> > > >> > > >> >> > > > > > > source it in dtcli rather than changing
>> bashrc
>> > > >> etc...
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/
>> shall
>> > > be
>> > > >> the
>> > > >> > > >> >> cleaner
>> > > >> > > >> >> > way
>> > > >> > > >> >> > > > to
>> > > >> > > >> >> > > > > > achieve the same.
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > > Thanks,
>> > > >> > > >> >> > > > > > > Chinmay.
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
>> > > Thombare
>> > > >> <
>> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > > > > Hi,
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > > @Chinmay,
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > > We need to consider following:
>> > > >> > > >> >> > > > > > > > System wide default config files can be
>> > located
>> > > at
>> > > >> > > >> following
>> > > >> > > >> >> > > > > locations:
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change
>> > > dtcli
>> > > >> > > >> behaviour
>> > > >> > > >> >> on
>> > > >> > > >> >> > > how
>> > > >> > > >> >> > > > it
>> > > >> > > >> >> > > > > > > finds
>> > > >> > > >> >> > > > > > > > those locations)
>> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to
>> change
>> > > dtcli
>> > > >> > > >> >> behaviour on
>> > > >> > > >> >> > > how
>> > > >> > > >> >> > > > > it
>> > > >> > > >> >> > > > > > > > finds those locations)
>> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files
>> > (for
>> > > >> new
>> > > >> > > users
>> > > >> > > >> >> that
>> > > >> > > >> >> > > will
>> > > >> > > >> >> > > > > be
>> > > >> > > >> >> > > > > > > > created in system in future.)
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > > We may also have to modify bashrc /
>> > bashprofile
>> > > >> for
>> > > >> > > >> >> population
>> > > >> > > >> >> > > any
>> > > >> > > >> >> > > > > > > > variables that are required.
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > > @dev, please put in your comments /
>> > suggestion.
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > > Thanks,
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > > Aniruddha
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas
>> Weise
>> > <
>> > > >> > > >> >> > > > > thomas@datatorrent.com>
>> > > >> > > >> >> > > > > > > > wrote:
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > > > > -->
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
>> > > >> Kolhatkar <
>> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > wrote:
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > Continuing with packaging effort
>> (rpm+deb)
>> > > of
>> > > >> > apex,
>> > > >> > > >> here
>> > > >> > > >> >> > are
>> > > >> > > >> >> > > > some
>> > > >> > > >> >> > > > > > > > > proposals
>> > > >> > > >> >> > > > > > > > > > about package structure etc..
>> > > >> > > >> >> > > > > > > > > > Before posting it on bbigtop mailing
>> > list, I
>> > > >> have
>> > > >> > > >> some
>> > > >> > > >> >> > > question
>> > > >> > > >> >> > > > > for
>> > > >> > > >> >> > > > > > > > apex
>> > > >> > > >> >> > > > > > > > > > community.
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > Proposed Directory structure of apex
>> > package
>> > > >> for
>> > > >> > > both
>> > > >> > > >> >> deb &
>> > > >> > > >> >> > > > rpm:
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
>> > > >> > > >> >> > > > > > > > > >
>> /usr/lib/apex/lib/apex-api-<version>.jar
>> > > >> > > >> >> > > > > > > > > >
>> > /usr/lib/apex/lib/apex-engine-<version>.jar
>> > > >> > > >> >> > > > > > > > > >
>> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
>> > > >> > > >> >> > > > > > > > > >
>> > /usr/lib/apex/lib/apex-common-<version>.jar
>> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent
>> jars>
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
>> > > >> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
>> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
>> > > >> > > >> >> > > > > > > > > >
>> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
>> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
>> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
>> > > >> > > >> >> > > > > > > > > > <other dependent jars> = All the 3rd
>> party
>> > > >> jars
>> > > >> > > which
>> > > >> > > >> >> are
>> > > >> > > >> >> > > > > required
>> > > >> > > >> >> > > > > > > for
>> > > >> > > >> >> > > > > > > > > apex
>> > > >> > > >> >> > > > > > > > > > to run. Usually the dependencies are
>> > > packaged
>> > > >> as
>> > > >> > > >> part of
>> > > >> > > >> >> > > > rpm/deb
>> > > >> > > >> >> > > > > by
>> > > >> > > >> >> > > > > > > any
>> > > >> > > >> >> > > > > > > > > > software in bigtop.
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > Can you specify what those jars are in
>> > Bigtop
>> > > >> > > context?
>> > > >> > > >> >> Same
>> > > >> > > >> >> > as
>> > > >> > > >> >> > > > > > shipped
>> > > >> > > >> >> > > > > > > > with
>> > > >> > > >> >> > > > > > > > > DT community addition under lib/ ?
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of
>> > > corresponding
>> > > >> of
>> > > >> > > 3rd
>> > > >> > > >> >> party
>> > > >> > > >> >> > > jars
>> > > >> > > >> >> > > > > > which
>> > > >> > > >> >> > > > > > > > > needs
>> > > >> > > >> >> > > > > > > > > > to included while packaging.
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > Questions related to this:
>> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli of apex as
>> > "apex"
>> > > >> > instead
>> > > >> > > >> of
>> > > >> > > >> >> > > "dtcli"
>> > > >> > > >> >> > > > in
>> > > >> > > >> >> > > > > > > > bigtop
>> > > >> > > >> >> > > > > > > > > > package?
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > I think we should keep the name until we
>> are
>> > > >> able
>> > > >> > to
>> > > >> > > >> >> change
>> > > >> > > >> >> > it
>> > > >> > > >> >> > > in
>> > > >> > > >> >> > > > > > Apex.
>> > > >> > > >> >> > > > > > > > > Otherwise this may get confusing.
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares in bigtop
>> have
>> > > man
>> > > >> > page
>> > > >> > > >> for
>> > > >> > > >> >> > their
>> > > >> > > >> >> > > > > > > > > executables.
>> > > >> > > >> >> > > > > > > > > > I think we should have it too for
>> dtcli.
>> > Is
>> > > >> there
>> > > >> > > any
>> > > >> > > >> >> > > > > documentation
>> > > >> > > >> >> > > > > > > > > which I
>> > > >> > > >> >> > > > > > > > > > can convert to man page? or can I use
>> > output
>> > > >> of
>> > > >> > > >> "dtcli
>> > > >> > > >> >> > > --help"?
>> > > >> > > >> >> > > > > > > > > > 3. Do we want to call version of apex
>> in
>> > > >> Bigtop
>> > > >> > as
>> > > >> > > >> >> 3.3.0 OR
>> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > Has to be -incubating
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of bigtop
>> to
>> > > >> depend
>> > > >> > on
>> > > >> > > >> >> 2.7.1
>> > > >> > > >> >> > > > > version
>> > > >> > > >> >> > > > > > of
>> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that we see
>> > with
>> > > >> this
>> > > >> > > >> >> > dependency?
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > Should work without changed. I thought we
>> > > >> certified
>> > > >> > > >> >> against
>> > > >> > > >> >> > > > 2.7.0?
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > 5. Following is the apache mirror from
>> > which
>> > > >> > bigtop
>> > > >> > > >> will
>> > > >> > > >> >> > pick
>> > > >> > > >> >> > > > the
>> > > >> > > >> >> > > > > > > apex
>> > > >> > > >> >> > > > > > > > > > source for compilation and packaging.
>> > Please
>> > > >> > > correct
>> > > >> > > >> if
>> > > >> > > >> >> > > > > incorrect:
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > >
>> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>> > > >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the
>> mirror
>> > > >> used
>> > > >> > by
>> > > >> > > >> all
>> > > >> > > >> >> the
>> > > >> > > >> >> > > > > > softwares
>> > > >> > > >> >> > > > > > > in
>> > > >> > > >> >> > > > > > > > > > bigtop.
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > Please share your opinion.
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > Thanks,
>> > > >> > > >> >> > > > > > > > > > Chinmay.
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM,
>> Chinmay
>> > > >> > Kolhatkar
>> > > >> > > <
>> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
>> > > >> > > >> >> > > > > > > > > > wrote:
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the
>> packaging.
>> > > >> > > >> >> > > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > > I've already started discussion on
>> > bigtop
>> > > >> dev
>> > > >> > > >> mailing
>> > > >> > > >> >> > list
>> > > >> > > >> >> > > > for
>> > > >> > > >> >> > > > > > > > > > > integration. Also created a Jira for
>> the
>> > > >> same.
>> > > >> > > For
>> > > >> > > >> >> this
>> > > >> > > >> >> > > > > > communities
>> > > >> > > >> >> > > > > > > > > > > reference, here is the bigtop Jira:
>> > > >> > BIGTOP-2313.
>> > > >> > > >> >> > > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM,
>> Thomas
>> > > >> Weise <
>> > > >> > > >> >> > > > > > > thomas@datatorrent.com
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > > wrote:
>> > > >> > > >> >> > > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > >> Chinmay,
>> > > >> > > >> >> > > > > > > > > > >>
>> > > >> > > >> >> > > > > > > > > > >> I don't see anything under
>> > prerequisites
>> > > >> that
>> > > >> > > >> would
>> > > >> > > >> >> be a
>> > > >> > > >> >> > > > > > problem.
>> > > >> > > >> >> > > > > > > We
>> > > >> > > >> >> > > > > > > > > > >> looked
>> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing compatibility
>> as
>> > > part
>> > > >> of
>> > > >> > > >> >> becoming
>> > > >> > > >> >> > an
>> > > >> > > >> >> > > > > > > incubator
>> > > >> > > >> >> > > > > > > > > > >> project.
>> > > >> > > >> >> > > > > > > > > > >>
>> > > >> > > >> >> > > > > > > > > > >> Please focus on the packaging during
>> > the
>> > > >> next
>> > > >> > > >> weeks.
>> > > >> > > >> >> > Since
>> > > >> > > >> >> > > > the
>> > > >> > > >> >> > > > > > > work
>> > > >> > > >> >> > > > > > > > > will
>> > > >> > > >> >> > > > > > > > > > >> be
>> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related discussions
>> and
>> > > >> JIRAs
>> > > >> > > >> should
>> > > >> > > >> >> > also
>> > > >> > > >> >> > > be
>> > > >> > > >> >> > > > > > > there.
>> > > >> > > >> >> > > > > > > > > > >>
>> > > >> > > >> >> > > > > > > > > > >> Would be good to have the packaging
>> in
>> > > >> place
>> > > >> > by
>> > > >> > > >> end
>> > > >> > > >> >> Feb.
>> > > >> > > >> >> > > > > > > > > > >>
>> > > >> > > >> >> > > > > > > > > > >> Thanks,
>> > > >> > > >> >> > > > > > > > > > >> Thomas
>> > > >> > > >> >> > > > > > > > > > >>
>> > > >> > > >> >> > > > > > > > > > >>
>> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM,
>> > Chinmay
>> > > >> > > >> Kolhatkar <
>> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
>> > > >> > > >> >> > > > > > > > > > >> wrote:
>> > > >> > > >> >> > > > > > > > > > >>
>> > > >> > > >> >> > > > > > > > > > >> > Hi All,
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >> > We're planning a work on adding
>> > Apache
>> > > >> Apex
>> > > >> > > as a
>> > > >> > > >> >> > > component
>> > > >> > > >> >> > > > > to
>> > > >> > > >> >> > > > > > > > Apache
>> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
>> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging system for
>> > the
>> > > >> > Apache
>> > > >> > > >> big
>> > > >> > > >> >> data
>> > > >> > > >> >> > > > > > > ecosystem.
>> > > >> > > >> >> > > > > > > > > > >> Several
>> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most
>> recently
>> > > EMR.
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira task in
>> > > >> APEXCORE
>> > > >> > for
>> > > >> > > >> the
>> > > >> > > >> >> > same:
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of execution is as
>> > > follows:
>> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
>> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard and
>> soft
>> > > >> > > expectation
>> > > >> > > >> >> for
>> > > >> > > >> >> > new
>> > > >> > > >> >> > > > > > > > components
>> > > >> > > >> >> > > > > > > > > to
>> > > >> > > >> >> > > > > > > > > > >> get
>> > > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
>> > > >> > > >> >> > > > > > > > > > >> > Here is the list of it:
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >>
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > >
>> > > >> > > >> >> > > >
>> > > >> > > >> >> > >
>> > > >> > > >> >> >
>> > > >> > > >> >>
>> > > >> > > >>
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to be standard
>> ASF
>> > > >> based
>> > > >> > > >> >> > > requirements,
>> > > >> > > >> >> > > > > but
>> > > >> > > >> >> > > > > > > few
>> > > >> > > >> >> > > > > > > > > need
>> > > >> > > >> >> > > > > > > > > > >> to
>> > > >> > > >> >> > > > > > > > > > >> > be checked for:
>> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects are expected
>> to
>> > be
>> > > >> > > Licensed
>> > > >> > > >> >> under
>> > > >> > > >> >> > > > > Apache
>> > > >> > > >> >> > > > > > > > > License,
>> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
>> dependencies
>> > are
>> > > >> > > expected
>> > > >> > > >> >> to be
>> > > >> > > >> >> > > > > > > compatible
>> > > >> > > >> >> > > > > > > > > with
>> > > >> > > >> >> > > > > > > > > > >> this
>> > > >> > > >> >> > > > > > > > > > >> > license)
>> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but
>> need
>> > to
>> > > >> > check
>> > > >> > > if
>> > > >> > > >> >> > > > > dependencies
>> > > >> > > >> >> > > > > > of
>> > > >> > > >> >> > > > > > > > > Apex
>> > > >> > > >> >> > > > > > > > > > >> are
>> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I
>> guess
>> > > >> would
>> > > >> > > be a
>> > > >> > > >> >> > > > > verification
>> > > >> > > >> >> > > > > > > > check.
>> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects are expected
>> to
>> > be
>> > > >> > > >> compatible
>> > > >> > > >> >> > with
>> > > >> > > >> >> > > > all
>> > > >> > > >> >> > > > > of
>> > > >> > > >> >> > > > > > > the
>> > > >> > > >> >> > > > > > > > > > >> > supported platforms that Bigtop
>> > > >> distribution
>> > > >> > > is
>> > > >> > > >> >> > > targeting
>> > > >> > > >> >> > > > > > > > > > >> >     - This needs verifying whether
>> > our
>> > > >> > > software
>> > > >> > > >> >> runs
>> > > >> > > >> >> > > fine
>> > > >> > > >> >> > > > in
>> > > >> > > >> >> > > > > > > > > centos-6
>> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
>> > > debian-8
>> > > >> > > >> >> > opensuse-13.2.
>> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that should be
>> > > added
>> > > >> for
>> > > >> > > >> >> > deployment.
>> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test artifacts
>> > which
>> > > >> goes
>> > > >> > > >> beyond
>> > > >> > > >> >> > > smoke
>> > > >> > > >> >> > > > > test
>> > > >> > > >> >> > > > > > > > > > >> >     - These are basically the
>> > > integration
>> > > >> > > tests
>> > > >> > > >> for
>> > > >> > > >> >> > > > > > verification
>> > > >> > > >> >> > > > > > > > > after
>> > > >> > > >> >> > > > > > > > > > >> the
>> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a soft
>> > requirement,
>> > > >> but
>> > > >> > > aim
>> > > >> > > >> is
>> > > >> > > >> >> to
>> > > >> > > >> >> > > > > achieve
>> > > >> > > >> >> > > > > > > this
>> > > >> > > >> >> > > > > > > > > as
>> > > >> > > >> >> > > > > > > > > > >> well
>> > > >> > > >> >> > > > > > > > > > >> > or at least have explanation why
>> not
>> > to
>> > > >> > > include.
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >> > If there are any from the link
>> which
>> > > >> > > explicitly
>> > > >> > > >> >> needs
>> > > >> > > >> >> > to
>> > > >> > > >> >> > > > be
>> > > >> > > >> >> > > > > > > > checked
>> > > >> > > >> >> > > > > > > > > > >> other
>> > > >> > > >> >> > > > > > > > > > >> > than above 4, please let us know.
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as component
>> to
>> > > >> Bigtop*
>> > > >> > > >> >> > > > > > > > > > >> > From one of the mail archive of
>> > Bigtop,
>> > > >> it
>> > > >> > was
>> > > >> > > >> >> learnt
>> > > >> > > >> >> > > that
>> > > >> > > >> >> > > > > the
>> > > >> > > >> >> > > > > > > > > bigtop
>> > > >> > > >> >> > > > > > > > > > >> > community want to see the
>> addition of
>> > > new
>> > > >> > > >> >> components
>> > > >> > > >> >> > in
>> > > >> > > >> >> > > > > > phases.
>> > > >> > > >> >> > > > > > > > Here
>> > > >> > > >> >> > > > > > > > > > are
>> > > >> > > >> >> > > > > > > > > > >> > the phases:
>> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
>> > > >> > > >> >> > > > > > > > > > >> >     - This needs creating of
>> package
>> > > i.e.
>> > > >> > rpm
>> > > >> > > &
>> > > >> > > >> deb
>> > > >> > > >> >> > > files.
>> > > >> > > >> >> > > > > > > > > > >> >     - documentations/READMEs,
>> > LICENSE,
>> > > >> > > >> DISCLAMER,
>> > > >> > > >> >> > NOTES
>> > > >> > > >> >> > > > etc
>> > > >> > > >> >> > > > > if
>> > > >> > > >> >> > > > > > > any
>> > > >> > > >> >> > > > > > > > > > >> needed.
>> > > >> > > >> >> > > > > > > > > > >> >     - Any documentation that need
>> to
>> > be
>> > > >> > added
>> > > >> > > to
>> > > >> > > >> >> > > > > distribution
>> > > >> > > >> >> > > > > > of
>> > > >> > > >> >> > > > > > > > our
>> > > >> > > >> >> > > > > > > > > > >> > software.
>> > > >> > > >> >> > > > > > > > > > >> >     - Any license information of
>> > > >> > dependencies
>> > > >> > > >> >> required
>> > > >> > > >> >> > > to
>> > > >> > > >> >> > > > be
>> > > >> > > >> >> > > > > > > added
>> > > >> > > >> >> > > > > > > > > to
>> > > >> > > >> >> > > > > > > > > > >> > package
>> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
>> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke test for
>> > packaging.
>> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic
>> > > >> deployment
>> > > >> > and
>> > > >> > > >> >> > > > configuration
>> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for
>> > automatic
>> > > >> > > >> deployment
>> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
>> > > >> > > >> >> > > > > > > > > > >> >     - For verification of
>> > deployments.
>> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
>> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure
>> all
>> > > new
>> > > >> > > stuff
>> > > >> > > >> is
>> > > >> > > >> >> > > > compliant
>> > > >> > > >> >> > > > > > > with
>> > > >> > > >> >> > > > > > > > > ASF
>> > > >> > > >> >> > > > > > > > > > >> > license requirements. If you add
>> code
>> > > >> > licenses
>> > > >> > > >> >> under
>> > > >> > > >> >> > > > > different
>> > > >> > > >> >> > > > > > > > > > licenses,
>> > > >> > > >> >> > > > > > > > > > >> > those would need to be listed in
>> the
>> > > >> NOTICE.
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >> > Please share your thoughts on the
>> > > >> approach.
>> > > >> > > >> >> > > > > > > > > > >> > We'll start corresponding
>> > communication
>> > > >> on
>> > > >> > > >> bigtop
>> > > >> > > >> >> > > mailing
>> > > >> > > >> >> > > > > list
>> > > >> > > >> >> > > > > > > as
>> > > >> > > >> >> > > > > > > > > > well.
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >> > We have some specific
>> > > >> questions/suggestions
>> > > >> > > >> >> related to
>> > > >> > > >> >> > > > what
>> > > >> > > >> >> > > > > > > should
>> > > >> > > >> >> > > > > > > > > be
>> > > >> > > >> >> > > > > > > > > > >> the
>> > > >> > > >> >> > > > > > > > > > >> > content of the package and what
>> > should
>> > > be
>> > > >> > the
>> > > >> > > >> smoke
>> > > >> > > >> >> > > tests,
>> > > >> > > >> >> > > > > but
>> > > >> > > >> >> > > > > > > in
>> > > >> > > >> >> > > > > > > > > the
>> > > >> > > >> >> > > > > > > > > > >> > interest of not having too much
>> > content
>> > > >> > here,
>> > > >> > > >> we'll
>> > > >> > > >> >> > put
>> > > >> > > >> >> > > > the
>> > > >> > > >> >> > > > > > > > > questions
>> > > >> > > >> >> > > > > > > > > > >> as a
>> > > >> > > >> >> > > > > > > > > > >> > separate mail in this mailthread.
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >> > Thanks,
>> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
>> > > >> > > >> >> > > > > > > > > > >> >
>> > > >> > > >> >> > > > > > > > > > >>
>> > > >> > > >> >> > > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > > >
>> > > >> > > >> >> > > > > > > > > >
>> > > >> > > >> >> > > > > > > > >
>> > > >> > > >> >> > > > > > > >
>> > > >> > > >> >> > > > > > >
>> > > >> > > >> >> > > > > >
>> > > >> > > >> >> > > > >
>> > > >> > > >> >> > > >
>> > > >> > > >> >> > >
>> > > >> > > >> >> >
>> > > >> > > >> >>
>> > > >> > > >> >
>> > > >> > > >> >
>> > > >> > > >>
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

I've already started communication there.

On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <th...@datatorrent.com>
wrote:

> This is a question you should ask on the Bigtop mailing list.
>
> --
> sent from mobile
> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org> wrote:
>
> > Yes. In case of RPM, the only restriction on version is that it can not
> > contain '-'.
> > However, as a workaround, one can use 'incubating' as a part of
> %{release}.
> >
> > Version: 3.3.0
> > Release: incubating
> >
> > apex-3.3.0-incubating.noarch.rpm
> >
> > It is the same practice followed for pre-release packages e.g. with rcX,
> > alphaX, betaX being used in %{release}.
> >
> > Regards,
> > Pradeep A. Dalvi
> >
> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > In one of the previous mails in this thread, we decided that version of
> > the
> > > package will be "3.3.0-incubating".
> > > The problem that I'm facing is in RPM base version "-" is not allowed.
> > >
> > > Can we have package version to "3.3.0"?
> > > i.e. the packages created will be as follows:
> > > For RPM: apex-3.3.0-1.noarch.rpm
> > > For DEB: apex_3.3.0-1_all.deb
> > >
> > > There is another integration for Zepplin in Bigtop which is incubating
> > and
> > > is having base version as "0.5.6".
> > >
> > > Please let me know your thoughts.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > >
> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> chinmay@apache.org>
> > > wrote:
> > >
> > > > Thanks everyone for the feedback.
> > > >
> > > > Also as Ram and David mentioned, I'll start creating the man page as
> > > well.
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> thomas@datatorrent.com>
> > > > wrote:
> > > >
> > > >> I think we already agreed that we will retain the current script
> name
> > in
> > > >> this bigtop iteration. There is a ticket for renaming, add you
> > > suggestions
> > > >> there.
> > > >>
> > > >> --
> > > >> sent from mobile
> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <sa...@datatorrent.com>
> > > wrote:
> > > >>
> > > >> > +1 for Symlink from apex to dtcli or start the transition process
> by
> > > >> making
> > > >> > dtcli as the symlink to apex cli.
> > > >> >
> > > >> > Renaming is a breaking change should be done during major version
> > > number
> > > >> > change.
> > > >> >
> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <da...@datatorrent.com>
> > > >> wrote:
> > > >> >
> > > >> > > Apologies for hitting the "Send" button too early.
> > > >> > >
> > > >> > > What is the scope of the name change?  If we simply have a
> symlink
> > > >> from
> > > >> > > apex to dtcli, then it should have no backward compatibility
> > problem
> > > >> at
> > > >> > > all.
> > > >> > >
> > > >> > > But the proper name change would involve:
> > > >> > > - Change the name of DTCli class to ApexCli
> > > >> > > - Change the prompt from dt> to apex>
> > > >> > > - Change the initial greeting when running the CLI from
> something
> > > like
> > > >> > "DT
> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > >> > >
> > > >> > > The above changes may break existing scripts that rely on the
> the
> > > >> string
> > > >> > > "DT".
> > > >> > > But if folks are okay with this potential problem, I am okay
> with
> > > the
> > > >> > name
> > > >> > > change.
> > > >> > >
> > > >> > > For the man page, we can start with the content of this page:
> > > >> > >
> > > >> > > http://docs.datatorrent.com/dtcli/
> > > >> > >
> > > >> > > Plus the command line options when doing dtcli -h:
> > > >> > >
> > > >> > > usage: DTCli
> > > >> > >  -e <arg>    Commands are read from the argument
> > > >> > >  -f <arg>    Use the specified prompt at all time
> > > >> > >  -h          Print this help
> > > >> > >  -kp <arg>   Use the specified kerberos principal
> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> > > >> > >  -p <arg>    JSONP padding function
> > > >> > >  -r          JSON Raw mode
> > > >> > >  -v          Verbose mode level 1
> > > >> > >  -vv         Verbose mode level 2
> > > >> > >  -vvv        Verbose mode level 3
> > > >> > >  -vvvv       Verbose mode level 4
> > > >> > >
> > > >> > > David
> > > >> > >
> > > >> > >
> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> david@datatorrent.com
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > > > What is the scope of the name change?  If we simply have a
> > symlink
> > > >> from
> > > >> > > > apex to dtcli, then it should have no backward compatibility
> > > >> problem at
> > > >> > > all.
> > > >> > > >
> > > >> > > > But the proper name change would involve:
> > > >> > > > - Change the name of DTCli class to ApexCli
> > > >> > > > - Change the prompt from dt> to apex>
> > > >> > > > - Change the initial greeting when running the CLI from
> > something
> > > >> like
> > > >> > > "DT
> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > >> > > >
> > > >> > > > The above changes may break existing scripts that rely on the
> > the
> > > >> > string
> > > >> > > > "DT".
> > > >> > > > But if folks are okay with this potential problem, I am okay
> > with
> > > >> the
> > > >> > > name
> > > >> > > > change.
> > > >> > > >
> > > >> > > > For the man page, we can start with the content of this page:
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> > > >> chinmay@apache.org
> > > >> > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > >> Here is a suggestion I have related to name change:
> > > >> > > >>
> > > >> > > >> 1. While packaging we change the name of "dtcli" script to
> > > "apex".
> > > >> > This
> > > >> > > is
> > > >> > > >> done so that in future when we change the name to "apex",
> users
> > > of
> > > >> > > bigtop
> > > >> > > >> apex don't have to transition much.
> > > >> > > >> 2. We also keep a symlink named dtcli which points to apex
> > > script.
> > > >> > This
> > > >> > > is
> > > >> > > >> for backward compatibility.
> > > >> > > >> 3. We still keep conf/dt-env.sh for first integration in
> > bigtop.
> > > >> This
> > > >> > is
> > > >> > > >> in
> > > >> > > >> the interest of not changing the content of dtcli or apex
> > script.
> > > >> > > >> 4. In apex 3.4.0, we take care of all these naming related
> > > changes
> > > >> and
> > > >> > > >> update bigtop repository later to remove references to dtcli
> > all
> > > >> > > together.
> > > >> > > >>
> > > >> > > >> Please share your thoughts on above approach.
> > > >> > > >>
> > > >> > > >> Also, please share what could be the man page content for
> > "apex".
> > > >> > > >>
> > > >> > > >> Thanks,
> > > >> > > >> Chinmay.
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> > > >> > chinmay@apache.org>
> > > >> > > >> wrote:
> > > >> > > >>
> > > >> > > >> > Yes.. It'll be build from source tar downloaded from one of
> > the
> > > >> > apache
> > > >> > > >> > mirror: apache.osuosl.org
> > > >> > > >> >
> > > >> > > >> > The reason for dt-env.sh is it's sourced from dtcli. This
> > > leaves
> > > >> us
> > > >> > 3
> > > >> > > >> > options:
> > > >> > > >> > If we should not have any dt-env.sh, dtcli would need a
> > change
> > > >> while
> > > >> > > >> > packaging.
> > > >> > > >> >
> > > >> > > >> > But if we want to use dtcli as it is, then we would need
> > > >> dt-env.sh
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> > > >> > thomas@datatorrent.com
> > > >> > > >
> > > >> > > >> > wrote:
> > > >> > > >> >
> > > >> > > >> >> Looks good overall, though there shouldn't be any
> dt-env.sh
> > > >> > > >> >>
> > > >> > > >> >> What will the bigtop package be built from, the source
> tar?
> > > >> > > >> >>
> > > >> > > >> >>
> > > >> > > >> >>
> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > > >> > > chinmay@apache.org
> > > >> > > >> >
> > > >> > > >> >> wrote:
> > > >> > > >> >>
> > > >> > > >> >> > Hi Thomas,
> > > >> > > >> >> >
> > > >> > > >> >> > Thanks for the feedback.
> > > >> > > >> >> >
> > > >> > > >> >> > First, we're not changing the name anywhere. We'll
> follow
> > > what
> > > >> > > >> >> currently is
> > > >> > > >> >> > in source tarball for 3.3.0-incubating version.
> > > >> > > >> >> >
> > > >> > > >> >> > Secondly, I've mentioned a directory structure below
> which
> > > is
> > > >> > > inline
> > > >> > > >> >> with
> > > >> > > >> >> > other existing integrations with bigtop.
> > > >> > > >> >> >
> > > >> > > >> >> > The need for each file is as follows:
> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> > > >> engine/src/main/scripts/
> > > >> > > of
> > > >> > > >> >> source
> > > >> > > >> >> > code apex 3.3.0-incubating.
> > > >> > > >> >> >
> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This
> > > searches
> > > >> for
> > > >> > > >> hadoop
> > > >> > > >> >> > binary path and exports an env variable for dtcli to use
> > it.
> > > >> > > >> >> >
> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is extracted in
> a
> > > >> similar
> > > >> > > >> way to
> > > >> > > >> >> > how DT Community Edition finds dependency jars.
> > > >> > > >> >> > Please note that this is the first iteration list of
> jars.
> > > I'm
> > > >> > > >> trying to
> > > >> > > >> >> > narrow this down to only those which are really
> required.
> > > >> > > >> >> > The test that I'm running to check if dtcli runs fine
> with
> > > >> given
> > > >> > > >> >> dependency
> > > >> > > >> >> > is to launch+shutdown+kill+status for pi demo.
> > > >> > > >> >> >
> > > >> > > >> >> > Above is the directory structure which is required for
> CLI
> > > to
> > > >> > work.
> > > >> > > >> >> > What aniruddha and pradeep has mentioned are some
> > additional
> > > >> > files
> > > >> > > >> which
> > > >> > > >> >> > makes the like of administrator easier. For eg.
> /etc/skel
> > > >> > > >> >> >
> > > >> > > >> >> > Please let me know if this is inline with what you're
> > > >> thinking.
> > > >> > > >> >> >
> > > >> > > >> >> > Thanks,
> > > >> > > >> >> > Chinmay.
> > > >> > > >> >> >
> > > >> > > >> >> >
> > > >> > > >> >> > .
> > > >> > > >> >> > |-- bin
> > > >> > > >> >> > |   `-- dtcli
> > > >> > > >> >> > |-- conf
> > > >> > > >> >> > |   `-- dt-env.sh
> > > >> > > >> >> > `-- lib
> > > >> > > >> >> >     |-- ant-1.9.2.jar
> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > > >> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > > >> > > >> >> >     |-- apex-engine.jar
> > > >> > > >> >> >     |-- apex-engine-tests.jar
> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > > >> > > >> >> >     |-- bval-core-0.5.jar
> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > > >> > > >> >> >     |-- jline-2.11.jar
> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > > >> > > >> >> >     |-- minlog-1.2.jar
> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > > >> > > >> >> >     |-- objenesis-2.1.jar
> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > > >> > > >> >> >
> > > >> > > >> >> > 3 directories, 40 files
> > > >> > > >> >> >
> > > >> > > >> >> >
> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > > >> > > >> thomas@datatorrent.com>
> > > >> > > >> >> > wrote:
> > > >> > > >> >> >
> > > >> > > >> >> > > Chinmay,
> > > >> > > >> >> > >
> > > >> > > >> >> > > Before discussing where to put the files, let's make
> > sure
> > > >> they
> > > >> > > are
> > > >> > > >> >> really
> > > >> > > >> >> > > needed for the operation of the CLI. As for names,
> > > anything
> > > >> new
> > > >> > > >> needs
> > > >> > > >> >> to
> > > >> > > >> >> > > reflect Apex in the name and should follow common
> > > >> conventions,
> > > >> > > >> >> especially
> > > >> > > >> >> > > in Bigtop where there are many existing integrations
> to
> > > look
> > > >> > at.
> > > >> > > >> >> > >
> > > >> > > >> >> > > Thanks,
> > > >> > > >> >> > > Thomas
> > > >> > > >> >> > >
> > > >> > > >> >> > >
> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > > >> > > >> >> > > chinmay@datatorrent.com>
> > > >> > > >> >> > > wrote:
> > > >> > > >> >> > >
> > > >> > > >> >> > > > Really good points Pradeep and Aniruddha.
> > > >> > > >> >> > > >
> > > >> > > >> >> > > > 1. I believe we won't need to change the dtcli
> > > >> considering it
> > > >> > > >> works
> > > >> > > >> >> > with
> > > >> > > >> >> > > DT
> > > >> > > >> >> > > > Community edition. We can keep the directory
> structure
> > > >> > similar
> > > >> > > to
> > > >> > > >> >> that.
> > > >> > > >> >> > > > dt-env.sh has variables which contains information
> > > >> required
> > > >> > for
> > > >> > > >> >> dtcli
> > > >> > > >> >> > to
> > > >> > > >> >> > > > launch.
> > > >> > > >> >> > > >
> > > >> > > >> >> > > > 2. Let me check with bigtop community that whether
> > they
> > > >> > > >> facilitate
> > > >> > > >> >> the
> > > >> > > >> >> > > > installation of rpms/debs before user is created. In
> > > >> either
> > > >> > > case,
> > > >> > > >> >> > current
> > > >> > > >> >> > > > dtcli creates a .dt folder in home directory. Also
> > > before
> > > >> > > putting
> > > >> > > >> >> > > anything
> > > >> > > >> >> > > > in /etc/skel we need to define what are the default
> > > >> contents
> > > >> > > that
> > > >> > > >> >> > should
> > > >> > > >> >> > > go
> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults, probably
> we
> > > >> should
> > > >> > > not
> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> > > >> > > >> >> > > >
> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This way
> > > contents
> > > >> of
> > > >> > > >> >> dt-env.sh
> > > >> > > >> >> > > are
> > > >> > > >> >> > > > present as env variables. But I see a catch there.
> > > Adding
> > > >> > > >> dt-env.sh
> > > >> > > >> >> to
> > > >> > > >> >> > > > /etc/profile.d would make all the variables
> available
> > at
> > > >> > > runtime
> > > >> > > >> all
> > > >> > > >> >> > the
> > > >> > > >> >> > > > time. I feel a little skeptical about that. Maybe a
> > > >> possible
> > > >> > > >> >> collision
> > > >> > > >> >> > > can
> > > >> > > >> >> > > > occur with other application vars.
> > > >> > > >> >> > > > Moreover, current dtcli does source
> > "../conf/dt-env.sh"
> > > >> and
> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway
> > > >> available.
> > > >> > > >> >> > > >
> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is happening at
> > > Bigtop
> > > >> > > >> >> > > Jira/repository.
> > > >> > > >> >> > > > Packaging related code usually goes there. (That's
> > what
> > > >> all
> > > >> > the
> > > >> > > >> >> > > components
> > > >> > > >> >> > > > in bigtop does).
> > > >> > > >> >> > > > Having said that, once a PR is created, I'll be
> > sharing
> > > >> the
> > > >> > > link
> > > >> > > >> of
> > > >> > > >> >> PR
> > > >> > > >> >> > > > here, so that apex community as well can review it.
> > > >> > > >> >> > > >
> > > >> > > >> >> > > > Thanks,
> > > >> > > >> >> > > > Chinmay.
> > > >> > > >> >> > > >
> > > >> > > >> >> > > >
> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare
> <
> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > > >> > > >> >> > > >
> > > >> > > >> >> > > > > +1 on suggestions and approach.
> > > >> > > >> >> > > > > We may need to iron out details about exact paths
> > etc.
> > > >> > > >> >> > > > > Which can be done on jira / PR comments.
> > > >> > > >> >> > > > > Is that right @dev?
> > > >> > > >> >> > > > >
> > > >> > > >> >> > > > >
> > > >> > > >> >> > > > >
> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > >> > > >> >> > apache@pradeepdalvi.com>
> > > >> > > >> >> > > > > wrote:
> > > >> > > >> >> > > > >
> > > >> > > >> >> > > > > > Inline comments...
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
> > Kolhatkar <
> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > > >> > > >> >> > > > > > wrote:
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present in DT
> community
> > > >> > edition
> > > >> > > >> will
> > > >> > > >> >> be
> > > >> > > >> >> > > > > included
> > > >> > > >> >> > > > > > > there.
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > > > Here is the list of jars I found from common
> > > >> > dependencies
> > > >> > > >> >> between
> > > >> > > >> >> > > of
> > > >> > > >> >> > > > > apex
> > > >> > > >> >> > > > > > > and DT Community edition:
> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > > >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > > >> > > >> >> > > > > > > jline-2.11.jar
> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some tests do check
> > that
> > > >> > dtcli
> > > >> > > >> works
> > > >> > > >> >> > > > properly
> > > >> > > >> >> > > > > > for
> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with only these
> > > >> libraries
> > > >> > > >> >> present in
> > > >> > > >> >> > > > > > isolation
> > > >> > > >> >> > > > > > > without dependency on local m2.
> > > >> > > >> >> > > > > > > I'm believe that there are unwanted jars which
> > are
> > > >> used
> > > >> > > for
> > > >> > > >> >> > compile
> > > >> > > >> >> > > > > time
> > > >> > > >> >> > > > > > > dependency and not runtime in above list
> which I
> > > can
> > > >> > drop
> > > >> > > >> to
> > > >> > > >> >> keep
> > > >> > > >> >> > > > > package
> > > >> > > >> >> > > > > > > size minimal.
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > > > @Anniruddha,
> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
> > > >> dt-env.sh is
> > > >> > > >> >> required.
> > > >> > > >> >> > > > Only
> > > >> > > >> >> > > > > > > required values I see in it are classpath. and
> > > >> dtcli is
> > > >> > > >> >> capable
> > > >> > > >> >> > of
> > > >> > > >> >> > > > > > building
> > > >> > > >> >> > > > > > > it on the fly.
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how
> > dtcli
> > > >> can
> > > >> > > find
> > > >> > > >> >> the
> > > >> > > >> >> > jar
> > > >> > > >> >> > > > > path,
> > > >> > > >> >> > > > > > > dt-env.sh and all such conf files. If we need
> a
> > > >> change
> > > >> > in
> > > >> > > >> >> dtcli,
> > > >> > > >> >> > > then
> > > >> > > >> >> > > > > > > community's opinion is required for whether
> > dtcli
> > > >> > should
> > > >> > > >> >> change
> > > >> > > >> >> > in
> > > >> > > >> >> > > > our
> > > >> > > >> >> > > > > > repo
> > > >> > > >> >> > > > > > > or a copy of that with required changes exist
> in
> > > >> bigtop
> > > >> > > >> repo
> > > >> > > >> >> > until
> > > >> > > >> >> > > we
> > > >> > > >> >> > > > > > make
> > > >> > > >> >> > > > > > > the dtcli generic enough.
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > > Can we have these file paths set using
> environment
> > > >> > > variables
> > > >> > > >> >> with
> > > >> > > >> >> > > some
> > > >> > > >> >> > > > > > default values in dtcli?
> > > >> > > >> >> > > > > > Then we can set such params in dt-env.sh.
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file different
> > from
> > > >> > > >> >> dt-site.xml?
> > > >> > > >> >> > Not
> > > >> > > >> >> > > > > sure
> > > >> > > >> >> > > > > > if
> > > >> > > >> >> > > > > > > adding in /etc/skel would help. Files & Dirs
> > from
> > > >> > > /etc/skel
> > > >> > > >> >> are
> > > >> > > >> >> > > > copied
> > > >> > > >> >> > > > > to
> > > >> > > >> >> > > > > > > home of new user when useradd program is
> called.
> > > But
> > > >> > for
> > > >> > > >> >> existing
> > > >> > > >> >> > > > users
> > > >> > > >> >> > > > > > > this won't be of any use. Moreover, dtcli
> > creates
> > > >> > ~/.dt/
> > > >> > > on
> > > >> > > >> >> the
> > > >> > > >> >> > fly
> > > >> > > >> >> > > > if
> > > >> > > >> >> > > > > > not
> > > >> > > >> >> > > > > > > encountered for the first time. Again correct
> me
> > > if
> > > >> I'm
> > > >> > > >> wrong.
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > > Yes, you are right. However this would probably
> be
> > > >> > > necessary
> > > >> > > >> >> step
> > > >> > > >> >> > for
> > > >> > > >> >> > > > > > rpm/deb, as they may also get installed during
> OS
> > > >> install
> > > >> > > and
> > > >> > > >> >> > before
> > > >> > > >> >> > > > user
> > > >> > > >> >> > > > > > accounts were created.
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not
> > sure
> > > >> its
> > > >> > the
> > > >> > > >> best
> > > >> > > >> >> > > idea.
> > > >> > > >> >> > > > > > We'll
> > > >> > > >> >> > > > > > > anyway put dtcli in location which is by
> default
> > > >> > present
> > > >> > > in
> > > >> > > >> >> path
> > > >> > > >> >> > > i.e.
> > > >> > > >> >> > > > > > > /usr/bin.
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > > For other env variables specific to apex, I'll
> > > prefer
> > > >> to
> > > >> > > use
> > > >> > > >> >> > > dt-env.sh
> > > >> > > >> >> > > > > and
> > > >> > > >> >> > > > > > > source it in dtcli rather than changing bashrc
> > > >> etc...
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/
> shall
> > > be
> > > >> the
> > > >> > > >> >> cleaner
> > > >> > > >> >> > way
> > > >> > > >> >> > > > to
> > > >> > > >> >> > > > > > achieve the same.
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > > Thanks,
> > > >> > > >> >> > > > > > > Chinmay.
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
> > > Thombare
> > > >> <
> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > > > > Hi,
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > > @Chinmay,
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > > We need to consider following:
> > > >> > > >> >> > > > > > > > System wide default config files can be
> > located
> > > at
> > > >> > > >> following
> > > >> > > >> >> > > > > locations:
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change
> > > dtcli
> > > >> > > >> behaviour
> > > >> > > >> >> on
> > > >> > > >> >> > > how
> > > >> > > >> >> > > > it
> > > >> > > >> >> > > > > > > finds
> > > >> > > >> >> > > > > > > > those locations)
> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change
> > > dtcli
> > > >> > > >> >> behaviour on
> > > >> > > >> >> > > how
> > > >> > > >> >> > > > > it
> > > >> > > >> >> > > > > > > > finds those locations)
> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files
> > (for
> > > >> new
> > > >> > > users
> > > >> > > >> >> that
> > > >> > > >> >> > > will
> > > >> > > >> >> > > > > be
> > > >> > > >> >> > > > > > > > created in system in future.)
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > > We may also have to modify bashrc /
> > bashprofile
> > > >> for
> > > >> > > >> >> population
> > > >> > > >> >> > > any
> > > >> > > >> >> > > > > > > > variables that are required.
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > > @dev, please put in your comments /
> > suggestion.
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > > Thanks,
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > > Aniruddha
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas
> Weise
> > <
> > > >> > > >> >> > > > > thomas@datatorrent.com>
> > > >> > > >> >> > > > > > > > wrote:
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > > > > -->
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> > > >> Kolhatkar <
> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > wrote:
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > Continuing with packaging effort
> (rpm+deb)
> > > of
> > > >> > apex,
> > > >> > > >> here
> > > >> > > >> >> > are
> > > >> > > >> >> > > > some
> > > >> > > >> >> > > > > > > > > proposals
> > > >> > > >> >> > > > > > > > > > about package structure etc..
> > > >> > > >> >> > > > > > > > > > Before posting it on bbigtop mailing
> > list, I
> > > >> have
> > > >> > > >> some
> > > >> > > >> >> > > question
> > > >> > > >> >> > > > > for
> > > >> > > >> >> > > > > > > > apex
> > > >> > > >> >> > > > > > > > > > community.
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > Proposed Directory structure of apex
> > package
> > > >> for
> > > >> > > both
> > > >> > > >> >> deb &
> > > >> > > >> >> > > > rpm:
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > >> > > >> >> > > > > > > > > >
> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > >> > > >> >> > > > > > > > > >
> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > >> > > >> >> > > > > > > > > >
> > /usr/lib/apex/lib/apex-common-<version>.jar
> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > >> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > >> > > >> >> > > > > > > > > >
> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> > > >> > > >> >> > > > > > > > > > <other dependent jars> = All the 3rd
> party
> > > >> jars
> > > >> > > which
> > > >> > > >> >> are
> > > >> > > >> >> > > > > required
> > > >> > > >> >> > > > > > > for
> > > >> > > >> >> > > > > > > > > apex
> > > >> > > >> >> > > > > > > > > > to run. Usually the dependencies are
> > > packaged
> > > >> as
> > > >> > > >> part of
> > > >> > > >> >> > > > rpm/deb
> > > >> > > >> >> > > > > by
> > > >> > > >> >> > > > > > > any
> > > >> > > >> >> > > > > > > > > > software in bigtop.
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > Can you specify what those jars are in
> > Bigtop
> > > >> > > context?
> > > >> > > >> >> Same
> > > >> > > >> >> > as
> > > >> > > >> >> > > > > > shipped
> > > >> > > >> >> > > > > > > > with
> > > >> > > >> >> > > > > > > > > DT community addition under lib/ ?
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of
> > > corresponding
> > > >> of
> > > >> > > 3rd
> > > >> > > >> >> party
> > > >> > > >> >> > > jars
> > > >> > > >> >> > > > > > which
> > > >> > > >> >> > > > > > > > > needs
> > > >> > > >> >> > > > > > > > > > to included while packaging.
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > Questions related to this:
> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli of apex as
> > "apex"
> > > >> > instead
> > > >> > > >> of
> > > >> > > >> >> > > "dtcli"
> > > >> > > >> >> > > > in
> > > >> > > >> >> > > > > > > > bigtop
> > > >> > > >> >> > > > > > > > > > package?
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > I think we should keep the name until we
> are
> > > >> able
> > > >> > to
> > > >> > > >> >> change
> > > >> > > >> >> > it
> > > >> > > >> >> > > in
> > > >> > > >> >> > > > > > Apex.
> > > >> > > >> >> > > > > > > > > Otherwise this may get confusing.
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares in bigtop
> have
> > > man
> > > >> > page
> > > >> > > >> for
> > > >> > > >> >> > their
> > > >> > > >> >> > > > > > > > > executables.
> > > >> > > >> >> > > > > > > > > > I think we should have it too for dtcli.
> > Is
> > > >> there
> > > >> > > any
> > > >> > > >> >> > > > > documentation
> > > >> > > >> >> > > > > > > > > which I
> > > >> > > >> >> > > > > > > > > > can convert to man page? or can I use
> > output
> > > >> of
> > > >> > > >> "dtcli
> > > >> > > >> >> > > --help"?
> > > >> > > >> >> > > > > > > > > > 3. Do we want to call version of apex in
> > > >> Bigtop
> > > >> > as
> > > >> > > >> >> 3.3.0 OR
> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > Has to be -incubating
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of bigtop
> to
> > > >> depend
> > > >> > on
> > > >> > > >> >> 2.7.1
> > > >> > > >> >> > > > > version
> > > >> > > >> >> > > > > > of
> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that we see
> > with
> > > >> this
> > > >> > > >> >> > dependency?
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > Should work without changed. I thought we
> > > >> certified
> > > >> > > >> >> against
> > > >> > > >> >> > > > 2.7.0?
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > > 5. Following is the apache mirror from
> > which
> > > >> > bigtop
> > > >> > > >> will
> > > >> > > >> >> > pick
> > > >> > > >> >> > > > the
> > > >> > > >> >> > > > > > > apex
> > > >> > > >> >> > > > > > > > > > source for compilation and packaging.
> > Please
> > > >> > > correct
> > > >> > > >> if
> > > >> > > >> >> > > > > incorrect:
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > >
> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the
> mirror
> > > >> used
> > > >> > by
> > > >> > > >> all
> > > >> > > >> >> the
> > > >> > > >> >> > > > > > softwares
> > > >> > > >> >> > > > > > > in
> > > >> > > >> >> > > > > > > > > > bigtop.
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > Please share your opinion.
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > Thanks,
> > > >> > > >> >> > > > > > > > > > Chinmay.
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> > > >> > Kolhatkar
> > > >> > > <
> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > > >> > > >> >> > > > > > > > > > wrote:
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the
> packaging.
> > > >> > > >> >> > > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > > I've already started discussion on
> > bigtop
> > > >> dev
> > > >> > > >> mailing
> > > >> > > >> >> > list
> > > >> > > >> >> > > > for
> > > >> > > >> >> > > > > > > > > > > integration. Also created a Jira for
> the
> > > >> same.
> > > >> > > For
> > > >> > > >> >> this
> > > >> > > >> >> > > > > > communities
> > > >> > > >> >> > > > > > > > > > > reference, here is the bigtop Jira:
> > > >> > BIGTOP-2313.
> > > >> > > >> >> > > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas
> > > >> Weise <
> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > > > > > wrote:
> > > >> > > >> >> > > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> > > >> > > >> >> > > > > > > > > > >>
> > > >> > > >> >> > > > > > > > > > >> I don't see anything under
> > prerequisites
> > > >> that
> > > >> > > >> would
> > > >> > > >> >> be a
> > > >> > > >> >> > > > > > problem.
> > > >> > > >> >> > > > > > > We
> > > >> > > >> >> > > > > > > > > > >> looked
> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing compatibility as
> > > part
> > > >> of
> > > >> > > >> >> becoming
> > > >> > > >> >> > an
> > > >> > > >> >> > > > > > > incubator
> > > >> > > >> >> > > > > > > > > > >> project.
> > > >> > > >> >> > > > > > > > > > >>
> > > >> > > >> >> > > > > > > > > > >> Please focus on the packaging during
> > the
> > > >> next
> > > >> > > >> weeks.
> > > >> > > >> >> > Since
> > > >> > > >> >> > > > the
> > > >> > > >> >> > > > > > > work
> > > >> > > >> >> > > > > > > > > will
> > > >> > > >> >> > > > > > > > > > >> be
> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related discussions
> and
> > > >> JIRAs
> > > >> > > >> should
> > > >> > > >> >> > also
> > > >> > > >> >> > > be
> > > >> > > >> >> > > > > > > there.
> > > >> > > >> >> > > > > > > > > > >>
> > > >> > > >> >> > > > > > > > > > >> Would be good to have the packaging
> in
> > > >> place
> > > >> > by
> > > >> > > >> end
> > > >> > > >> >> Feb.
> > > >> > > >> >> > > > > > > > > > >>
> > > >> > > >> >> > > > > > > > > > >> Thanks,
> > > >> > > >> >> > > > > > > > > > >> Thomas
> > > >> > > >> >> > > > > > > > > > >>
> > > >> > > >> >> > > > > > > > > > >>
> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM,
> > Chinmay
> > > >> > > >> Kolhatkar <
> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> > > >> > > >> >> > > > > > > > > > >> wrote:
> > > >> > > >> >> > > > > > > > > > >>
> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >> > We're planning a work on adding
> > Apache
> > > >> Apex
> > > >> > > as a
> > > >> > > >> >> > > component
> > > >> > > >> >> > > > > to
> > > >> > > >> >> > > > > > > > Apache
> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging system for
> > the
> > > >> > Apache
> > > >> > > >> big
> > > >> > > >> >> data
> > > >> > > >> >> > > > > > > ecosystem.
> > > >> > > >> >> > > > > > > > > > >> Several
> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most
> recently
> > > EMR.
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira task in
> > > >> APEXCORE
> > > >> > for
> > > >> > > >> the
> > > >> > > >> >> > same:
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of execution is as
> > > follows:
> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard and
> soft
> > > >> > > expectation
> > > >> > > >> >> for
> > > >> > > >> >> > new
> > > >> > > >> >> > > > > > > > components
> > > >> > > >> >> > > > > > > > > to
> > > >> > > >> >> > > > > > > > > > >> get
> > > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
> > > >> > > >> >> > > > > > > > > > >> > Here is the list of it:
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >>
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > >
> > > >> > > >> >> > > >
> > > >> > > >> >> > >
> > > >> > > >> >> >
> > > >> > > >> >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to be standard
> ASF
> > > >> based
> > > >> > > >> >> > > requirements,
> > > >> > > >> >> > > > > but
> > > >> > > >> >> > > > > > > few
> > > >> > > >> >> > > > > > > > > need
> > > >> > > >> >> > > > > > > > > > >> to
> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects are expected
> to
> > be
> > > >> > > Licensed
> > > >> > > >> >> under
> > > >> > > >> >> > > > > Apache
> > > >> > > >> >> > > > > > > > > License,
> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their dependencies
> > are
> > > >> > > expected
> > > >> > > >> >> to be
> > > >> > > >> >> > > > > > > compatible
> > > >> > > >> >> > > > > > > > > with
> > > >> > > >> >> > > > > > > > > > >> this
> > > >> > > >> >> > > > > > > > > > >> > license)
> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but
> need
> > to
> > > >> > check
> > > >> > > if
> > > >> > > >> >> > > > > dependencies
> > > >> > > >> >> > > > > > of
> > > >> > > >> >> > > > > > > > > Apex
> > > >> > > >> >> > > > > > > > > > >> are
> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I
> guess
> > > >> would
> > > >> > > be a
> > > >> > > >> >> > > > > verification
> > > >> > > >> >> > > > > > > > check.
> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects are expected
> to
> > be
> > > >> > > >> compatible
> > > >> > > >> >> > with
> > > >> > > >> >> > > > all
> > > >> > > >> >> > > > > of
> > > >> > > >> >> > > > > > > the
> > > >> > > >> >> > > > > > > > > > >> > supported platforms that Bigtop
> > > >> distribution
> > > >> > > is
> > > >> > > >> >> > > targeting
> > > >> > > >> >> > > > > > > > > > >> >     - This needs verifying whether
> > our
> > > >> > > software
> > > >> > > >> >> runs
> > > >> > > >> >> > > fine
> > > >> > > >> >> > > > in
> > > >> > > >> >> > > > > > > > > centos-6
> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
> > > debian-8
> > > >> > > >> >> > opensuse-13.2.
> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that should be
> > > added
> > > >> for
> > > >> > > >> >> > deployment.
> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test artifacts
> > which
> > > >> goes
> > > >> > > >> beyond
> > > >> > > >> >> > > smoke
> > > >> > > >> >> > > > > test
> > > >> > > >> >> > > > > > > > > > >> >     - These are basically the
> > > integration
> > > >> > > tests
> > > >> > > >> for
> > > >> > > >> >> > > > > > verification
> > > >> > > >> >> > > > > > > > > after
> > > >> > > >> >> > > > > > > > > > >> the
> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a soft
> > requirement,
> > > >> but
> > > >> > > aim
> > > >> > > >> is
> > > >> > > >> >> to
> > > >> > > >> >> > > > > achieve
> > > >> > > >> >> > > > > > > this
> > > >> > > >> >> > > > > > > > > as
> > > >> > > >> >> > > > > > > > > > >> well
> > > >> > > >> >> > > > > > > > > > >> > or at least have explanation why
> not
> > to
> > > >> > > include.
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >> > If there are any from the link
> which
> > > >> > > explicitly
> > > >> > > >> >> needs
> > > >> > > >> >> > to
> > > >> > > >> >> > > > be
> > > >> > > >> >> > > > > > > > checked
> > > >> > > >> >> > > > > > > > > > >> other
> > > >> > > >> >> > > > > > > > > > >> > than above 4, please let us know.
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as component
> to
> > > >> Bigtop*
> > > >> > > >> >> > > > > > > > > > >> > From one of the mail archive of
> > Bigtop,
> > > >> it
> > > >> > was
> > > >> > > >> >> learnt
> > > >> > > >> >> > > that
> > > >> > > >> >> > > > > the
> > > >> > > >> >> > > > > > > > > bigtop
> > > >> > > >> >> > > > > > > > > > >> > community want to see the addition
> of
> > > new
> > > >> > > >> >> components
> > > >> > > >> >> > in
> > > >> > > >> >> > > > > > phases.
> > > >> > > >> >> > > > > > > > Here
> > > >> > > >> >> > > > > > > > > > are
> > > >> > > >> >> > > > > > > > > > >> > the phases:
> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > > >> > > >> >> > > > > > > > > > >> >     - This needs creating of
> package
> > > i.e.
> > > >> > rpm
> > > >> > > &
> > > >> > > >> deb
> > > >> > > >> >> > > files.
> > > >> > > >> >> > > > > > > > > > >> >     - documentations/READMEs,
> > LICENSE,
> > > >> > > >> DISCLAMER,
> > > >> > > >> >> > NOTES
> > > >> > > >> >> > > > etc
> > > >> > > >> >> > > > > if
> > > >> > > >> >> > > > > > > any
> > > >> > > >> >> > > > > > > > > > >> needed.
> > > >> > > >> >> > > > > > > > > > >> >     - Any documentation that need
> to
> > be
> > > >> > added
> > > >> > > to
> > > >> > > >> >> > > > > distribution
> > > >> > > >> >> > > > > > of
> > > >> > > >> >> > > > > > > > our
> > > >> > > >> >> > > > > > > > > > >> > software.
> > > >> > > >> >> > > > > > > > > > >> >     - Any license information of
> > > >> > dependencies
> > > >> > > >> >> required
> > > >> > > >> >> > > to
> > > >> > > >> >> > > > be
> > > >> > > >> >> > > > > > > added
> > > >> > > >> >> > > > > > > > > to
> > > >> > > >> >> > > > > > > > > > >> > package
> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke test for
> > packaging.
> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic
> > > >> deployment
> > > >> > and
> > > >> > > >> >> > > > configuration
> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for
> > automatic
> > > >> > > >> deployment
> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> > > >> > > >> >> > > > > > > > > > >> >     - For verification of
> > deployments.
> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure
> all
> > > new
> > > >> > > stuff
> > > >> > > >> is
> > > >> > > >> >> > > > compliant
> > > >> > > >> >> > > > > > > with
> > > >> > > >> >> > > > > > > > > ASF
> > > >> > > >> >> > > > > > > > > > >> > license requirements. If you add
> code
> > > >> > licenses
> > > >> > > >> >> under
> > > >> > > >> >> > > > > different
> > > >> > > >> >> > > > > > > > > > licenses,
> > > >> > > >> >> > > > > > > > > > >> > those would need to be listed in
> the
> > > >> NOTICE.
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >> > Please share your thoughts on the
> > > >> approach.
> > > >> > > >> >> > > > > > > > > > >> > We'll start corresponding
> > communication
> > > >> on
> > > >> > > >> bigtop
> > > >> > > >> >> > > mailing
> > > >> > > >> >> > > > > list
> > > >> > > >> >> > > > > > > as
> > > >> > > >> >> > > > > > > > > > well.
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >> > We have some specific
> > > >> questions/suggestions
> > > >> > > >> >> related to
> > > >> > > >> >> > > > what
> > > >> > > >> >> > > > > > > should
> > > >> > > >> >> > > > > > > > > be
> > > >> > > >> >> > > > > > > > > > >> the
> > > >> > > >> >> > > > > > > > > > >> > content of the package and what
> > should
> > > be
> > > >> > the
> > > >> > > >> smoke
> > > >> > > >> >> > > tests,
> > > >> > > >> >> > > > > but
> > > >> > > >> >> > > > > > > in
> > > >> > > >> >> > > > > > > > > the
> > > >> > > >> >> > > > > > > > > > >> > interest of not having too much
> > content
> > > >> > here,
> > > >> > > >> we'll
> > > >> > > >> >> > put
> > > >> > > >> >> > > > the
> > > >> > > >> >> > > > > > > > > questions
> > > >> > > >> >> > > > > > > > > > >> as a
> > > >> > > >> >> > > > > > > > > > >> > separate mail in this mailthread.
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > > >> > > >> >> > > > > > > > > > >> >
> > > >> > > >> >> > > > > > > > > > >>
> > > >> > > >> >> > > > > > > > > > >
> > > >> > > >> >> > > > > > > > > > >
> > > >> > > >> >> > > > > > > > > >
> > > >> > > >> >> > > > > > > > >
> > > >> > > >> >> > > > > > > >
> > > >> > > >> >> > > > > > >
> > > >> > > >> >> > > > > >
> > > >> > > >> >> > > > >
> > > >> > > >> >> > > >
> > > >> > > >> >> > >
> > > >> > > >> >> >
> > > >> > > >> >>
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >>
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

This is a question you should ask on the Bigtop mailing list.

--
sent from mobile
On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <pr...@apache.org> wrote:

> Yes. In case of RPM, the only restriction on version is that it can not
> contain '-'.
> However, as a workaround, one can use 'incubating' as a part of %{release}.
>
> Version: 3.3.0
> Release: incubating
>
> apex-3.3.0-incubating.noarch.rpm
>
> It is the same practice followed for pre-release packages e.g. with rcX,
> alphaX, betaX being used in %{release}.
>
> Regards,
> Pradeep A. Dalvi
>
> On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Hi Everyone,
> >
> > In one of the previous mails in this thread, we decided that version of
> the
> > package will be "3.3.0-incubating".
> > The problem that I'm facing is in RPM base version "-" is not allowed.
> >
> > Can we have package version to "3.3.0"?
> > i.e. the packages created will be as follows:
> > For RPM: apex-3.3.0-1.noarch.rpm
> > For DEB: apex_3.3.0-1_all.deb
> >
> > There is another integration for Zepplin in Bigtop which is incubating
> and
> > is having base version as "0.5.6".
> >
> > Please let me know your thoughts.
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> > > Thanks everyone for the feedback.
> > >
> > > Also as Ram and David mentioned, I'll start creating the man page as
> > well.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <th...@datatorrent.com>
> > > wrote:
> > >
> > >> I think we already agreed that we will retain the current script name
> in
> > >> this bigtop iteration. There is a ticket for renaming, add you
> > suggestions
> > >> there.
> > >>
> > >> --
> > >> sent from mobile
> > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <sa...@datatorrent.com>
> > wrote:
> > >>
> > >> > +1 for Symlink from apex to dtcli or start the transition process by
> > >> making
> > >> > dtcli as the symlink to apex cli.
> > >> >
> > >> > Renaming is a breaking change should be done during major version
> > number
> > >> > change.
> > >> >
> > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <da...@datatorrent.com>
> > >> wrote:
> > >> >
> > >> > > Apologies for hitting the "Send" button too early.
> > >> > >
> > >> > > What is the scope of the name change?  If we simply have a symlink
> > >> from
> > >> > > apex to dtcli, then it should have no backward compatibility
> problem
> > >> at
> > >> > > all.
> > >> > >
> > >> > > But the proper name change would involve:
> > >> > > - Change the name of DTCli class to ApexCli
> > >> > > - Change the prompt from dt> to apex>
> > >> > > - Change the initial greeting when running the CLI from something
> > like
> > >> > "DT
> > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > >> > >
> > >> > > The above changes may break existing scripts that rely on the the
> > >> string
> > >> > > "DT".
> > >> > > But if folks are okay with this potential problem, I am okay with
> > the
> > >> > name
> > >> > > change.
> > >> > >
> > >> > > For the man page, we can start with the content of this page:
> > >> > >
> > >> > > http://docs.datatorrent.com/dtcli/
> > >> > >
> > >> > > Plus the command line options when doing dtcli -h:
> > >> > >
> > >> > > usage: DTCli
> > >> > >  -e <arg>    Commands are read from the argument
> > >> > >  -f <arg>    Use the specified prompt at all time
> > >> > >  -h          Print this help
> > >> > >  -kp <arg>   Use the specified kerberos principal
> > >> > >  -kt <arg>   Use the specified kerberos keytab
> > >> > >  -p <arg>    JSONP padding function
> > >> > >  -r          JSON Raw mode
> > >> > >  -v          Verbose mode level 1
> > >> > >  -vv         Verbose mode level 2
> > >> > >  -vvv        Verbose mode level 3
> > >> > >  -vvvv       Verbose mode level 4
> > >> > >
> > >> > > David
> > >> > >
> > >> > >
> > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <david@datatorrent.com
> >
> > >> > wrote:
> > >> > >
> > >> > > > What is the scope of the name change?  If we simply have a
> symlink
> > >> from
> > >> > > > apex to dtcli, then it should have no backward compatibility
> > >> problem at
> > >> > > all.
> > >> > > >
> > >> > > > But the proper name change would involve:
> > >> > > > - Change the name of DTCli class to ApexCli
> > >> > > > - Change the prompt from dt> to apex>
> > >> > > > - Change the initial greeting when running the CLI from
> something
> > >> like
> > >> > > "DT
> > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > >> > > >
> > >> > > > The above changes may break existing scripts that rely on the
> the
> > >> > string
> > >> > > > "DT".
> > >> > > > But if folks are okay with this potential problem, I am okay
> with
> > >> the
> > >> > > name
> > >> > > > change.
> > >> > > >
> > >> > > > For the man page, we can start with the content of this page:
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> > >> chinmay@apache.org
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > >> Here is a suggestion I have related to name change:
> > >> > > >>
> > >> > > >> 1. While packaging we change the name of "dtcli" script to
> > "apex".
> > >> > This
> > >> > > is
> > >> > > >> done so that in future when we change the name to "apex", users
> > of
> > >> > > bigtop
> > >> > > >> apex don't have to transition much.
> > >> > > >> 2. We also keep a symlink named dtcli which points to apex
> > script.
> > >> > This
> > >> > > is
> > >> > > >> for backward compatibility.
> > >> > > >> 3. We still keep conf/dt-env.sh for first integration in
> bigtop.
> > >> This
> > >> > is
> > >> > > >> in
> > >> > > >> the interest of not changing the content of dtcli or apex
> script.
> > >> > > >> 4. In apex 3.4.0, we take care of all these naming related
> > changes
> > >> and
> > >> > > >> update bigtop repository later to remove references to dtcli
> all
> > >> > > together.
> > >> > > >>
> > >> > > >> Please share your thoughts on above approach.
> > >> > > >>
> > >> > > >> Also, please share what could be the man page content for
> "apex".
> > >> > > >>
> > >> > > >> Thanks,
> > >> > > >> Chinmay.
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> > >> > chinmay@apache.org>
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> > Yes.. It'll be build from source tar downloaded from one of
> the
> > >> > apache
> > >> > > >> > mirror: apache.osuosl.org
> > >> > > >> >
> > >> > > >> > The reason for dt-env.sh is it's sourced from dtcli. This
> > leaves
> > >> us
> > >> > 3
> > >> > > >> > options:
> > >> > > >> > If we should not have any dt-env.sh, dtcli would need a
> change
> > >> while
> > >> > > >> > packaging.
> > >> > > >> >
> > >> > > >> > But if we want to use dtcli as it is, then we would need
> > >> dt-env.sh
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> > >> > thomas@datatorrent.com
> > >> > > >
> > >> > > >> > wrote:
> > >> > > >> >
> > >> > > >> >> Looks good overall, though there shouldn't be any dt-env.sh
> > >> > > >> >>
> > >> > > >> >> What will the bigtop package be built from, the source tar?
> > >> > > >> >>
> > >> > > >> >>
> > >> > > >> >>
> > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > >> > > chinmay@apache.org
> > >> > > >> >
> > >> > > >> >> wrote:
> > >> > > >> >>
> > >> > > >> >> > Hi Thomas,
> > >> > > >> >> >
> > >> > > >> >> > Thanks for the feedback.
> > >> > > >> >> >
> > >> > > >> >> > First, we're not changing the name anywhere. We'll follow
> > what
> > >> > > >> >> currently is
> > >> > > >> >> > in source tarball for 3.3.0-incubating version.
> > >> > > >> >> >
> > >> > > >> >> > Secondly, I've mentioned a directory structure below which
> > is
> > >> > > inline
> > >> > > >> >> with
> > >> > > >> >> > other existing integrations with bigtop.
> > >> > > >> >> >
> > >> > > >> >> > The need for each file is as follows:
> > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> > >> engine/src/main/scripts/
> > >> > > of
> > >> > > >> >> source
> > >> > > >> >> > code apex 3.3.0-incubating.
> > >> > > >> >> >
> > >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This
> > searches
> > >> for
> > >> > > >> hadoop
> > >> > > >> >> > binary path and exports an env variable for dtcli to use
> it.
> > >> > > >> >> >
> > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is extracted in a
> > >> similar
> > >> > > >> way to
> > >> > > >> >> > how DT Community Edition finds dependency jars.
> > >> > > >> >> > Please note that this is the first iteration list of jars.
> > I'm
> > >> > > >> trying to
> > >> > > >> >> > narrow this down to only those which are really required.
> > >> > > >> >> > The test that I'm running to check if dtcli runs fine with
> > >> given
> > >> > > >> >> dependency
> > >> > > >> >> > is to launch+shutdown+kill+status for pi demo.
> > >> > > >> >> >
> > >> > > >> >> > Above is the directory structure which is required for CLI
> > to
> > >> > work.
> > >> > > >> >> > What aniruddha and pradeep has mentioned are some
> additional
> > >> > files
> > >> > > >> which
> > >> > > >> >> > makes the like of administrator easier. For eg. /etc/skel
> > >> > > >> >> >
> > >> > > >> >> > Please let me know if this is inline with what you're
> > >> thinking.
> > >> > > >> >> >
> > >> > > >> >> > Thanks,
> > >> > > >> >> > Chinmay.
> > >> > > >> >> >
> > >> > > >> >> >
> > >> > > >> >> > .
> > >> > > >> >> > |-- bin
> > >> > > >> >> > |   `-- dtcli
> > >> > > >> >> > |-- conf
> > >> > > >> >> > |   `-- dt-env.sh
> > >> > > >> >> > `-- lib
> > >> > > >> >> >     |-- ant-1.9.2.jar
> > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > >> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > >> > > >> >> >     |-- apex-engine.jar
> > >> > > >> >> >     |-- apex-engine-tests.jar
> > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> > >> > > >> >> >     |-- bval-core-0.5.jar
> > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > >> > > >> >> >     |-- commons-codec-1.10.jar
> > >> > > >> >> >     |-- commons-lang3-3.1.jar
> > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > >> > > >> >> >     |-- httpclient-4.3.5.jar
> > >> > > >> >> >     |-- httpcore-4.3.2.jar
> > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > >> > > >> >> >     |-- javax.servlet-3.1.jar
> > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > >> > > >> >> >     |-- jersey-client-1.9.jar
> > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > >> > > >> >> >     |-- jline-2.11.jar
> > >> > > >> >> >     |-- kryo-2.24.0.jar
> > >> > > >> >> >     |-- mbassador-1.1.9.jar
> > >> > > >> >> >     |-- minlog-1.2.jar
> > >> > > >> >> >     |-- netlet-1.2.0.jar
> > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> > >> > > >> >> >     |-- objenesis-2.1.jar
> > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > >> > > >> >> >     `-- zip4j-1.3.2.jar
> > >> > > >> >> >
> > >> > > >> >> > 3 directories, 40 files
> > >> > > >> >> >
> > >> > > >> >> >
> > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > >> > > >> thomas@datatorrent.com>
> > >> > > >> >> > wrote:
> > >> > > >> >> >
> > >> > > >> >> > > Chinmay,
> > >> > > >> >> > >
> > >> > > >> >> > > Before discussing where to put the files, let's make
> sure
> > >> they
> > >> > > are
> > >> > > >> >> really
> > >> > > >> >> > > needed for the operation of the CLI. As for names,
> > anything
> > >> new
> > >> > > >> needs
> > >> > > >> >> to
> > >> > > >> >> > > reflect Apex in the name and should follow common
> > >> conventions,
> > >> > > >> >> especially
> > >> > > >> >> > > in Bigtop where there are many existing integrations to
> > look
> > >> > at.
> > >> > > >> >> > >
> > >> > > >> >> > > Thanks,
> > >> > > >> >> > > Thomas
> > >> > > >> >> > >
> > >> > > >> >> > >
> > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > >> > > >> >> > > chinmay@datatorrent.com>
> > >> > > >> >> > > wrote:
> > >> > > >> >> > >
> > >> > > >> >> > > > Really good points Pradeep and Aniruddha.
> > >> > > >> >> > > >
> > >> > > >> >> > > > 1. I believe we won't need to change the dtcli
> > >> considering it
> > >> > > >> works
> > >> > > >> >> > with
> > >> > > >> >> > > DT
> > >> > > >> >> > > > Community edition. We can keep the directory structure
> > >> > similar
> > >> > > to
> > >> > > >> >> that.
> > >> > > >> >> > > > dt-env.sh has variables which contains information
> > >> required
> > >> > for
> > >> > > >> >> dtcli
> > >> > > >> >> > to
> > >> > > >> >> > > > launch.
> > >> > > >> >> > > >
> > >> > > >> >> > > > 2. Let me check with bigtop community that whether
> they
> > >> > > >> facilitate
> > >> > > >> >> the
> > >> > > >> >> > > > installation of rpms/debs before user is created. In
> > >> either
> > >> > > case,
> > >> > > >> >> > current
> > >> > > >> >> > > > dtcli creates a .dt folder in home directory. Also
> > before
> > >> > > putting
> > >> > > >> >> > > anything
> > >> > > >> >> > > > in /etc/skel we need to define what are the default
> > >> contents
> > >> > > that
> > >> > > >> >> > should
> > >> > > >> >> > > go
> > >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults, probably we
> > >> should
> > >> > > not
> > >> > > >> >> > > > explicitly add it in /etc/skel.
> > >> > > >> >> > > >
> > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This way
> > contents
> > >> of
> > >> > > >> >> dt-env.sh
> > >> > > >> >> > > are
> > >> > > >> >> > > > present as env variables. But I see a catch there.
> > Adding
> > >> > > >> dt-env.sh
> > >> > > >> >> to
> > >> > > >> >> > > > /etc/profile.d would make all the variables available
> at
> > >> > > runtime
> > >> > > >> all
> > >> > > >> >> > the
> > >> > > >> >> > > > time. I feel a little skeptical about that. Maybe a
> > >> possible
> > >> > > >> >> collision
> > >> > > >> >> > > can
> > >> > > >> >> > > > occur with other application vars.
> > >> > > >> >> > > > Moreover, current dtcli does source
> "../conf/dt-env.sh"
> > >> and
> > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway
> > >> available.
> > >> > > >> >> > > >
> > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is happening at
> > Bigtop
> > >> > > >> >> > > Jira/repository.
> > >> > > >> >> > > > Packaging related code usually goes there. (That's
> what
> > >> all
> > >> > the
> > >> > > >> >> > > components
> > >> > > >> >> > > > in bigtop does).
> > >> > > >> >> > > > Having said that, once a PR is created, I'll be
> sharing
> > >> the
> > >> > > link
> > >> > > >> of
> > >> > > >> >> PR
> > >> > > >> >> > > > here, so that apex community as well can review it.
> > >> > > >> >> > > >
> > >> > > >> >> > > > Thanks,
> > >> > > >> >> > > > Chinmay.
> > >> > > >> >> > > >
> > >> > > >> >> > > >
> > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > >> > > >> >> > > >
> > >> > > >> >> > > > > +1 on suggestions and approach.
> > >> > > >> >> > > > > We may need to iron out details about exact paths
> etc.
> > >> > > >> >> > > > > Which can be done on jira / PR comments.
> > >> > > >> >> > > > > Is that right @dev?
> > >> > > >> >> > > > >
> > >> > > >> >> > > > >
> > >> > > >> >> > > > >
> > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > >> > > >> >> > apache@pradeepdalvi.com>
> > >> > > >> >> > > > > wrote:
> > >> > > >> >> > > > >
> > >> > > >> >> > > > > > Inline comments...
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
> Kolhatkar <
> > >> > > >> >> > > > > > chinmay@datatorrent.com>
> > >> > > >> >> > > > > > wrote:
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > > > @Thomas, Not all the jar present in DT community
> > >> > edition
> > >> > > >> will
> > >> > > >> >> be
> > >> > > >> >> > > > > included
> > >> > > >> >> > > > > > > there.
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > > > Here is the list of jars I found from common
> > >> > dependencies
> > >> > > >> >> between
> > >> > > >> >> > > of
> > >> > > >> >> > > > > apex
> > >> > > >> >> > > > > > > and DT Community edition:
> > >> > > >> >> > > > > > > netlet-1.2.0.jar
> > >> > > >> >> > > > > > > kryo-2.24.0.jar
> > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > >> > > >> >> > > > > > > bval-core-0.5.jar
> > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> > >> > > >> >> > > > > > > commons-codec-1.10.jar
> > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> > >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > >> > > >> >> > > > > > > jline-2.11.jar
> > >> > > >> >> > > > > > > ant-1.9.2.jar
> > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > > > Ofcourse, I'll be running some tests do check
> that
> > >> > dtcli
> > >> > > >> works
> > >> > > >> >> > > > properly
> > >> > > >> >> > > > > > for
> > >> > > >> >> > > > > > > launch+shutdown+kill of apps with only these
> > >> libraries
> > >> > > >> >> present in
> > >> > > >> >> > > > > > isolation
> > >> > > >> >> > > > > > > without dependency on local m2.
> > >> > > >> >> > > > > > > I'm believe that there are unwanted jars which
> are
> > >> used
> > >> > > for
> > >> > > >> >> > compile
> > >> > > >> >> > > > > time
> > >> > > >> >> > > > > > > dependency and not runtime in above list which I
> > can
> > >> > drop
> > >> > > >> to
> > >> > > >> >> keep
> > >> > > >> >> > > > > package
> > >> > > >> >> > > > > > > size minimal.
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > > > @Anniruddha,
> > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
> > >> dt-env.sh is
> > >> > > >> >> required.
> > >> > > >> >> > > > Only
> > >> > > >> >> > > > > > > required values I see in it are classpath. and
> > >> dtcli is
> > >> > > >> >> capable
> > >> > > >> >> > of
> > >> > > >> >> > > > > > building
> > >> > > >> >> > > > > > > it on the fly.
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how
> dtcli
> > >> can
> > >> > > find
> > >> > > >> >> the
> > >> > > >> >> > jar
> > >> > > >> >> > > > > path,
> > >> > > >> >> > > > > > > dt-env.sh and all such conf files. If we need a
> > >> change
> > >> > in
> > >> > > >> >> dtcli,
> > >> > > >> >> > > then
> > >> > > >> >> > > > > > > community's opinion is required for whether
> dtcli
> > >> > should
> > >> > > >> >> change
> > >> > > >> >> > in
> > >> > > >> >> > > > our
> > >> > > >> >> > > > > > repo
> > >> > > >> >> > > > > > > or a copy of that with required changes exist in
> > >> bigtop
> > >> > > >> repo
> > >> > > >> >> > until
> > >> > > >> >> > > we
> > >> > > >> >> > > > > > make
> > >> > > >> >> > > > > > > the dtcli generic enough.
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > > Can we have these file paths set using environment
> > >> > > variables
> > >> > > >> >> with
> > >> > > >> >> > > some
> > >> > > >> >> > > > > > default values in dtcli?
> > >> > > >> >> > > > > > Then we can set such params in dt-env.sh.
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file different
> from
> > >> > > >> >> dt-site.xml?
> > >> > > >> >> > Not
> > >> > > >> >> > > > > sure
> > >> > > >> >> > > > > > if
> > >> > > >> >> > > > > > > adding in /etc/skel would help. Files & Dirs
> from
> > >> > > /etc/skel
> > >> > > >> >> are
> > >> > > >> >> > > > copied
> > >> > > >> >> > > > > to
> > >> > > >> >> > > > > > > home of new user when useradd program is called.
> > But
> > >> > for
> > >> > > >> >> existing
> > >> > > >> >> > > > users
> > >> > > >> >> > > > > > > this won't be of any use. Moreover, dtcli
> creates
> > >> > ~/.dt/
> > >> > > on
> > >> > > >> >> the
> > >> > > >> >> > fly
> > >> > > >> >> > > > if
> > >> > > >> >> > > > > > not
> > >> > > >> >> > > > > > > encountered for the first time. Again correct me
> > if
> > >> I'm
> > >> > > >> wrong.
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > > Yes, you are right. However this would probably be
> > >> > > necessary
> > >> > > >> >> step
> > >> > > >> >> > for
> > >> > > >> >> > > > > > rpm/deb, as they may also get installed during OS
> > >> install
> > >> > > and
> > >> > > >> >> > before
> > >> > > >> >> > > > user
> > >> > > >> >> > > > > > accounts were created.
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not
> sure
> > >> its
> > >> > the
> > >> > > >> best
> > >> > > >> >> > > idea.
> > >> > > >> >> > > > > > We'll
> > >> > > >> >> > > > > > > anyway put dtcli in location which is by default
> > >> > present
> > >> > > in
> > >> > > >> >> path
> > >> > > >> >> > > i.e.
> > >> > > >> >> > > > > > > /usr/bin.
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > > For other env variables specific to apex, I'll
> > prefer
> > >> to
> > >> > > use
> > >> > > >> >> > > dt-env.sh
> > >> > > >> >> > > > > and
> > >> > > >> >> > > > > > > source it in dtcli rather than changing bashrc
> > >> etc...
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall
> > be
> > >> the
> > >> > > >> >> cleaner
> > >> > > >> >> > way
> > >> > > >> >> > > > to
> > >> > > >> >> > > > > > achieve the same.
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > > Thanks,
> > >> > > >> >> > > > > > > Chinmay.
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
> > Thombare
> > >> <
> > >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > > > > Hi,
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > > @Chinmay,
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > > We need to consider following:
> > >> > > >> >> > > > > > > > System wide default config files can be
> located
> > at
> > >> > > >> following
> > >> > > >> >> > > > > locations:
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change
> > dtcli
> > >> > > >> behaviour
> > >> > > >> >> on
> > >> > > >> >> > > how
> > >> > > >> >> > > > it
> > >> > > >> >> > > > > > > finds
> > >> > > >> >> > > > > > > > those locations)
> > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change
> > dtcli
> > >> > > >> >> behaviour on
> > >> > > >> >> > > how
> > >> > > >> >> > > > > it
> > >> > > >> >> > > > > > > > finds those locations)
> > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files
> (for
> > >> new
> > >> > > users
> > >> > > >> >> that
> > >> > > >> >> > > will
> > >> > > >> >> > > > > be
> > >> > > >> >> > > > > > > > created in system in future.)
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > > We may also have to modify bashrc /
> bashprofile
> > >> for
> > >> > > >> >> population
> > >> > > >> >> > > any
> > >> > > >> >> > > > > > > > variables that are required.
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > > @dev, please put in your comments /
> suggestion.
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > > Thanks,
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > > Aniruddha
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise
> <
> > >> > > >> >> > > > > thomas@datatorrent.com>
> > >> > > >> >> > > > > > > > wrote:
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > > > > -->
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> > >> Kolhatkar <
> > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > wrote:
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > > Hello Everyone!!
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > > Continuing with packaging effort (rpm+deb)
> > of
> > >> > apex,
> > >> > > >> here
> > >> > > >> >> > are
> > >> > > >> >> > > > some
> > >> > > >> >> > > > > > > > > proposals
> > >> > > >> >> > > > > > > > > > about package structure etc..
> > >> > > >> >> > > > > > > > > > Before posting it on bbigtop mailing
> list, I
> > >> have
> > >> > > >> some
> > >> > > >> >> > > question
> > >> > > >> >> > > > > for
> > >> > > >> >> > > > > > > > apex
> > >> > > >> >> > > > > > > > > > community.
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > > Proposed Directory structure of apex
> package
> > >> for
> > >> > > both
> > >> > > >> >> deb &
> > >> > > >> >> > > > rpm:
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > >> > > >> >> > > > > > > > > >
> /usr/lib/apex/lib/apex-engine-<version>.jar
> > >> > > >> >> > > > > > > > > >
> > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > >> > > >> >> > > > > > > > > >
> /usr/lib/apex/lib/apex-common-<version>.jar
> > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > >> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > >> > > >> >> > > > > > > > > >
> > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> > >> > > >> >> > > > > > > > > > <other dependent jars> = All the 3rd party
> > >> jars
> > >> > > which
> > >> > > >> >> are
> > >> > > >> >> > > > > required
> > >> > > >> >> > > > > > > for
> > >> > > >> >> > > > > > > > > apex
> > >> > > >> >> > > > > > > > > > to run. Usually the dependencies are
> > packaged
> > >> as
> > >> > > >> part of
> > >> > > >> >> > > > rpm/deb
> > >> > > >> >> > > > > by
> > >> > > >> >> > > > > > > any
> > >> > > >> >> > > > > > > > > > software in bigtop.
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > Can you specify what those jars are in
> Bigtop
> > >> > > context?
> > >> > > >> >> Same
> > >> > > >> >> > as
> > >> > > >> >> > > > > > shipped
> > >> > > >> >> > > > > > > > with
> > >> > > >> >> > > > > > > > > DT community addition under lib/ ?
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of
> > corresponding
> > >> of
> > >> > > 3rd
> > >> > > >> >> party
> > >> > > >> >> > > jars
> > >> > > >> >> > > > > > which
> > >> > > >> >> > > > > > > > > needs
> > >> > > >> >> > > > > > > > > > to included while packaging.
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > > Questions related to this:
> > >> > > >> >> > > > > > > > > > 1. Should we call the cli of apex as
> "apex"
> > >> > instead
> > >> > > >> of
> > >> > > >> >> > > "dtcli"
> > >> > > >> >> > > > in
> > >> > > >> >> > > > > > > > bigtop
> > >> > > >> >> > > > > > > > > > package?
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > I think we should keep the name until we are
> > >> able
> > >> > to
> > >> > > >> >> change
> > >> > > >> >> > it
> > >> > > >> >> > > in
> > >> > > >> >> > > > > > Apex.
> > >> > > >> >> > > > > > > > > Otherwise this may get confusing.
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > > 2. I see that all softwares in bigtop have
> > man
> > >> > page
> > >> > > >> for
> > >> > > >> >> > their
> > >> > > >> >> > > > > > > > > executables.
> > >> > > >> >> > > > > > > > > > I think we should have it too for dtcli.
> Is
> > >> there
> > >> > > any
> > >> > > >> >> > > > > documentation
> > >> > > >> >> > > > > > > > > which I
> > >> > > >> >> > > > > > > > > > can convert to man page? or can I use
> output
> > >> of
> > >> > > >> "dtcli
> > >> > > >> >> > > --help"?
> > >> > > >> >> > > > > > > > > > 3. Do we want to call version of apex in
> > >> Bigtop
> > >> > as
> > >> > > >> >> 3.3.0 OR
> > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > Has to be -incubating
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of bigtop to
> > >> depend
> > >> > on
> > >> > > >> >> 2.7.1
> > >> > > >> >> > > > > version
> > >> > > >> >> > > > > > of
> > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that we see
> with
> > >> this
> > >> > > >> >> > dependency?
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > Should work without changed. I thought we
> > >> certified
> > >> > > >> >> against
> > >> > > >> >> > > > 2.7.0?
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > > 5. Following is the apache mirror from
> which
> > >> > bigtop
> > >> > > >> will
> > >> > > >> >> > pick
> > >> > > >> >> > > > the
> > >> > > >> >> > > > > > > apex
> > >> > > >> >> > > > > > > > > > source for compilation and packaging.
> Please
> > >> > > correct
> > >> > > >> if
> > >> > > >> >> > > > > incorrect:
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > >
> > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror
> > >> used
> > >> > by
> > >> > > >> all
> > >> > > >> >> the
> > >> > > >> >> > > > > > softwares
> > >> > > >> >> > > > > > > in
> > >> > > >> >> > > > > > > > > > bigtop.
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > > Please share your opinion.
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > > Thanks,
> > >> > > >> >> > > > > > > > > > Chinmay.
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> > >> > Kolhatkar
> > >> > > <
> > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > >> > > >> >> > > > > > > > > > wrote:
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > >> > > >> >> > > > > > > > > > >
> > >> > > >> >> > > > > > > > > > > I've already started discussion on
> bigtop
> > >> dev
> > >> > > >> mailing
> > >> > > >> >> > list
> > >> > > >> >> > > > for
> > >> > > >> >> > > > > > > > > > > integration. Also created a Jira for the
> > >> same.
> > >> > > For
> > >> > > >> >> this
> > >> > > >> >> > > > > > communities
> > >> > > >> >> > > > > > > > > > > reference, here is the bigtop Jira:
> > >> > BIGTOP-2313.
> > >> > > >> >> > > > > > > > > > >
> > >> > > >> >> > > > > > > > > > >
> > >> > > >> >> > > > > > > > > > >
> > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas
> > >> Weise <
> > >> > > >> >> > > > > > > thomas@datatorrent.com
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > > > > > wrote:
> > >> > > >> >> > > > > > > > > > >
> > >> > > >> >> > > > > > > > > > >> Chinmay,
> > >> > > >> >> > > > > > > > > > >>
> > >> > > >> >> > > > > > > > > > >> I don't see anything under
> prerequisites
> > >> that
> > >> > > >> would
> > >> > > >> >> be a
> > >> > > >> >> > > > > > problem.
> > >> > > >> >> > > > > > > We
> > >> > > >> >> > > > > > > > > > >> looked
> > >> > > >> >> > > > > > > > > > >> at the ASF licencing compatibility as
> > part
> > >> of
> > >> > > >> >> becoming
> > >> > > >> >> > an
> > >> > > >> >> > > > > > > incubator
> > >> > > >> >> > > > > > > > > > >> project.
> > >> > > >> >> > > > > > > > > > >>
> > >> > > >> >> > > > > > > > > > >> Please focus on the packaging during
> the
> > >> next
> > >> > > >> weeks.
> > >> > > >> >> > Since
> > >> > > >> >> > > > the
> > >> > > >> >> > > > > > > work
> > >> > > >> >> > > > > > > > > will
> > >> > > >> >> > > > > > > > > > >> be
> > >> > > >> >> > > > > > > > > > >> part of Bigtop, related discussions and
> > >> JIRAs
> > >> > > >> should
> > >> > > >> >> > also
> > >> > > >> >> > > be
> > >> > > >> >> > > > > > > there.
> > >> > > >> >> > > > > > > > > > >>
> > >> > > >> >> > > > > > > > > > >> Would be good to have the packaging in
> > >> place
> > >> > by
> > >> > > >> end
> > >> > > >> >> Feb.
> > >> > > >> >> > > > > > > > > > >>
> > >> > > >> >> > > > > > > > > > >> Thanks,
> > >> > > >> >> > > > > > > > > > >> Thomas
> > >> > > >> >> > > > > > > > > > >>
> > >> > > >> >> > > > > > > > > > >>
> > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM,
> Chinmay
> > >> > > >> Kolhatkar <
> > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> > >> > > >> >> > > > > > > > > > >> wrote:
> > >> > > >> >> > > > > > > > > > >>
> > >> > > >> >> > > > > > > > > > >> > Hi All,
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >> > We're planning a work on adding
> Apache
> > >> Apex
> > >> > > as a
> > >> > > >> >> > > component
> > >> > > >> >> > > > > to
> > >> > > >> >> > > > > > > > Apache
> > >> > > >> >> > > > > > > > > > >> > Bigtop.
> > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging system for
> the
> > >> > Apache
> > >> > > >> big
> > >> > > >> >> data
> > >> > > >> >> > > > > > > ecosystem.
> > >> > > >> >> > > > > > > > > > >> Several
> > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most recently
> > EMR.
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira task in
> > >> APEXCORE
> > >> > for
> > >> > > >> the
> > >> > > >> >> > same:
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >> > Proposed plan of execution is as
> > follows:
> > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> > >> > > expectation
> > >> > > >> >> for
> > >> > > >> >> > new
> > >> > > >> >> > > > > > > > components
> > >> > > >> >> > > > > > > > > to
> > >> > > >> >> > > > > > > > > > >> get
> > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
> > >> > > >> >> > > > > > > > > > >> > Here is the list of it:
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >>
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > >
> > >> > > >> >> > > >
> > >> > > >> >> > >
> > >> > > >> >> >
> > >> > > >> >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >> > Most of them seems to be standard ASF
> > >> based
> > >> > > >> >> > > requirements,
> > >> > > >> >> > > > > but
> > >> > > >> >> > > > > > > few
> > >> > > >> >> > > > > > > > > need
> > >> > > >> >> > > > > > > > > > >> to
> > >> > > >> >> > > > > > > > > > >> > be checked for:
> > >> > > >> >> > > > > > > > > > >> > 1. Software projects are expected to
> be
> > >> > > Licensed
> > >> > > >> >> under
> > >> > > >> >> > > > > Apache
> > >> > > >> >> > > > > > > > > License,
> > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their dependencies
> are
> > >> > > expected
> > >> > > >> >> to be
> > >> > > >> >> > > > > > > compatible
> > >> > > >> >> > > > > > > > > with
> > >> > > >> >> > > > > > > > > > >> this
> > >> > > >> >> > > > > > > > > > >> > license)
> > >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need
> to
> > >> > check
> > >> > > if
> > >> > > >> >> > > > > dependencies
> > >> > > >> >> > > > > > of
> > >> > > >> >> > > > > > > > > Apex
> > >> > > >> >> > > > > > > > > > >> are
> > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess
> > >> would
> > >> > > be a
> > >> > > >> >> > > > > verification
> > >> > > >> >> > > > > > > > check.
> > >> > > >> >> > > > > > > > > > >> > 2. Software projects are expected to
> be
> > >> > > >> compatible
> > >> > > >> >> > with
> > >> > > >> >> > > > all
> > >> > > >> >> > > > > of
> > >> > > >> >> > > > > > > the
> > >> > > >> >> > > > > > > > > > >> > supported platforms that Bigtop
> > >> distribution
> > >> > > is
> > >> > > >> >> > > targeting
> > >> > > >> >> > > > > > > > > > >> >     - This needs verifying whether
> our
> > >> > > software
> > >> > > >> >> runs
> > >> > > >> >> > > fine
> > >> > > >> >> > > > in
> > >> > > >> >> > > > > > > > > centos-6
> > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
> > debian-8
> > >> > > >> >> > opensuse-13.2.
> > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that should be
> > added
> > >> for
> > >> > > >> >> > deployment.
> > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test artifacts
> which
> > >> goes
> > >> > > >> beyond
> > >> > > >> >> > > smoke
> > >> > > >> >> > > > > test
> > >> > > >> >> > > > > > > > > > >> >     - These are basically the
> > integration
> > >> > > tests
> > >> > > >> for
> > >> > > >> >> > > > > > verification
> > >> > > >> >> > > > > > > > > after
> > >> > > >> >> > > > > > > > > > >> the
> > >> > > >> >> > > > > > > > > > >> > deployment. This is a soft
> requirement,
> > >> but
> > >> > > aim
> > >> > > >> is
> > >> > > >> >> to
> > >> > > >> >> > > > > achieve
> > >> > > >> >> > > > > > > this
> > >> > > >> >> > > > > > > > > as
> > >> > > >> >> > > > > > > > > > >> well
> > >> > > >> >> > > > > > > > > > >> > or at least have explanation why not
> to
> > >> > > include.
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >> > If there are any from the link which
> > >> > > explicitly
> > >> > > >> >> needs
> > >> > > >> >> > to
> > >> > > >> >> > > > be
> > >> > > >> >> > > > > > > > checked
> > >> > > >> >> > > > > > > > > > >> other
> > >> > > >> >> > > > > > > > > > >> > than above 4, please let us know.
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as component to
> > >> Bigtop*
> > >> > > >> >> > > > > > > > > > >> > From one of the mail archive of
> Bigtop,
> > >> it
> > >> > was
> > >> > > >> >> learnt
> > >> > > >> >> > > that
> > >> > > >> >> > > > > the
> > >> > > >> >> > > > > > > > > bigtop
> > >> > > >> >> > > > > > > > > > >> > community want to see the addition of
> > new
> > >> > > >> >> components
> > >> > > >> >> > in
> > >> > > >> >> > > > > > phases.
> > >> > > >> >> > > > > > > > Here
> > >> > > >> >> > > > > > > > > > are
> > >> > > >> >> > > > > > > > > > >> > the phases:
> > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> > >> > > >> >> > > > > > > > > > >> >     - This needs creating of package
> > i.e.
> > >> > rpm
> > >> > > &
> > >> > > >> deb
> > >> > > >> >> > > files.
> > >> > > >> >> > > > > > > > > > >> >     - documentations/READMEs,
> LICENSE,
> > >> > > >> DISCLAMER,
> > >> > > >> >> > NOTES
> > >> > > >> >> > > > etc
> > >> > > >> >> > > > > if
> > >> > > >> >> > > > > > > any
> > >> > > >> >> > > > > > > > > > >> needed.
> > >> > > >> >> > > > > > > > > > >> >     - Any documentation that need to
> be
> > >> > added
> > >> > > to
> > >> > > >> >> > > > > distribution
> > >> > > >> >> > > > > > of
> > >> > > >> >> > > > > > > > our
> > >> > > >> >> > > > > > > > > > >> > software.
> > >> > > >> >> > > > > > > > > > >> >     - Any license information of
> > >> > dependencies
> > >> > > >> >> required
> > >> > > >> >> > > to
> > >> > > >> >> > > > be
> > >> > > >> >> > > > > > > added
> > >> > > >> >> > > > > > > > > to
> > >> > > >> >> > > > > > > > > > >> > package
> > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > >> > > >> >> > > > > > > > > > >> >     - Adding smoke test for
> packaging.
> > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic
> > >> deployment
> > >> > and
> > >> > > >> >> > > > configuration
> > >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for
> automatic
> > >> > > >> deployment
> > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> > >> > > >> >> > > > > > > > > > >> >     - For verification of
> deployments.
> > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all
> > new
> > >> > > stuff
> > >> > > >> is
> > >> > > >> >> > > > compliant
> > >> > > >> >> > > > > > > with
> > >> > > >> >> > > > > > > > > ASF
> > >> > > >> >> > > > > > > > > > >> > license requirements. If you add code
> > >> > licenses
> > >> > > >> >> under
> > >> > > >> >> > > > > different
> > >> > > >> >> > > > > > > > > > licenses,
> > >> > > >> >> > > > > > > > > > >> > those would need to be listed in the
> > >> NOTICE.
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >> > Please share your thoughts on the
> > >> approach.
> > >> > > >> >> > > > > > > > > > >> > We'll start corresponding
> communication
> > >> on
> > >> > > >> bigtop
> > >> > > >> >> > > mailing
> > >> > > >> >> > > > > list
> > >> > > >> >> > > > > > > as
> > >> > > >> >> > > > > > > > > > well.
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >> > We have some specific
> > >> questions/suggestions
> > >> > > >> >> related to
> > >> > > >> >> > > > what
> > >> > > >> >> > > > > > > should
> > >> > > >> >> > > > > > > > > be
> > >> > > >> >> > > > > > > > > > >> the
> > >> > > >> >> > > > > > > > > > >> > content of the package and what
> should
> > be
> > >> > the
> > >> > > >> smoke
> > >> > > >> >> > > tests,
> > >> > > >> >> > > > > but
> > >> > > >> >> > > > > > > in
> > >> > > >> >> > > > > > > > > the
> > >> > > >> >> > > > > > > > > > >> > interest of not having too much
> content
> > >> > here,
> > >> > > >> we'll
> > >> > > >> >> > put
> > >> > > >> >> > > > the
> > >> > > >> >> > > > > > > > > questions
> > >> > > >> >> > > > > > > > > > >> as a
> > >> > > >> >> > > > > > > > > > >> > separate mail in this mailthread.
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >> > Thanks,
> > >> > > >> >> > > > > > > > > > >> > Chinmay.
> > >> > > >> >> > > > > > > > > > >> >
> > >> > > >> >> > > > > > > > > > >>
> > >> > > >> >> > > > > > > > > > >
> > >> > > >> >> > > > > > > > > > >
> > >> > > >> >> > > > > > > > > >
> > >> > > >> >> > > > > > > > >
> > >> > > >> >> > > > > > > >
> > >> > > >> >> > > > > > >
> > >> > > >> >> > > > > >
> > >> > > >> >> > > > >
> > >> > > >> >> > > >
> > >> > > >> >> > >
> > >> > > >> >> >
> > >> > > >> >>
> > >> > > >> >
> > >> > > >> >
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

["Pradeep A. Dalvi" <pr...@apache.org>]

Yes. In case of RPM, the only restriction on version is that it can not
contain '-'.
However, as a workaround, one can use 'incubating' as a part of %{release}.

Version: 3.3.0
Release: incubating

apex-3.3.0-incubating.noarch.rpm

It is the same practice followed for pre-release packages e.g. with rcX,
alphaX, betaX being used in %{release}.

Regards,
Pradeep A. Dalvi

On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Hi Everyone,
>
> In one of the previous mails in this thread, we decided that version of the
> package will be "3.3.0-incubating".
> The problem that I'm facing is in RPM base version "-" is not allowed.
>
> Can we have package version to "3.3.0"?
> i.e. the packages created will be as follows:
> For RPM: apex-3.3.0-1.noarch.rpm
> For DEB: apex_3.3.0-1_all.deb
>
> There is another integration for Zepplin in Bigtop which is incubating and
> is having base version as "0.5.6".
>
> Please let me know your thoughts.
>
> Thanks,
> Chinmay.
>
>
>
> On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Thanks everyone for the feedback.
> >
> > Also as Ram and David mentioned, I'll start creating the man page as
> well.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> >> I think we already agreed that we will retain the current script name in
> >> this bigtop iteration. There is a ticket for renaming, add you
> suggestions
> >> there.
> >>
> >> --
> >> sent from mobile
> >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <sa...@datatorrent.com>
> wrote:
> >>
> >> > +1 for Symlink from apex to dtcli or start the transition process by
> >> making
> >> > dtcli as the symlink to apex cli.
> >> >
> >> > Renaming is a breaking change should be done during major version
> number
> >> > change.
> >> >
> >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <da...@datatorrent.com>
> >> wrote:
> >> >
> >> > > Apologies for hitting the "Send" button too early.
> >> > >
> >> > > What is the scope of the name change?  If we simply have a symlink
> >> from
> >> > > apex to dtcli, then it should have no backward compatibility problem
> >> at
> >> > > all.
> >> > >
> >> > > But the proper name change would involve:
> >> > > - Change the name of DTCli class to ApexCli
> >> > > - Change the prompt from dt> to apex>
> >> > > - Change the initial greeting when running the CLI from something
> like
> >> > "DT
> >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> >> > >
> >> > > The above changes may break existing scripts that rely on the the
> >> string
> >> > > "DT".
> >> > > But if folks are okay with this potential problem, I am okay with
> the
> >> > name
> >> > > change.
> >> > >
> >> > > For the man page, we can start with the content of this page:
> >> > >
> >> > > http://docs.datatorrent.com/dtcli/
> >> > >
> >> > > Plus the command line options when doing dtcli -h:
> >> > >
> >> > > usage: DTCli
> >> > >  -e <arg>    Commands are read from the argument
> >> > >  -f <arg>    Use the specified prompt at all time
> >> > >  -h          Print this help
> >> > >  -kp <arg>   Use the specified kerberos principal
> >> > >  -kt <arg>   Use the specified kerberos keytab
> >> > >  -p <arg>    JSONP padding function
> >> > >  -r          JSON Raw mode
> >> > >  -v          Verbose mode level 1
> >> > >  -vv         Verbose mode level 2
> >> > >  -vvv        Verbose mode level 3
> >> > >  -vvvv       Verbose mode level 4
> >> > >
> >> > > David
> >> > >
> >> > >
> >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <da...@datatorrent.com>
> >> > wrote:
> >> > >
> >> > > > What is the scope of the name change?  If we simply have a symlink
> >> from
> >> > > > apex to dtcli, then it should have no backward compatibility
> >> problem at
> >> > > all.
> >> > > >
> >> > > > But the proper name change would involve:
> >> > > > - Change the name of DTCli class to ApexCli
> >> > > > - Change the prompt from dt> to apex>
> >> > > > - Change the initial greeting when running the CLI from something
> >> like
> >> > > "DT
> >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> >> > > >
> >> > > > The above changes may break existing scripts that rely on the the
> >> > string
> >> > > > "DT".
> >> > > > But if folks are okay with this potential problem, I am okay with
> >> the
> >> > > name
> >> > > > change.
> >> > > >
> >> > > > For the man page, we can start with the content of this page:
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> >> chinmay@apache.org
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > >> Here is a suggestion I have related to name change:
> >> > > >>
> >> > > >> 1. While packaging we change the name of "dtcli" script to
> "apex".
> >> > This
> >> > > is
> >> > > >> done so that in future when we change the name to "apex", users
> of
> >> > > bigtop
> >> > > >> apex don't have to transition much.
> >> > > >> 2. We also keep a symlink named dtcli which points to apex
> script.
> >> > This
> >> > > is
> >> > > >> for backward compatibility.
> >> > > >> 3. We still keep conf/dt-env.sh for first integration in bigtop.
> >> This
> >> > is
> >> > > >> in
> >> > > >> the interest of not changing the content of dtcli or apex script.
> >> > > >> 4. In apex 3.4.0, we take care of all these naming related
> changes
> >> and
> >> > > >> update bigtop repository later to remove references to dtcli all
> >> > > together.
> >> > > >>
> >> > > >> Please share your thoughts on above approach.
> >> > > >>
> >> > > >> Also, please share what could be the man page content for "apex".
> >> > > >>
> >> > > >> Thanks,
> >> > > >> Chinmay.
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> >> > chinmay@apache.org>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Yes.. It'll be build from source tar downloaded from one of the
> >> > apache
> >> > > >> > mirror: apache.osuosl.org
> >> > > >> >
> >> > > >> > The reason for dt-env.sh is it's sourced from dtcli. This
> leaves
> >> us
> >> > 3
> >> > > >> > options:
> >> > > >> > If we should not have any dt-env.sh, dtcli would need a change
> >> while
> >> > > >> > packaging.
> >> > > >> >
> >> > > >> > But if we want to use dtcli as it is, then we would need
> >> dt-env.sh
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> >> > thomas@datatorrent.com
> >> > > >
> >> > > >> > wrote:
> >> > > >> >
> >> > > >> >> Looks good overall, though there shouldn't be any dt-env.sh
> >> > > >> >>
> >> > > >> >> What will the bigtop package be built from, the source tar?
> >> > > >> >>
> >> > > >> >>
> >> > > >> >>
> >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> >> > > chinmay@apache.org
> >> > > >> >
> >> > > >> >> wrote:
> >> > > >> >>
> >> > > >> >> > Hi Thomas,
> >> > > >> >> >
> >> > > >> >> > Thanks for the feedback.
> >> > > >> >> >
> >> > > >> >> > First, we're not changing the name anywhere. We'll follow
> what
> >> > > >> >> currently is
> >> > > >> >> > in source tarball for 3.3.0-incubating version.
> >> > > >> >> >
> >> > > >> >> > Secondly, I've mentioned a directory structure below which
> is
> >> > > inline
> >> > > >> >> with
> >> > > >> >> > other existing integrations with bigtop.
> >> > > >> >> >
> >> > > >> >> > The need for each file is as follows:
> >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> >> engine/src/main/scripts/
> >> > > of
> >> > > >> >> source
> >> > > >> >> > code apex 3.3.0-incubating.
> >> > > >> >> >
> >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This
> searches
> >> for
> >> > > >> hadoop
> >> > > >> >> > binary path and exports an env variable for dtcli to use it.
> >> > > >> >> >
> >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is extracted in a
> >> similar
> >> > > >> way to
> >> > > >> >> > how DT Community Edition finds dependency jars.
> >> > > >> >> > Please note that this is the first iteration list of jars.
> I'm
> >> > > >> trying to
> >> > > >> >> > narrow this down to only those which are really required.
> >> > > >> >> > The test that I'm running to check if dtcli runs fine with
> >> given
> >> > > >> >> dependency
> >> > > >> >> > is to launch+shutdown+kill+status for pi demo.
> >> > > >> >> >
> >> > > >> >> > Above is the directory structure which is required for CLI
> to
> >> > work.
> >> > > >> >> > What aniruddha and pradeep has mentioned are some additional
> >> > files
> >> > > >> which
> >> > > >> >> > makes the like of administrator easier. For eg. /etc/skel
> >> > > >> >> >
> >> > > >> >> > Please let me know if this is inline with what you're
> >> thinking.
> >> > > >> >> >
> >> > > >> >> > Thanks,
> >> > > >> >> > Chinmay.
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> > .
> >> > > >> >> > |-- bin
> >> > > >> >> > |   `-- dtcli
> >> > > >> >> > |-- conf
> >> > > >> >> > |   `-- dt-env.sh
> >> > > >> >> > `-- lib
> >> > > >> >> >     |-- ant-1.9.2.jar
> >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> >> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> >> > > >> >> >     |-- apex-engine.jar
> >> > > >> >> >     |-- apex-engine-tests.jar
> >> > > >> >> >     |-- async-http-client-1.7.20.jar
> >> > > >> >> >     |-- bval-core-0.5.jar
> >> > > >> >> >     |-- bval-jsr303-0.5.jar
> >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> >> > > >> >> >     |-- commons-codec-1.10.jar
> >> > > >> >> >     |-- commons-lang3-3.1.jar
> >> > > >> >> >     |-- commons-logging-1.1.3.jar
> >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> >> > > >> >> >     |-- httpclient-4.3.5.jar
> >> > > >> >> >     |-- httpcore-4.3.2.jar
> >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> >> > > >> >> >     |-- javax.servlet-3.1.jar
> >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> >> > > >> >> >     |-- jersey-client-1.9.jar
> >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> >> > > >> >> >     |-- jline-2.11.jar
> >> > > >> >> >     |-- kryo-2.24.0.jar
> >> > > >> >> >     |-- mbassador-1.1.9.jar
> >> > > >> >> >     |-- minlog-1.2.jar
> >> > > >> >> >     |-- netlet-1.2.0.jar
> >> > > >> >> >     |-- netty-3.6.6.Final.jar
> >> > > >> >> >     |-- objenesis-2.1.jar
> >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> >> > > >> >> >     `-- zip4j-1.3.2.jar
> >> > > >> >> >
> >> > > >> >> > 3 directories, 40 files
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> >> > > >> thomas@datatorrent.com>
> >> > > >> >> > wrote:
> >> > > >> >> >
> >> > > >> >> > > Chinmay,
> >> > > >> >> > >
> >> > > >> >> > > Before discussing where to put the files, let's make sure
> >> they
> >> > > are
> >> > > >> >> really
> >> > > >> >> > > needed for the operation of the CLI. As for names,
> anything
> >> new
> >> > > >> needs
> >> > > >> >> to
> >> > > >> >> > > reflect Apex in the name and should follow common
> >> conventions,
> >> > > >> >> especially
> >> > > >> >> > > in Bigtop where there are many existing integrations to
> look
> >> > at.
> >> > > >> >> > >
> >> > > >> >> > > Thanks,
> >> > > >> >> > > Thomas
> >> > > >> >> > >
> >> > > >> >> > >
> >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> >> > > >> >> > > chinmay@datatorrent.com>
> >> > > >> >> > > wrote:
> >> > > >> >> > >
> >> > > >> >> > > > Really good points Pradeep and Aniruddha.
> >> > > >> >> > > >
> >> > > >> >> > > > 1. I believe we won't need to change the dtcli
> >> considering it
> >> > > >> works
> >> > > >> >> > with
> >> > > >> >> > > DT
> >> > > >> >> > > > Community edition. We can keep the directory structure
> >> > similar
> >> > > to
> >> > > >> >> that.
> >> > > >> >> > > > dt-env.sh has variables which contains information
> >> required
> >> > for
> >> > > >> >> dtcli
> >> > > >> >> > to
> >> > > >> >> > > > launch.
> >> > > >> >> > > >
> >> > > >> >> > > > 2. Let me check with bigtop community that whether they
> >> > > >> facilitate
> >> > > >> >> the
> >> > > >> >> > > > installation of rpms/debs before user is created. In
> >> either
> >> > > case,
> >> > > >> >> > current
> >> > > >> >> > > > dtcli creates a .dt folder in home directory. Also
> before
> >> > > putting
> >> > > >> >> > > anything
> >> > > >> >> > > > in /etc/skel we need to define what are the default
> >> contents
> >> > > that
> >> > > >> >> > should
> >> > > >> >> > > go
> >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults, probably we
> >> should
> >> > > not
> >> > > >> >> > > > explicitly add it in /etc/skel.
> >> > > >> >> > > >
> >> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This way
> contents
> >> of
> >> > > >> >> dt-env.sh
> >> > > >> >> > > are
> >> > > >> >> > > > present as env variables. But I see a catch there.
> Adding
> >> > > >> dt-env.sh
> >> > > >> >> to
> >> > > >> >> > > > /etc/profile.d would make all the variables available at
> >> > > runtime
> >> > > >> all
> >> > > >> >> > the
> >> > > >> >> > > > time. I feel a little skeptical about that. Maybe a
> >> possible
> >> > > >> >> collision
> >> > > >> >> > > can
> >> > > >> >> > > > occur with other application vars.
> >> > > >> >> > > > Moreover, current dtcli does source "../conf/dt-env.sh"
> >> and
> >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway
> >> available.
> >> > > >> >> > > >
> >> > > >> >> > > > @Aniruddha, the Jira and PR effort is happening at
> Bigtop
> >> > > >> >> > > Jira/repository.
> >> > > >> >> > > > Packaging related code usually goes there. (That's what
> >> all
> >> > the
> >> > > >> >> > > components
> >> > > >> >> > > > in bigtop does).
> >> > > >> >> > > > Having said that, once a PR is created, I'll be sharing
> >> the
> >> > > link
> >> > > >> of
> >> > > >> >> PR
> >> > > >> >> > > > here, so that apex community as well can review it.
> >> > > >> >> > > >
> >> > > >> >> > > > Thanks,
> >> > > >> >> > > > Chinmay.
> >> > > >> >> > > >
> >> > > >> >> > > >
> >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> >> > > >> >> > > >
> >> > > >> >> > > > > +1 on suggestions and approach.
> >> > > >> >> > > > > We may need to iron out details about exact paths etc.
> >> > > >> >> > > > > Which can be done on jira / PR comments.
> >> > > >> >> > > > > Is that right @dev?
> >> > > >> >> > > > >
> >> > > >> >> > > > >
> >> > > >> >> > > > >
> >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> >> > > >> >> > apache@pradeepdalvi.com>
> >> > > >> >> > > > > wrote:
> >> > > >> >> > > > >
> >> > > >> >> > > > > > Inline comments...
> >> > > >> >> > > > > >
> >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> >> > > >> >> > > > > > chinmay@datatorrent.com>
> >> > > >> >> > > > > > wrote:
> >> > > >> >> > > > > >
> >> > > >> >> > > > > > > @Thomas, Not all the jar present in DT community
> >> > edition
> >> > > >> will
> >> > > >> >> be
> >> > > >> >> > > > > included
> >> > > >> >> > > > > > > there.
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > > > Here is the list of jars I found from common
> >> > dependencies
> >> > > >> >> between
> >> > > >> >> > > of
> >> > > >> >> > > > > apex
> >> > > >> >> > > > > > > and DT Community edition:
> >> > > >> >> > > > > > > netlet-1.2.0.jar
> >> > > >> >> > > > > > > kryo-2.24.0.jar
> >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> >> > > >> >> > > > > > > bval-core-0.5.jar
> >> > > >> >> > > > > > > commons-lang3-3.1.jar
> >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> >> > > >> >> > > > > > > httpclient-4.3.5.jar
> >> > > >> >> > > > > > > commons-codec-1.10.jar
> >> > > >> >> > > > > > > zip4j-1.3.2.jar
> >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> >> > > >> >> > > > > > > jline-2.11.jar
> >> > > >> >> > > > > > > ant-1.9.2.jar
> >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> >> > > >> >> > > > > > > mbassador-1.1.9.jar
> >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > > > Ofcourse, I'll be running some tests do check that
> >> > dtcli
> >> > > >> works
> >> > > >> >> > > > properly
> >> > > >> >> > > > > > for
> >> > > >> >> > > > > > > launch+shutdown+kill of apps with only these
> >> libraries
> >> > > >> >> present in
> >> > > >> >> > > > > > isolation
> >> > > >> >> > > > > > > without dependency on local m2.
> >> > > >> >> > > > > > > I'm believe that there are unwanted jars which are
> >> used
> >> > > for
> >> > > >> >> > compile
> >> > > >> >> > > > > time
> >> > > >> >> > > > > > > dependency and not runtime in above list which I
> can
> >> > drop
> >> > > >> to
> >> > > >> >> keep
> >> > > >> >> > > > > package
> >> > > >> >> > > > > > > size minimal.
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > > > @Anniruddha,
> >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
> >> dt-env.sh is
> >> > > >> >> required.
> >> > > >> >> > > > Only
> >> > > >> >> > > > > > > required values I see in it are classpath. and
> >> dtcli is
> >> > > >> >> capable
> >> > > >> >> > of
> >> > > >> >> > > > > > building
> >> > > >> >> > > > > > > it on the fly.
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli
> >> can
> >> > > find
> >> > > >> >> the
> >> > > >> >> > jar
> >> > > >> >> > > > > path,
> >> > > >> >> > > > > > > dt-env.sh and all such conf files. If we need a
> >> change
> >> > in
> >> > > >> >> dtcli,
> >> > > >> >> > > then
> >> > > >> >> > > > > > > community's opinion is required for whether dtcli
> >> > should
> >> > > >> >> change
> >> > > >> >> > in
> >> > > >> >> > > > our
> >> > > >> >> > > > > > repo
> >> > > >> >> > > > > > > or a copy of that with required changes exist in
> >> bigtop
> >> > > >> repo
> >> > > >> >> > until
> >> > > >> >> > > we
> >> > > >> >> > > > > > make
> >> > > >> >> > > > > > > the dtcli generic enough.
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > >
> >> > > >> >> > > > > > Can we have these file paths set using environment
> >> > > variables
> >> > > >> >> with
> >> > > >> >> > > some
> >> > > >> >> > > > > > default values in dtcli?
> >> > > >> >> > > > > > Then we can set such params in dt-env.sh.
> >> > > >> >> > > > > >
> >> > > >> >> > > > > >
> >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file different from
> >> > > >> >> dt-site.xml?
> >> > > >> >> > Not
> >> > > >> >> > > > > sure
> >> > > >> >> > > > > > if
> >> > > >> >> > > > > > > adding in /etc/skel would help. Files & Dirs from
> >> > > /etc/skel
> >> > > >> >> are
> >> > > >> >> > > > copied
> >> > > >> >> > > > > to
> >> > > >> >> > > > > > > home of new user when useradd program is called.
> But
> >> > for
> >> > > >> >> existing
> >> > > >> >> > > > users
> >> > > >> >> > > > > > > this won't be of any use. Moreover, dtcli creates
> >> > ~/.dt/
> >> > > on
> >> > > >> >> the
> >> > > >> >> > fly
> >> > > >> >> > > > if
> >> > > >> >> > > > > > not
> >> > > >> >> > > > > > > encountered for the first time. Again correct me
> if
> >> I'm
> >> > > >> wrong.
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > >
> >> > > >> >> > > > > > Yes, you are right. However this would probably be
> >> > > necessary
> >> > > >> >> step
> >> > > >> >> > for
> >> > > >> >> > > > > > rpm/deb, as they may also get installed during OS
> >> install
> >> > > and
> >> > > >> >> > before
> >> > > >> >> > > > user
> >> > > >> >> > > > > > accounts were created.
> >> > > >> >> > > > > >
> >> > > >> >> > > > > >
> >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure
> >> its
> >> > the
> >> > > >> best
> >> > > >> >> > > idea.
> >> > > >> >> > > > > > We'll
> >> > > >> >> > > > > > > anyway put dtcli in location which is by default
> >> > present
> >> > > in
> >> > > >> >> path
> >> > > >> >> > > i.e.
> >> > > >> >> > > > > > > /usr/bin.
> >> > > >> >> > > > > >
> >> > > >> >> > > > > > For other env variables specific to apex, I'll
> prefer
> >> to
> >> > > use
> >> > > >> >> > > dt-env.sh
> >> > > >> >> > > > > and
> >> > > >> >> > > > > > > source it in dtcli rather than changing bashrc
> >> etc...
> >> > > >> >> > > > > >
> >> > > >> >> > > > > >
> >> > > >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall
> be
> >> the
> >> > > >> >> cleaner
> >> > > >> >> > way
> >> > > >> >> > > > to
> >> > > >> >> > > > > > achieve the same.
> >> > > >> >> > > > > >
> >> > > >> >> > > > > >
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > > Thanks,
> >> > > >> >> > > > > > > Chinmay.
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
> Thombare
> >> <
> >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > > > > Hi,
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > > @Chinmay,
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > > We need to consider following:
> >> > > >> >> > > > > > > > System wide default config files can be located
> at
> >> > > >> following
> >> > > >> >> > > > > locations:
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change
> dtcli
> >> > > >> behaviour
> >> > > >> >> on
> >> > > >> >> > > how
> >> > > >> >> > > > it
> >> > > >> >> > > > > > > finds
> >> > > >> >> > > > > > > > those locations)
> >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change
> dtcli
> >> > > >> >> behaviour on
> >> > > >> >> > > how
> >> > > >> >> > > > > it
> >> > > >> >> > > > > > > > finds those locations)
> >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for
> >> new
> >> > > users
> >> > > >> >> that
> >> > > >> >> > > will
> >> > > >> >> > > > > be
> >> > > >> >> > > > > > > > created in system in future.)
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > > We may also have to modify bashrc / bashprofile
> >> for
> >> > > >> >> population
> >> > > >> >> > > any
> >> > > >> >> > > > > > > > variables that are required.
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > > @dev, please put in your comments / suggestion.
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > > Thanks,
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > > Aniruddha
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> >> > > >> >> > > > > thomas@datatorrent.com>
> >> > > >> >> > > > > > > > wrote:
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > > > > -->
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> >> Kolhatkar <
> >> > > >> >> > > > > > > > chinmay@datatorrent.com
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > wrote:
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > > Hello Everyone!!
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > > Continuing with packaging effort (rpm+deb)
> of
> >> > apex,
> >> > > >> here
> >> > > >> >> > are
> >> > > >> >> > > > some
> >> > > >> >> > > > > > > > > proposals
> >> > > >> >> > > > > > > > > > about package structure etc..
> >> > > >> >> > > > > > > > > > Before posting it on bbigtop mailing list, I
> >> have
> >> > > >> some
> >> > > >> >> > > question
> >> > > >> >> > > > > for
> >> > > >> >> > > > > > > > apex
> >> > > >> >> > > > > > > > > > community.
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > > Proposed Directory structure of apex package
> >> for
> >> > > both
> >> > > >> >> deb &
> >> > > >> >> > > > rpm:
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> >> > > >> >> > > > > > > > > >
> >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> >> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> >> > > >> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> >> > > >> >> > > > > > > > > >
> >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> >> > > >> >> > > > > > > > > > <other dependent jars> = All the 3rd party
> >> jars
> >> > > which
> >> > > >> >> are
> >> > > >> >> > > > > required
> >> > > >> >> > > > > > > for
> >> > > >> >> > > > > > > > > apex
> >> > > >> >> > > > > > > > > > to run. Usually the dependencies are
> packaged
> >> as
> >> > > >> part of
> >> > > >> >> > > > rpm/deb
> >> > > >> >> > > > > by
> >> > > >> >> > > > > > > any
> >> > > >> >> > > > > > > > > > software in bigtop.
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > Can you specify what those jars are in Bigtop
> >> > > context?
> >> > > >> >> Same
> >> > > >> >> > as
> >> > > >> >> > > > > > shipped
> >> > > >> >> > > > > > > > with
> >> > > >> >> > > > > > > > > DT community addition under lib/ ?
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of
> corresponding
> >> of
> >> > > 3rd
> >> > > >> >> party
> >> > > >> >> > > jars
> >> > > >> >> > > > > > which
> >> > > >> >> > > > > > > > > needs
> >> > > >> >> > > > > > > > > > to included while packaging.
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > > Questions related to this:
> >> > > >> >> > > > > > > > > > 1. Should we call the cli of apex as "apex"
> >> > instead
> >> > > >> of
> >> > > >> >> > > "dtcli"
> >> > > >> >> > > > in
> >> > > >> >> > > > > > > > bigtop
> >> > > >> >> > > > > > > > > > package?
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > I think we should keep the name until we are
> >> able
> >> > to
> >> > > >> >> change
> >> > > >> >> > it
> >> > > >> >> > > in
> >> > > >> >> > > > > > Apex.
> >> > > >> >> > > > > > > > > Otherwise this may get confusing.
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > > 2. I see that all softwares in bigtop have
> man
> >> > page
> >> > > >> for
> >> > > >> >> > their
> >> > > >> >> > > > > > > > > executables.
> >> > > >> >> > > > > > > > > > I think we should have it too for dtcli. Is
> >> there
> >> > > any
> >> > > >> >> > > > > documentation
> >> > > >> >> > > > > > > > > which I
> >> > > >> >> > > > > > > > > > can convert to man page? or can I use output
> >> of
> >> > > >> "dtcli
> >> > > >> >> > > --help"?
> >> > > >> >> > > > > > > > > > 3. Do we want to call version of apex in
> >> Bigtop
> >> > as
> >> > > >> >> 3.3.0 OR
> >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > Has to be -incubating
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of bigtop to
> >> depend
> >> > on
> >> > > >> >> 2.7.1
> >> > > >> >> > > > > version
> >> > > >> >> > > > > > of
> >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that we see with
> >> this
> >> > > >> >> > dependency?
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > Should work without changed. I thought we
> >> certified
> >> > > >> >> against
> >> > > >> >> > > > 2.7.0?
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > > 5. Following is the apache mirror from which
> >> > bigtop
> >> > > >> will
> >> > > >> >> > pick
> >> > > >> >> > > > the
> >> > > >> >> > > > > > > apex
> >> > > >> >> > > > > > > > > > source for compilation and packaging. Please
> >> > > correct
> >> > > >> if
> >> > > >> >> > > > > incorrect:
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > >
> http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror
> >> used
> >> > by
> >> > > >> all
> >> > > >> >> the
> >> > > >> >> > > > > > softwares
> >> > > >> >> > > > > > > in
> >> > > >> >> > > > > > > > > > bigtop.
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > > Please share your opinion.
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > > Thanks,
> >> > > >> >> > > > > > > > > > Chinmay.
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> >> > Kolhatkar
> >> > > <
> >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> >> > > >> >> > > > > > > > > > wrote:
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> >> > > >> >> > > > > > > > > > >
> >> > > >> >> > > > > > > > > > > I've already started discussion on bigtop
> >> dev
> >> > > >> mailing
> >> > > >> >> > list
> >> > > >> >> > > > for
> >> > > >> >> > > > > > > > > > > integration. Also created a Jira for the
> >> same.
> >> > > For
> >> > > >> >> this
> >> > > >> >> > > > > > communities
> >> > > >> >> > > > > > > > > > > reference, here is the bigtop Jira:
> >> > BIGTOP-2313.
> >> > > >> >> > > > > > > > > > >
> >> > > >> >> > > > > > > > > > >
> >> > > >> >> > > > > > > > > > >
> >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas
> >> Weise <
> >> > > >> >> > > > > > > thomas@datatorrent.com
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > > > > > wrote:
> >> > > >> >> > > > > > > > > > >
> >> > > >> >> > > > > > > > > > >> Chinmay,
> >> > > >> >> > > > > > > > > > >>
> >> > > >> >> > > > > > > > > > >> I don't see anything under prerequisites
> >> that
> >> > > >> would
> >> > > >> >> be a
> >> > > >> >> > > > > > problem.
> >> > > >> >> > > > > > > We
> >> > > >> >> > > > > > > > > > >> looked
> >> > > >> >> > > > > > > > > > >> at the ASF licencing compatibility as
> part
> >> of
> >> > > >> >> becoming
> >> > > >> >> > an
> >> > > >> >> > > > > > > incubator
> >> > > >> >> > > > > > > > > > >> project.
> >> > > >> >> > > > > > > > > > >>
> >> > > >> >> > > > > > > > > > >> Please focus on the packaging during the
> >> next
> >> > > >> weeks.
> >> > > >> >> > Since
> >> > > >> >> > > > the
> >> > > >> >> > > > > > > work
> >> > > >> >> > > > > > > > > will
> >> > > >> >> > > > > > > > > > >> be
> >> > > >> >> > > > > > > > > > >> part of Bigtop, related discussions and
> >> JIRAs
> >> > > >> should
> >> > > >> >> > also
> >> > > >> >> > > be
> >> > > >> >> > > > > > > there.
> >> > > >> >> > > > > > > > > > >>
> >> > > >> >> > > > > > > > > > >> Would be good to have the packaging in
> >> place
> >> > by
> >> > > >> end
> >> > > >> >> Feb.
> >> > > >> >> > > > > > > > > > >>
> >> > > >> >> > > > > > > > > > >> Thanks,
> >> > > >> >> > > > > > > > > > >> Thomas
> >> > > >> >> > > > > > > > > > >>
> >> > > >> >> > > > > > > > > > >>
> >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> >> > > >> Kolhatkar <
> >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> >> > > >> >> > > > > > > > > > >> wrote:
> >> > > >> >> > > > > > > > > > >>
> >> > > >> >> > > > > > > > > > >> > Hi All,
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >> > We're planning a work on adding Apache
> >> Apex
> >> > > as a
> >> > > >> >> > > component
> >> > > >> >> > > > > to
> >> > > >> >> > > > > > > > Apache
> >> > > >> >> > > > > > > > > > >> > Bigtop.
> >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging system for the
> >> > Apache
> >> > > >> big
> >> > > >> >> data
> >> > > >> >> > > > > > > ecosystem.
> >> > > >> >> > > > > > > > > > >> Several
> >> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most recently
> EMR.
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira task in
> >> APEXCORE
> >> > for
> >> > > >> the
> >> > > >> >> > same:
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >> > Proposed plan of execution is as
> follows:
> >> > > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> >> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> >> > > expectation
> >> > > >> >> for
> >> > > >> >> > new
> >> > > >> >> > > > > > > > components
> >> > > >> >> > > > > > > > > to
> >> > > >> >> > > > > > > > > > >> get
> >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
> >> > > >> >> > > > > > > > > > >> > Here is the list of it:
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >>
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > >
> >> > > >> >> > > > >
> >> > > >> >> > > >
> >> > > >> >> > >
> >> > > >> >> >
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >> > Most of them seems to be standard ASF
> >> based
> >> > > >> >> > > requirements,
> >> > > >> >> > > > > but
> >> > > >> >> > > > > > > few
> >> > > >> >> > > > > > > > > need
> >> > > >> >> > > > > > > > > > >> to
> >> > > >> >> > > > > > > > > > >> > be checked for:
> >> > > >> >> > > > > > > > > > >> > 1. Software projects are expected to be
> >> > > Licensed
> >> > > >> >> under
> >> > > >> >> > > > > Apache
> >> > > >> >> > > > > > > > > License,
> >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
> >> > > expected
> >> > > >> >> to be
> >> > > >> >> > > > > > > compatible
> >> > > >> >> > > > > > > > > with
> >> > > >> >> > > > > > > > > > >> this
> >> > > >> >> > > > > > > > > > >> > license)
> >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to
> >> > check
> >> > > if
> >> > > >> >> > > > > dependencies
> >> > > >> >> > > > > > of
> >> > > >> >> > > > > > > > > Apex
> >> > > >> >> > > > > > > > > > >> are
> >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess
> >> would
> >> > > be a
> >> > > >> >> > > > > verification
> >> > > >> >> > > > > > > > check.
> >> > > >> >> > > > > > > > > > >> > 2. Software projects are expected to be
> >> > > >> compatible
> >> > > >> >> > with
> >> > > >> >> > > > all
> >> > > >> >> > > > > of
> >> > > >> >> > > > > > > the
> >> > > >> >> > > > > > > > > > >> > supported platforms that Bigtop
> >> distribution
> >> > > is
> >> > > >> >> > > targeting
> >> > > >> >> > > > > > > > > > >> >     - This needs verifying whether our
> >> > > software
> >> > > >> >> runs
> >> > > >> >> > > fine
> >> > > >> >> > > > in
> >> > > >> >> > > > > > > > > centos-6
> >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
> debian-8
> >> > > >> >> > opensuse-13.2.
> >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that should be
> added
> >> for
> >> > > >> >> > deployment.
> >> > > >> >> > > > > > > > > > >> > 4. Identifying the test artifacts which
> >> goes
> >> > > >> beyond
> >> > > >> >> > > smoke
> >> > > >> >> > > > > test
> >> > > >> >> > > > > > > > > > >> >     - These are basically the
> integration
> >> > > tests
> >> > > >> for
> >> > > >> >> > > > > > verification
> >> > > >> >> > > > > > > > > after
> >> > > >> >> > > > > > > > > > >> the
> >> > > >> >> > > > > > > > > > >> > deployment. This is a soft requirement,
> >> but
> >> > > aim
> >> > > >> is
> >> > > >> >> to
> >> > > >> >> > > > > achieve
> >> > > >> >> > > > > > > this
> >> > > >> >> > > > > > > > > as
> >> > > >> >> > > > > > > > > > >> well
> >> > > >> >> > > > > > > > > > >> > or at least have explanation why not to
> >> > > include.
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >> > If there are any from the link which
> >> > > explicitly
> >> > > >> >> needs
> >> > > >> >> > to
> >> > > >> >> > > > be
> >> > > >> >> > > > > > > > checked
> >> > > >> >> > > > > > > > > > >> other
> >> > > >> >> > > > > > > > > > >> > than above 4, please let us know.
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as component to
> >> Bigtop*
> >> > > >> >> > > > > > > > > > >> > From one of the mail archive of Bigtop,
> >> it
> >> > was
> >> > > >> >> learnt
> >> > > >> >> > > that
> >> > > >> >> > > > > the
> >> > > >> >> > > > > > > > > bigtop
> >> > > >> >> > > > > > > > > > >> > community want to see the addition of
> new
> >> > > >> >> components
> >> > > >> >> > in
> >> > > >> >> > > > > > phases.
> >> > > >> >> > > > > > > > Here
> >> > > >> >> > > > > > > > > > are
> >> > > >> >> > > > > > > > > > >> > the phases:
> >> > > >> >> > > > > > > > > > >> > 1. Packaging
> >> > > >> >> > > > > > > > > > >> >     - This needs creating of package
> i.e.
> >> > rpm
> >> > > &
> >> > > >> deb
> >> > > >> >> > > files.
> >> > > >> >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> >> > > >> DISCLAMER,
> >> > > >> >> > NOTES
> >> > > >> >> > > > etc
> >> > > >> >> > > > > if
> >> > > >> >> > > > > > > any
> >> > > >> >> > > > > > > > > > >> needed.
> >> > > >> >> > > > > > > > > > >> >     - Any documentation that need to be
> >> > added
> >> > > to
> >> > > >> >> > > > > distribution
> >> > > >> >> > > > > > of
> >> > > >> >> > > > > > > > our
> >> > > >> >> > > > > > > > > > >> > software.
> >> > > >> >> > > > > > > > > > >> >     - Any license information of
> >> > dependencies
> >> > > >> >> required
> >> > > >> >> > > to
> >> > > >> >> > > > be
> >> > > >> >> > > > > > > added
> >> > > >> >> > > > > > > > > to
> >> > > >> >> > > > > > > > > > >> > package
> >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> >> > > >> >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic
> >> deployment
> >> > and
> >> > > >> >> > > > configuration
> >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for automatic
> >> > > >> deployment
> >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> >> > > >> >> > > > > > > > > > >> >     - For verification of deployments.
> >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all
> new
> >> > > stuff
> >> > > >> is
> >> > > >> >> > > > compliant
> >> > > >> >> > > > > > > with
> >> > > >> >> > > > > > > > > ASF
> >> > > >> >> > > > > > > > > > >> > license requirements. If you add code
> >> > licenses
> >> > > >> >> under
> >> > > >> >> > > > > different
> >> > > >> >> > > > > > > > > > licenses,
> >> > > >> >> > > > > > > > > > >> > those would need to be listed in the
> >> NOTICE.
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >> > Please share your thoughts on the
> >> approach.
> >> > > >> >> > > > > > > > > > >> > We'll start corresponding communication
> >> on
> >> > > >> bigtop
> >> > > >> >> > > mailing
> >> > > >> >> > > > > list
> >> > > >> >> > > > > > > as
> >> > > >> >> > > > > > > > > > well.
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >> > We have some specific
> >> questions/suggestions
> >> > > >> >> related to
> >> > > >> >> > > > what
> >> > > >> >> > > > > > > should
> >> > > >> >> > > > > > > > > be
> >> > > >> >> > > > > > > > > > >> the
> >> > > >> >> > > > > > > > > > >> > content of the package and what should
> be
> >> > the
> >> > > >> smoke
> >> > > >> >> > > tests,
> >> > > >> >> > > > > but
> >> > > >> >> > > > > > > in
> >> > > >> >> > > > > > > > > the
> >> > > >> >> > > > > > > > > > >> > interest of not having too much content
> >> > here,
> >> > > >> we'll
> >> > > >> >> > put
> >> > > >> >> > > > the
> >> > > >> >> > > > > > > > > questions
> >> > > >> >> > > > > > > > > > >> as a
> >> > > >> >> > > > > > > > > > >> > separate mail in this mailthread.
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >> > Thanks,
> >> > > >> >> > > > > > > > > > >> > Chinmay.
> >> > > >> >> > > > > > > > > > >> >
> >> > > >> >> > > > > > > > > > >>
> >> > > >> >> > > > > > > > > > >
> >> > > >> >> > > > > > > > > > >
> >> > > >> >> > > > > > > > > >
> >> > > >> >> > > > > > > > >
> >> > > >> >> > > > > > > >
> >> > > >> >> > > > > > >
> >> > > >> >> > > > > >
> >> > > >> >> > > > >
> >> > > >> >> > > >
> >> > > >> >> > >
> >> > > >> >> >
> >> > > >> >>
> >> > > >> >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Hi Everyone,

In one of the previous mails in this thread, we decided that version of the
package will be "3.3.0-incubating".
The problem that I'm facing is in RPM base version "-" is not allowed.

Can we have package version to "3.3.0"?
i.e. the packages created will be as follows:
For RPM: apex-3.3.0-1.noarch.rpm
For DEB: apex_3.3.0-1_all.deb

There is another integration for Zepplin in Bigtop which is incubating and
is having base version as "0.5.6".

Please let me know your thoughts.

Thanks,
Chinmay.



On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Thanks everyone for the feedback.
>
> Also as Ram and David mentioned, I'll start creating the man page as well.
>
> Thanks,
> Chinmay.
>
>
> On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
>> I think we already agreed that we will retain the current script name in
>> this bigtop iteration. There is a ticket for renaming, add you suggestions
>> there.
>>
>> --
>> sent from mobile
>> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <sa...@datatorrent.com> wrote:
>>
>> > +1 for Symlink from apex to dtcli or start the transition process by
>> making
>> > dtcli as the symlink to apex cli.
>> >
>> > Renaming is a breaking change should be done during major version number
>> > change.
>> >
>> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <da...@datatorrent.com>
>> wrote:
>> >
>> > > Apologies for hitting the "Send" button too early.
>> > >
>> > > What is the scope of the name change?  If we simply have a symlink
>> from
>> > > apex to dtcli, then it should have no backward compatibility problem
>> at
>> > > all.
>> > >
>> > > But the proper name change would involve:
>> > > - Change the name of DTCli class to ApexCli
>> > > - Change the prompt from dt> to apex>
>> > > - Change the initial greeting when running the CLI from something like
>> > "DT
>> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
>> > >
>> > > The above changes may break existing scripts that rely on the the
>> string
>> > > "DT".
>> > > But if folks are okay with this potential problem, I am okay with the
>> > name
>> > > change.
>> > >
>> > > For the man page, we can start with the content of this page:
>> > >
>> > > http://docs.datatorrent.com/dtcli/
>> > >
>> > > Plus the command line options when doing dtcli -h:
>> > >
>> > > usage: DTCli
>> > >  -e <arg>    Commands are read from the argument
>> > >  -f <arg>    Use the specified prompt at all time
>> > >  -h          Print this help
>> > >  -kp <arg>   Use the specified kerberos principal
>> > >  -kt <arg>   Use the specified kerberos keytab
>> > >  -p <arg>    JSONP padding function
>> > >  -r          JSON Raw mode
>> > >  -v          Verbose mode level 1
>> > >  -vv         Verbose mode level 2
>> > >  -vvv        Verbose mode level 3
>> > >  -vvvv       Verbose mode level 4
>> > >
>> > > David
>> > >
>> > >
>> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <da...@datatorrent.com>
>> > wrote:
>> > >
>> > > > What is the scope of the name change?  If we simply have a symlink
>> from
>> > > > apex to dtcli, then it should have no backward compatibility
>> problem at
>> > > all.
>> > > >
>> > > > But the proper name change would involve:
>> > > > - Change the name of DTCli class to ApexCli
>> > > > - Change the prompt from dt> to apex>
>> > > > - Change the initial greeting when running the CLI from something
>> like
>> > > "DT
>> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
>> > > >
>> > > > The above changes may break existing scripts that rely on the the
>> > string
>> > > > "DT".
>> > > > But if folks are okay with this potential problem, I am okay with
>> the
>> > > name
>> > > > change.
>> > > >
>> > > > For the man page, we can start with the content of this page:
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
>> chinmay@apache.org
>> > >
>> > > > wrote:
>> > > >
>> > > >> Here is a suggestion I have related to name change:
>> > > >>
>> > > >> 1. While packaging we change the name of "dtcli" script to "apex".
>> > This
>> > > is
>> > > >> done so that in future when we change the name to "apex", users of
>> > > bigtop
>> > > >> apex don't have to transition much.
>> > > >> 2. We also keep a symlink named dtcli which points to apex script.
>> > This
>> > > is
>> > > >> for backward compatibility.
>> > > >> 3. We still keep conf/dt-env.sh for first integration in bigtop.
>> This
>> > is
>> > > >> in
>> > > >> the interest of not changing the content of dtcli or apex script.
>> > > >> 4. In apex 3.4.0, we take care of all these naming related changes
>> and
>> > > >> update bigtop repository later to remove references to dtcli all
>> > > together.
>> > > >>
>> > > >> Please share your thoughts on above approach.
>> > > >>
>> > > >> Also, please share what could be the man page content for "apex".
>> > > >>
>> > > >> Thanks,
>> > > >> Chinmay.
>> > > >>
>> > > >>
>> > > >>
>> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
>> > chinmay@apache.org>
>> > > >> wrote:
>> > > >>
>> > > >> > Yes.. It'll be build from source tar downloaded from one of the
>> > apache
>> > > >> > mirror: apache.osuosl.org
>> > > >> >
>> > > >> > The reason for dt-env.sh is it's sourced from dtcli. This leaves
>> us
>> > 3
>> > > >> > options:
>> > > >> > If we should not have any dt-env.sh, dtcli would need a change
>> while
>> > > >> > packaging.
>> > > >> >
>> > > >> > But if we want to use dtcli as it is, then we would need
>> dt-env.sh
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
>> > thomas@datatorrent.com
>> > > >
>> > > >> > wrote:
>> > > >> >
>> > > >> >> Looks good overall, though there shouldn't be any dt-env.sh
>> > > >> >>
>> > > >> >> What will the bigtop package be built from, the source tar?
>> > > >> >>
>> > > >> >>
>> > > >> >>
>> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
>> > > chinmay@apache.org
>> > > >> >
>> > > >> >> wrote:
>> > > >> >>
>> > > >> >> > Hi Thomas,
>> > > >> >> >
>> > > >> >> > Thanks for the feedback.
>> > > >> >> >
>> > > >> >> > First, we're not changing the name anywhere. We'll follow what
>> > > >> >> currently is
>> > > >> >> > in source tarball for 3.3.0-incubating version.
>> > > >> >> >
>> > > >> >> > Secondly, I've mentioned a directory structure below which is
>> > > inline
>> > > >> >> with
>> > > >> >> > other existing integrations with bigtop.
>> > > >> >> >
>> > > >> >> > The need for each file is as follows:
>> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
>> engine/src/main/scripts/
>> > > of
>> > > >> >> source
>> > > >> >> > code apex 3.3.0-incubating.
>> > > >> >> >
>> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches
>> for
>> > > >> hadoop
>> > > >> >> > binary path and exports an env variable for dtcli to use it.
>> > > >> >> >
>> > > >> >> > 3. lib/ -> Libraries for CLI. This list is extracted in a
>> similar
>> > > >> way to
>> > > >> >> > how DT Community Edition finds dependency jars.
>> > > >> >> > Please note that this is the first iteration list of jars. I'm
>> > > >> trying to
>> > > >> >> > narrow this down to only those which are really required.
>> > > >> >> > The test that I'm running to check if dtcli runs fine with
>> given
>> > > >> >> dependency
>> > > >> >> > is to launch+shutdown+kill+status for pi demo.
>> > > >> >> >
>> > > >> >> > Above is the directory structure which is required for CLI to
>> > work.
>> > > >> >> > What aniruddha and pradeep has mentioned are some additional
>> > files
>> > > >> which
>> > > >> >> > makes the like of administrator easier. For eg. /etc/skel
>> > > >> >> >
>> > > >> >> > Please let me know if this is inline with what you're
>> thinking.
>> > > >> >> >
>> > > >> >> > Thanks,
>> > > >> >> > Chinmay.
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > .
>> > > >> >> > |-- bin
>> > > >> >> > |   `-- dtcli
>> > > >> >> > |-- conf
>> > > >> >> > |   `-- dt-env.sh
>> > > >> >> > `-- lib
>> > > >> >> >     |-- ant-1.9.2.jar
>> > > >> >> >     |-- ant-launcher-1.9.2.jar
>> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
>> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
>> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
>> > > >> >> >     |-- apex-engine.jar
>> > > >> >> >     |-- apex-engine-tests.jar
>> > > >> >> >     |-- async-http-client-1.7.20.jar
>> > > >> >> >     |-- bval-core-0.5.jar
>> > > >> >> >     |-- bval-jsr303-0.5.jar
>> > > >> >> >     |-- commons-beanutils-1.8.3.jar
>> > > >> >> >     |-- commons-codec-1.10.jar
>> > > >> >> >     |-- commons-lang3-3.1.jar
>> > > >> >> >     |-- commons-logging-1.1.3.jar
>> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
>> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
>> > > >> >> >     |-- httpclient-4.3.5.jar
>> > > >> >> >     |-- httpcore-4.3.2.jar
>> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
>> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
>> > > >> >> >     |-- javax.servlet-3.1.jar
>> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
>> > > >> >> >     |-- jersey-apache-client4-1.9.jar
>> > > >> >> >     |-- jersey-client-1.9.jar
>> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
>> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
>> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
>> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
>> > > >> >> >     |-- jline-2.11.jar
>> > > >> >> >     |-- kryo-2.24.0.jar
>> > > >> >> >     |-- mbassador-1.1.9.jar
>> > > >> >> >     |-- minlog-1.2.jar
>> > > >> >> >     |-- netlet-1.2.0.jar
>> > > >> >> >     |-- netty-3.6.6.Final.jar
>> > > >> >> >     |-- objenesis-2.1.jar
>> > > >> >> >     |-- validation-api-1.1.0.Final.jar
>> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
>> > > >> >> >     `-- zip4j-1.3.2.jar
>> > > >> >> >
>> > > >> >> > 3 directories, 40 files
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
>> > > >> thomas@datatorrent.com>
>> > > >> >> > wrote:
>> > > >> >> >
>> > > >> >> > > Chinmay,
>> > > >> >> > >
>> > > >> >> > > Before discussing where to put the files, let's make sure
>> they
>> > > are
>> > > >> >> really
>> > > >> >> > > needed for the operation of the CLI. As for names, anything
>> new
>> > > >> needs
>> > > >> >> to
>> > > >> >> > > reflect Apex in the name and should follow common
>> conventions,
>> > > >> >> especially
>> > > >> >> > > in Bigtop where there are many existing integrations to look
>> > at.
>> > > >> >> > >
>> > > >> >> > > Thanks,
>> > > >> >> > > Thomas
>> > > >> >> > >
>> > > >> >> > >
>> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
>> > > >> >> > > chinmay@datatorrent.com>
>> > > >> >> > > wrote:
>> > > >> >> > >
>> > > >> >> > > > Really good points Pradeep and Aniruddha.
>> > > >> >> > > >
>> > > >> >> > > > 1. I believe we won't need to change the dtcli
>> considering it
>> > > >> works
>> > > >> >> > with
>> > > >> >> > > DT
>> > > >> >> > > > Community edition. We can keep the directory structure
>> > similar
>> > > to
>> > > >> >> that.
>> > > >> >> > > > dt-env.sh has variables which contains information
>> required
>> > for
>> > > >> >> dtcli
>> > > >> >> > to
>> > > >> >> > > > launch.
>> > > >> >> > > >
>> > > >> >> > > > 2. Let me check with bigtop community that whether they
>> > > >> facilitate
>> > > >> >> the
>> > > >> >> > > > installation of rpms/debs before user is created. In
>> either
>> > > case,
>> > > >> >> > current
>> > > >> >> > > > dtcli creates a .dt folder in home directory. Also before
>> > > putting
>> > > >> >> > > anything
>> > > >> >> > > > in /etc/skel we need to define what are the default
>> contents
>> > > that
>> > > >> >> > should
>> > > >> >> > > go
>> > > >> >> > > > to ~/.dt/ folder. If there is no defaults, probably we
>> should
>> > > not
>> > > >> >> > > > explicitly add it in /etc/skel.
>> > > >> >> > > >
>> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This way contents
>> of
>> > > >> >> dt-env.sh
>> > > >> >> > > are
>> > > >> >> > > > present as env variables. But I see a catch there. Adding
>> > > >> dt-env.sh
>> > > >> >> to
>> > > >> >> > > > /etc/profile.d would make all the variables available at
>> > > runtime
>> > > >> all
>> > > >> >> > the
>> > > >> >> > > > time. I feel a little skeptical about that. Maybe a
>> possible
>> > > >> >> collision
>> > > >> >> > > can
>> > > >> >> > > > occur with other application vars.
>> > > >> >> > > > Moreover, current dtcli does source "../conf/dt-env.sh"
>> and
>> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway
>> available.
>> > > >> >> > > >
>> > > >> >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
>> > > >> >> > > Jira/repository.
>> > > >> >> > > > Packaging related code usually goes there. (That's what
>> all
>> > the
>> > > >> >> > > components
>> > > >> >> > > > in bigtop does).
>> > > >> >> > > > Having said that, once a PR is created, I'll be sharing
>> the
>> > > link
>> > > >> of
>> > > >> >> PR
>> > > >> >> > > > here, so that apex community as well can review it.
>> > > >> >> > > >
>> > > >> >> > > > Thanks,
>> > > >> >> > > > Chinmay.
>> > > >> >> > > >
>> > > >> >> > > >
>> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
>> > > >> >> > > > aniruddha@datatorrent.com> wrote:
>> > > >> >> > > >
>> > > >> >> > > > > +1 on suggestions and approach.
>> > > >> >> > > > > We may need to iron out details about exact paths etc.
>> > > >> >> > > > > Which can be done on jira / PR comments.
>> > > >> >> > > > > Is that right @dev?
>> > > >> >> > > > >
>> > > >> >> > > > >
>> > > >> >> > > > >
>> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
>> > > >> >> > apache@pradeepdalvi.com>
>> > > >> >> > > > > wrote:
>> > > >> >> > > > >
>> > > >> >> > > > > > Inline comments...
>> > > >> >> > > > > >
>> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
>> > > >> >> > > > > > chinmay@datatorrent.com>
>> > > >> >> > > > > > wrote:
>> > > >> >> > > > > >
>> > > >> >> > > > > > > @Thomas, Not all the jar present in DT community
>> > edition
>> > > >> will
>> > > >> >> be
>> > > >> >> > > > > included
>> > > >> >> > > > > > > there.
>> > > >> >> > > > > > >
>> > > >> >> > > > > > > Here is the list of jars I found from common
>> > dependencies
>> > > >> >> between
>> > > >> >> > > of
>> > > >> >> > > > > apex
>> > > >> >> > > > > > > and DT Community edition:
>> > > >> >> > > > > > > netlet-1.2.0.jar
>> > > >> >> > > > > > > kryo-2.24.0.jar
>> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
>> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
>> > > >> >> > > > > > > async-http-client-1.7.20.jar
>> > > >> >> > > > > > > netty-3.6.6.Final.jar
>> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
>> > > >> >> > > > > > > bval-jsr303-0.5.jar
>> > > >> >> > > > > > > bval-core-0.5.jar
>> > > >> >> > > > > > > commons-lang3-3.1.jar
>> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
>> > > >> >> > > > > > > httpclient-4.3.5.jar
>> > > >> >> > > > > > > commons-codec-1.10.jar
>> > > >> >> > > > > > > zip4j-1.3.2.jar
>> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
>> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
>> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
>> > > >> >> > > > > > > jline-2.11.jar
>> > > >> >> > > > > > > ant-1.9.2.jar
>> > > >> >> > > > > > > ant-launcher-1.9.2.jar
>> > > >> >> > > > > > > mbassador-1.1.9.jar
>> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
>> > > >> >> > > > > > > jackson-xc-1.9.2.jar
>> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
>> > > >> >> > > > > > >
>> > > >> >> > > > > > > Ofcourse, I'll be running some tests do check that
>> > dtcli
>> > > >> works
>> > > >> >> > > > properly
>> > > >> >> > > > > > for
>> > > >> >> > > > > > > launch+shutdown+kill of apps with only these
>> libraries
>> > > >> >> present in
>> > > >> >> > > > > > isolation
>> > > >> >> > > > > > > without dependency on local m2.
>> > > >> >> > > > > > > I'm believe that there are unwanted jars which are
>> used
>> > > for
>> > > >> >> > compile
>> > > >> >> > > > > time
>> > > >> >> > > > > > > dependency and not runtime in above list which I can
>> > drop
>> > > >> to
>> > > >> >> keep
>> > > >> >> > > > > package
>> > > >> >> > > > > > > size minimal.
>> > > >> >> > > > > > >
>> > > >> >> > > > > > >
>> > > >> >> > > > > > > @Anniruddha,
>> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
>> dt-env.sh is
>> > > >> >> required.
>> > > >> >> > > > Only
>> > > >> >> > > > > > > required values I see in it are classpath. and
>> dtcli is
>> > > >> >> capable
>> > > >> >> > of
>> > > >> >> > > > > > building
>> > > >> >> > > > > > > it on the fly.
>> > > >> >> > > > > > >
>> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli
>> can
>> > > find
>> > > >> >> the
>> > > >> >> > jar
>> > > >> >> > > > > path,
>> > > >> >> > > > > > > dt-env.sh and all such conf files. If we need a
>> change
>> > in
>> > > >> >> dtcli,
>> > > >> >> > > then
>> > > >> >> > > > > > > community's opinion is required for whether dtcli
>> > should
>> > > >> >> change
>> > > >> >> > in
>> > > >> >> > > > our
>> > > >> >> > > > > > repo
>> > > >> >> > > > > > > or a copy of that with required changes exist in
>> bigtop
>> > > >> repo
>> > > >> >> > until
>> > > >> >> > > we
>> > > >> >> > > > > > make
>> > > >> >> > > > > > > the dtcli generic enough.
>> > > >> >> > > > > > >
>> > > >> >> > > > > >
>> > > >> >> > > > > > Can we have these file paths set using environment
>> > > variables
>> > > >> >> with
>> > > >> >> > > some
>> > > >> >> > > > > > default values in dtcli?
>> > > >> >> > > > > > Then we can set such params in dt-env.sh.
>> > > >> >> > > > > >
>> > > >> >> > > > > >
>> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file different from
>> > > >> >> dt-site.xml?
>> > > >> >> > Not
>> > > >> >> > > > > sure
>> > > >> >> > > > > > if
>> > > >> >> > > > > > > adding in /etc/skel would help. Files & Dirs from
>> > > /etc/skel
>> > > >> >> are
>> > > >> >> > > > copied
>> > > >> >> > > > > to
>> > > >> >> > > > > > > home of new user when useradd program is called. But
>> > for
>> > > >> >> existing
>> > > >> >> > > > users
>> > > >> >> > > > > > > this won't be of any use. Moreover, dtcli creates
>> > ~/.dt/
>> > > on
>> > > >> >> the
>> > > >> >> > fly
>> > > >> >> > > > if
>> > > >> >> > > > > > not
>> > > >> >> > > > > > > encountered for the first time. Again correct me if
>> I'm
>> > > >> wrong.
>> > > >> >> > > > > > >
>> > > >> >> > > > > >
>> > > >> >> > > > > > Yes, you are right. However this would probably be
>> > > necessary
>> > > >> >> step
>> > > >> >> > for
>> > > >> >> > > > > > rpm/deb, as they may also get installed during OS
>> install
>> > > and
>> > > >> >> > before
>> > > >> >> > > > user
>> > > >> >> > > > > > accounts were created.
>> > > >> >> > > > > >
>> > > >> >> > > > > >
>> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure
>> its
>> > the
>> > > >> best
>> > > >> >> > > idea.
>> > > >> >> > > > > > We'll
>> > > >> >> > > > > > > anyway put dtcli in location which is by default
>> > present
>> > > in
>> > > >> >> path
>> > > >> >> > > i.e.
>> > > >> >> > > > > > > /usr/bin.
>> > > >> >> > > > > >
>> > > >> >> > > > > > For other env variables specific to apex, I'll prefer
>> to
>> > > use
>> > > >> >> > > dt-env.sh
>> > > >> >> > > > > and
>> > > >> >> > > > > > > source it in dtcli rather than changing bashrc
>> etc...
>> > > >> >> > > > > >
>> > > >> >> > > > > >
>> > > >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be
>> the
>> > > >> >> cleaner
>> > > >> >> > way
>> > > >> >> > > > to
>> > > >> >> > > > > > achieve the same.
>> > > >> >> > > > > >
>> > > >> >> > > > > >
>> > > >> >> > > > > > >
>> > > >> >> > > > > > Thanks,
>> > > >> >> > > > > > > Chinmay.
>> > > >> >> > > > > > >
>> > > >> >> > > > > > >
>> > > >> >> > > > > > >
>> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare
>> <
>> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
>> > > >> >> > > > > > >
>> > > >> >> > > > > > > > Hi,
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > > @Chinmay,
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > > We need to consider following:
>> > > >> >> > > > > > > > System wide default config files can be located at
>> > > >> following
>> > > >> >> > > > > locations:
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
>> > > >> behaviour
>> > > >> >> on
>> > > >> >> > > how
>> > > >> >> > > > it
>> > > >> >> > > > > > > finds
>> > > >> >> > > > > > > > those locations)
>> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
>> > > >> >> behaviour on
>> > > >> >> > > how
>> > > >> >> > > > > it
>> > > >> >> > > > > > > > finds those locations)
>> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for
>> new
>> > > users
>> > > >> >> that
>> > > >> >> > > will
>> > > >> >> > > > > be
>> > > >> >> > > > > > > > created in system in future.)
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > > We may also have to modify bashrc / bashprofile
>> for
>> > > >> >> population
>> > > >> >> > > any
>> > > >> >> > > > > > > > variables that are required.
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > > @dev, please put in your comments / suggestion.
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > > Thanks,
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > > Aniruddha
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
>> > > >> >> > > > > thomas@datatorrent.com>
>> > > >> >> > > > > > > > wrote:
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > > > > -->
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
>> Kolhatkar <
>> > > >> >> > > > > > > > chinmay@datatorrent.com
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > wrote:
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > > Hello Everyone!!
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of
>> > apex,
>> > > >> here
>> > > >> >> > are
>> > > >> >> > > > some
>> > > >> >> > > > > > > > > proposals
>> > > >> >> > > > > > > > > > about package structure etc..
>> > > >> >> > > > > > > > > > Before posting it on bbigtop mailing list, I
>> have
>> > > >> some
>> > > >> >> > > question
>> > > >> >> > > > > for
>> > > >> >> > > > > > > > apex
>> > > >> >> > > > > > > > > > community.
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > > Proposed Directory structure of apex package
>> for
>> > > both
>> > > >> >> deb &
>> > > >> >> > > > rpm:
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
>> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
>> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
>> > > >> >> > > > > > > > > >
>> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
>> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
>> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
>> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
>> > > >> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
>> > > >> >> > > > > > > > > >
>> /usr/share/doc/apex/license/<package>-LICENSE.txt
>> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
>> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
>> > > >> >> > > > > > > > > > <other dependent jars> = All the 3rd party
>> jars
>> > > which
>> > > >> >> are
>> > > >> >> > > > > required
>> > > >> >> > > > > > > for
>> > > >> >> > > > > > > > > apex
>> > > >> >> > > > > > > > > > to run. Usually the dependencies are packaged
>> as
>> > > >> part of
>> > > >> >> > > > rpm/deb
>> > > >> >> > > > > by
>> > > >> >> > > > > > > any
>> > > >> >> > > > > > > > > > software in bigtop.
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > Can you specify what those jars are in Bigtop
>> > > context?
>> > > >> >> Same
>> > > >> >> > as
>> > > >> >> > > > > > shipped
>> > > >> >> > > > > > > > with
>> > > >> >> > > > > > > > > DT community addition under lib/ ?
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of corresponding
>> of
>> > > 3rd
>> > > >> >> party
>> > > >> >> > > jars
>> > > >> >> > > > > > which
>> > > >> >> > > > > > > > > needs
>> > > >> >> > > > > > > > > > to included while packaging.
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > > Questions related to this:
>> > > >> >> > > > > > > > > > 1. Should we call the cli of apex as "apex"
>> > instead
>> > > >> of
>> > > >> >> > > "dtcli"
>> > > >> >> > > > in
>> > > >> >> > > > > > > > bigtop
>> > > >> >> > > > > > > > > > package?
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > I think we should keep the name until we are
>> able
>> > to
>> > > >> >> change
>> > > >> >> > it
>> > > >> >> > > in
>> > > >> >> > > > > > Apex.
>> > > >> >> > > > > > > > > Otherwise this may get confusing.
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > > 2. I see that all softwares in bigtop have man
>> > page
>> > > >> for
>> > > >> >> > their
>> > > >> >> > > > > > > > > executables.
>> > > >> >> > > > > > > > > > I think we should have it too for dtcli. Is
>> there
>> > > any
>> > > >> >> > > > > documentation
>> > > >> >> > > > > > > > > which I
>> > > >> >> > > > > > > > > > can convert to man page? or can I use output
>> of
>> > > >> "dtcli
>> > > >> >> > > --help"?
>> > > >> >> > > > > > > > > > 3. Do we want to call version of apex in
>> Bigtop
>> > as
>> > > >> >> 3.3.0 OR
>> > > >> >> > > > > > > > > > 3.3.0-incubating?
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > Has to be -incubating
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > > 4. Is it ok for apex package of bigtop to
>> depend
>> > on
>> > > >> >> 2.7.1
>> > > >> >> > > > > version
>> > > >> >> > > > > > of
>> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that we see with
>> this
>> > > >> >> > dependency?
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > Should work without changed. I thought we
>> certified
>> > > >> >> against
>> > > >> >> > > > 2.7.0?
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > > 5. Following is the apache mirror from which
>> > bigtop
>> > > >> will
>> > > >> >> > pick
>> > > >> >> > > > the
>> > > >> >> > > > > > > apex
>> > > >> >> > > > > > > > > > source for compilation and packaging. Please
>> > > correct
>> > > >> if
>> > > >> >> > > > > incorrect:
>> > > >> >> > > > > > > > > >
>> > > >> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror
>> used
>> > by
>> > > >> all
>> > > >> >> the
>> > > >> >> > > > > > softwares
>> > > >> >> > > > > > > in
>> > > >> >> > > > > > > > > > bigtop.
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > > Please share your opinion.
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > > Thanks,
>> > > >> >> > > > > > > > > > Chinmay.
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
>> > Kolhatkar
>> > > <
>> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
>> > > >> >> > > > > > > > > > wrote:
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > > > > Yes.. Starting to work on the packaging.
>> > > >> >> > > > > > > > > > >
>> > > >> >> > > > > > > > > > > I've already started discussion on bigtop
>> dev
>> > > >> mailing
>> > > >> >> > list
>> > > >> >> > > > for
>> > > >> >> > > > > > > > > > > integration. Also created a Jira for the
>> same.
>> > > For
>> > > >> >> this
>> > > >> >> > > > > > communities
>> > > >> >> > > > > > > > > > > reference, here is the bigtop Jira:
>> > BIGTOP-2313.
>> > > >> >> > > > > > > > > > >
>> > > >> >> > > > > > > > > > >
>> > > >> >> > > > > > > > > > >
>> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas
>> Weise <
>> > > >> >> > > > > > > thomas@datatorrent.com
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > > > > > wrote:
>> > > >> >> > > > > > > > > > >
>> > > >> >> > > > > > > > > > >> Chinmay,
>> > > >> >> > > > > > > > > > >>
>> > > >> >> > > > > > > > > > >> I don't see anything under prerequisites
>> that
>> > > >> would
>> > > >> >> be a
>> > > >> >> > > > > > problem.
>> > > >> >> > > > > > > We
>> > > >> >> > > > > > > > > > >> looked
>> > > >> >> > > > > > > > > > >> at the ASF licencing compatibility as part
>> of
>> > > >> >> becoming
>> > > >> >> > an
>> > > >> >> > > > > > > incubator
>> > > >> >> > > > > > > > > > >> project.
>> > > >> >> > > > > > > > > > >>
>> > > >> >> > > > > > > > > > >> Please focus on the packaging during the
>> next
>> > > >> weeks.
>> > > >> >> > Since
>> > > >> >> > > > the
>> > > >> >> > > > > > > work
>> > > >> >> > > > > > > > > will
>> > > >> >> > > > > > > > > > >> be
>> > > >> >> > > > > > > > > > >> part of Bigtop, related discussions and
>> JIRAs
>> > > >> should
>> > > >> >> > also
>> > > >> >> > > be
>> > > >> >> > > > > > > there.
>> > > >> >> > > > > > > > > > >>
>> > > >> >> > > > > > > > > > >> Would be good to have the packaging in
>> place
>> > by
>> > > >> end
>> > > >> >> Feb.
>> > > >> >> > > > > > > > > > >>
>> > > >> >> > > > > > > > > > >> Thanks,
>> > > >> >> > > > > > > > > > >> Thomas
>> > > >> >> > > > > > > > > > >>
>> > > >> >> > > > > > > > > > >>
>> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
>> > > >> Kolhatkar <
>> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
>> > > >> >> > > > > > > > > > >> wrote:
>> > > >> >> > > > > > > > > > >>
>> > > >> >> > > > > > > > > > >> > Hi All,
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >> > We're planning a work on adding Apache
>> Apex
>> > > as a
>> > > >> >> > > component
>> > > >> >> > > > > to
>> > > >> >> > > > > > > > Apache
>> > > >> >> > > > > > > > > > >> > Bigtop.
>> > > >> >> > > > > > > > > > >> > Bigtop is the packaging system for the
>> > Apache
>> > > >> big
>> > > >> >> data
>> > > >> >> > > > > > > ecosystem.
>> > > >> >> > > > > > > > > > >> Several
>> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >> > Here is the tracking Jira task in
>> APEXCORE
>> > for
>> > > >> the
>> > > >> >> > same:
>> > > >> >> > > > > > > > > > >> >
>> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >> > Proposed plan of execution is as follows:
>> > > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
>> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard and soft
>> > > expectation
>> > > >> >> for
>> > > >> >> > new
>> > > >> >> > > > > > > > components
>> > > >> >> > > > > > > > > to
>> > > >> >> > > > > > > > > > >> get
>> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
>> > > >> >> > > > > > > > > > >> > Here is the list of it:
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >>
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > >
>> > > >> >> > > > > >
>> > > >> >> > > > >
>> > > >> >> > > >
>> > > >> >> > >
>> > > >> >> >
>> > > >> >>
>> > > >>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >> > Most of them seems to be standard ASF
>> based
>> > > >> >> > > requirements,
>> > > >> >> > > > > but
>> > > >> >> > > > > > > few
>> > > >> >> > > > > > > > > need
>> > > >> >> > > > > > > > > > >> to
>> > > >> >> > > > > > > > > > >> > be checked for:
>> > > >> >> > > > > > > > > > >> > 1. Software projects are expected to be
>> > > Licensed
>> > > >> >> under
>> > > >> >> > > > > Apache
>> > > >> >> > > > > > > > > License,
>> > > >> >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
>> > > expected
>> > > >> >> to be
>> > > >> >> > > > > > > compatible
>> > > >> >> > > > > > > > > with
>> > > >> >> > > > > > > > > > >> this
>> > > >> >> > > > > > > > > > >> > license)
>> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to
>> > check
>> > > if
>> > > >> >> > > > > dependencies
>> > > >> >> > > > > > of
>> > > >> >> > > > > > > > > Apex
>> > > >> >> > > > > > > > > > >> are
>> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess
>> would
>> > > be a
>> > > >> >> > > > > verification
>> > > >> >> > > > > > > > check.
>> > > >> >> > > > > > > > > > >> > 2. Software projects are expected to be
>> > > >> compatible
>> > > >> >> > with
>> > > >> >> > > > all
>> > > >> >> > > > > of
>> > > >> >> > > > > > > the
>> > > >> >> > > > > > > > > > >> > supported platforms that Bigtop
>> distribution
>> > > is
>> > > >> >> > > targeting
>> > > >> >> > > > > > > > > > >> >     - This needs verifying whether our
>> > > software
>> > > >> >> runs
>> > > >> >> > > fine
>> > > >> >> > > > in
>> > > >> >> > > > > > > > > centos-6
>> > > >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
>> > > >> >> > opensuse-13.2.
>> > > >> >> > > > > > > > > > >> > 3. What smoke tests that should be added
>> for
>> > > >> >> > deployment.
>> > > >> >> > > > > > > > > > >> > 4. Identifying the test artifacts which
>> goes
>> > > >> beyond
>> > > >> >> > > smoke
>> > > >> >> > > > > test
>> > > >> >> > > > > > > > > > >> >     - These are basically the integration
>> > > tests
>> > > >> for
>> > > >> >> > > > > > verification
>> > > >> >> > > > > > > > > after
>> > > >> >> > > > > > > > > > >> the
>> > > >> >> > > > > > > > > > >> > deployment. This is a soft requirement,
>> but
>> > > aim
>> > > >> is
>> > > >> >> to
>> > > >> >> > > > > achieve
>> > > >> >> > > > > > > this
>> > > >> >> > > > > > > > > as
>> > > >> >> > > > > > > > > > >> well
>> > > >> >> > > > > > > > > > >> > or at least have explanation why not to
>> > > include.
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >> > If there are any from the link which
>> > > explicitly
>> > > >> >> needs
>> > > >> >> > to
>> > > >> >> > > > be
>> > > >> >> > > > > > > > checked
>> > > >> >> > > > > > > > > > >> other
>> > > >> >> > > > > > > > > > >> > than above 4, please let us know.
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as component to
>> Bigtop*
>> > > >> >> > > > > > > > > > >> > From one of the mail archive of Bigtop,
>> it
>> > was
>> > > >> >> learnt
>> > > >> >> > > that
>> > > >> >> > > > > the
>> > > >> >> > > > > > > > > bigtop
>> > > >> >> > > > > > > > > > >> > community want to see the addition of new
>> > > >> >> components
>> > > >> >> > in
>> > > >> >> > > > > > phases.
>> > > >> >> > > > > > > > Here
>> > > >> >> > > > > > > > > > are
>> > > >> >> > > > > > > > > > >> > the phases:
>> > > >> >> > > > > > > > > > >> > 1. Packaging
>> > > >> >> > > > > > > > > > >> >     - This needs creating of package i.e.
>> > rpm
>> > > &
>> > > >> deb
>> > > >> >> > > files.
>> > > >> >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
>> > > >> DISCLAMER,
>> > > >> >> > NOTES
>> > > >> >> > > > etc
>> > > >> >> > > > > if
>> > > >> >> > > > > > > any
>> > > >> >> > > > > > > > > > >> needed.
>> > > >> >> > > > > > > > > > >> >     - Any documentation that need to be
>> > added
>> > > to
>> > > >> >> > > > > distribution
>> > > >> >> > > > > > of
>> > > >> >> > > > > > > > our
>> > > >> >> > > > > > > > > > >> > software.
>> > > >> >> > > > > > > > > > >> >     - Any license information of
>> > dependencies
>> > > >> >> required
>> > > >> >> > > to
>> > > >> >> > > > be
>> > > >> >> > > > > > > added
>> > > >> >> > > > > > > > > to
>> > > >> >> > > > > > > > > > >> > package
>> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
>> > > >> >> > > > > > > > > > >> >     - Adding smoke test for packaging.
>> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic
>> deployment
>> > and
>> > > >> >> > > > configuration
>> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for automatic
>> > > >> deployment
>> > > >> >> > > > > > > > > > >> > 4. Integration tests
>> > > >> >> > > > > > > > > > >> >     - For verification of deployments.
>> > > >> >> > > > > > > > > > >> > 5. license clearance:
>> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new
>> > > stuff
>> > > >> is
>> > > >> >> > > > compliant
>> > > >> >> > > > > > > with
>> > > >> >> > > > > > > > > ASF
>> > > >> >> > > > > > > > > > >> > license requirements. If you add code
>> > licenses
>> > > >> >> under
>> > > >> >> > > > > different
>> > > >> >> > > > > > > > > > licenses,
>> > > >> >> > > > > > > > > > >> > those would need to be listed in the
>> NOTICE.
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >> > Please share your thoughts on the
>> approach.
>> > > >> >> > > > > > > > > > >> > We'll start corresponding communication
>> on
>> > > >> bigtop
>> > > >> >> > > mailing
>> > > >> >> > > > > list
>> > > >> >> > > > > > > as
>> > > >> >> > > > > > > > > > well.
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >> > We have some specific
>> questions/suggestions
>> > > >> >> related to
>> > > >> >> > > > what
>> > > >> >> > > > > > > should
>> > > >> >> > > > > > > > > be
>> > > >> >> > > > > > > > > > >> the
>> > > >> >> > > > > > > > > > >> > content of the package and what should be
>> > the
>> > > >> smoke
>> > > >> >> > > tests,
>> > > >> >> > > > > but
>> > > >> >> > > > > > > in
>> > > >> >> > > > > > > > > the
>> > > >> >> > > > > > > > > > >> > interest of not having too much content
>> > here,
>> > > >> we'll
>> > > >> >> > put
>> > > >> >> > > > the
>> > > >> >> > > > > > > > > questions
>> > > >> >> > > > > > > > > > >> as a
>> > > >> >> > > > > > > > > > >> > separate mail in this mailthread.
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >> > Thanks,
>> > > >> >> > > > > > > > > > >> > Chinmay.
>> > > >> >> > > > > > > > > > >> >
>> > > >> >> > > > > > > > > > >>
>> > > >> >> > > > > > > > > > >
>> > > >> >> > > > > > > > > > >
>> > > >> >> > > > > > > > > >
>> > > >> >> > > > > > > > >
>> > > >> >> > > > > > > >
>> > > >> >> > > > > > >
>> > > >> >> > > > > >
>> > > >> >> > > > >
>> > > >> >> > > >
>> > > >> >> > >
>> > > >> >> >
>> > > >> >>
>> > > >> >
>> > > >> >
>> > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Thanks everyone for the feedback.

Also as Ram and David mentioned, I'll start creating the man page as well.

Thanks,
Chinmay.


On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <th...@datatorrent.com>
wrote:

> I think we already agreed that we will retain the current script name in
> this bigtop iteration. There is a ticket for renaming, add you suggestions
> there.
>
> --
> sent from mobile
> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <sa...@datatorrent.com> wrote:
>
> > +1 for Symlink from apex to dtcli or start the transition process by
> making
> > dtcli as the symlink to apex cli.
> >
> > Renaming is a breaking change should be done during major version number
> > change.
> >
> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <da...@datatorrent.com> wrote:
> >
> > > Apologies for hitting the "Send" button too early.
> > >
> > > What is the scope of the name change?  If we simply have a symlink from
> > > apex to dtcli, then it should have no backward compatibility problem at
> > > all.
> > >
> > > But the proper name change would involve:
> > > - Change the name of DTCli class to ApexCli
> > > - Change the prompt from dt> to apex>
> > > - Change the initial greeting when running the CLI from something like
> > "DT
> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > >
> > > The above changes may break existing scripts that rely on the the
> string
> > > "DT".
> > > But if folks are okay with this potential problem, I am okay with the
> > name
> > > change.
> > >
> > > For the man page, we can start with the content of this page:
> > >
> > > http://docs.datatorrent.com/dtcli/
> > >
> > > Plus the command line options when doing dtcli -h:
> > >
> > > usage: DTCli
> > >  -e <arg>    Commands are read from the argument
> > >  -f <arg>    Use the specified prompt at all time
> > >  -h          Print this help
> > >  -kp <arg>   Use the specified kerberos principal
> > >  -kt <arg>   Use the specified kerberos keytab
> > >  -p <arg>    JSONP padding function
> > >  -r          JSON Raw mode
> > >  -v          Verbose mode level 1
> > >  -vv         Verbose mode level 2
> > >  -vvv        Verbose mode level 3
> > >  -vvvv       Verbose mode level 4
> > >
> > > David
> > >
> > >
> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <da...@datatorrent.com>
> > wrote:
> > >
> > > > What is the scope of the name change?  If we simply have a symlink
> from
> > > > apex to dtcli, then it should have no backward compatibility problem
> at
> > > all.
> > > >
> > > > But the proper name change would involve:
> > > > - Change the name of DTCli class to ApexCli
> > > > - Change the prompt from dt> to apex>
> > > > - Change the initial greeting when running the CLI from something
> like
> > > "DT
> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > > >
> > > > The above changes may break existing scripts that rely on the the
> > string
> > > > "DT".
> > > > But if folks are okay with this potential problem, I am okay with the
> > > name
> > > > change.
> > > >
> > > > For the man page, we can start with the content of this page:
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> chinmay@apache.org
> > >
> > > > wrote:
> > > >
> > > >> Here is a suggestion I have related to name change:
> > > >>
> > > >> 1. While packaging we change the name of "dtcli" script to "apex".
> > This
> > > is
> > > >> done so that in future when we change the name to "apex", users of
> > > bigtop
> > > >> apex don't have to transition much.
> > > >> 2. We also keep a symlink named dtcli which points to apex script.
> > This
> > > is
> > > >> for backward compatibility.
> > > >> 3. We still keep conf/dt-env.sh for first integration in bigtop.
> This
> > is
> > > >> in
> > > >> the interest of not changing the content of dtcli or apex script.
> > > >> 4. In apex 3.4.0, we take care of all these naming related changes
> and
> > > >> update bigtop repository later to remove references to dtcli all
> > > together.
> > > >>
> > > >> Please share your thoughts on above approach.
> > > >>
> > > >> Also, please share what could be the man page content for "apex".
> > > >>
> > > >> Thanks,
> > > >> Chinmay.
> > > >>
> > > >>
> > > >>
> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> > chinmay@apache.org>
> > > >> wrote:
> > > >>
> > > >> > Yes.. It'll be build from source tar downloaded from one of the
> > apache
> > > >> > mirror: apache.osuosl.org
> > > >> >
> > > >> > The reason for dt-env.sh is it's sourced from dtcli. This leaves
> us
> > 3
> > > >> > options:
> > > >> > If we should not have any dt-env.sh, dtcli would need a change
> while
> > > >> > packaging.
> > > >> >
> > > >> > But if we want to use dtcli as it is, then we would need dt-env.sh
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> > thomas@datatorrent.com
> > > >
> > > >> > wrote:
> > > >> >
> > > >> >> Looks good overall, though there shouldn't be any dt-env.sh
> > > >> >>
> > > >> >> What will the bigtop package be built from, the source tar?
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > > chinmay@apache.org
> > > >> >
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Hi Thomas,
> > > >> >> >
> > > >> >> > Thanks for the feedback.
> > > >> >> >
> > > >> >> > First, we're not changing the name anywhere. We'll follow what
> > > >> >> currently is
> > > >> >> > in source tarball for 3.3.0-incubating version.
> > > >> >> >
> > > >> >> > Secondly, I've mentioned a directory structure below which is
> > > inline
> > > >> >> with
> > > >> >> > other existing integrations with bigtop.
> > > >> >> >
> > > >> >> > The need for each file is as follows:
> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> engine/src/main/scripts/
> > > of
> > > >> >> source
> > > >> >> > code apex 3.3.0-incubating.
> > > >> >> >
> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches
> for
> > > >> hadoop
> > > >> >> > binary path and exports an env variable for dtcli to use it.
> > > >> >> >
> > > >> >> > 3. lib/ -> Libraries for CLI. This list is extracted in a
> similar
> > > >> way to
> > > >> >> > how DT Community Edition finds dependency jars.
> > > >> >> > Please note that this is the first iteration list of jars. I'm
> > > >> trying to
> > > >> >> > narrow this down to only those which are really required.
> > > >> >> > The test that I'm running to check if dtcli runs fine with
> given
> > > >> >> dependency
> > > >> >> > is to launch+shutdown+kill+status for pi demo.
> > > >> >> >
> > > >> >> > Above is the directory structure which is required for CLI to
> > work.
> > > >> >> > What aniruddha and pradeep has mentioned are some additional
> > files
> > > >> which
> > > >> >> > makes the like of administrator easier. For eg. /etc/skel
> > > >> >> >
> > > >> >> > Please let me know if this is inline with what you're thinking.
> > > >> >> >
> > > >> >> > Thanks,
> > > >> >> > Chinmay.
> > > >> >> >
> > > >> >> >
> > > >> >> > .
> > > >> >> > |-- bin
> > > >> >> > |   `-- dtcli
> > > >> >> > |-- conf
> > > >> >> > |   `-- dt-env.sh
> > > >> >> > `-- lib
> > > >> >> >     |-- ant-1.9.2.jar
> > > >> >> >     |-- ant-launcher-1.9.2.jar
> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > > >> >> >     |-- apex-engine.jar
> > > >> >> >     |-- apex-engine-tests.jar
> > > >> >> >     |-- async-http-client-1.7.20.jar
> > > >> >> >     |-- bval-core-0.5.jar
> > > >> >> >     |-- bval-jsr303-0.5.jar
> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> > > >> >> >     |-- commons-codec-1.10.jar
> > > >> >> >     |-- commons-lang3-3.1.jar
> > > >> >> >     |-- commons-logging-1.1.3.jar
> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > > >> >> >     |-- httpclient-4.3.5.jar
> > > >> >> >     |-- httpcore-4.3.2.jar
> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > > >> >> >     |-- javax.servlet-3.1.jar
> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> > > >> >> >     |-- jersey-client-1.9.jar
> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > > >> >> >     |-- jline-2.11.jar
> > > >> >> >     |-- kryo-2.24.0.jar
> > > >> >> >     |-- mbassador-1.1.9.jar
> > > >> >> >     |-- minlog-1.2.jar
> > > >> >> >     |-- netlet-1.2.0.jar
> > > >> >> >     |-- netty-3.6.6.Final.jar
> > > >> >> >     |-- objenesis-2.1.jar
> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > > >> >> >     `-- zip4j-1.3.2.jar
> > > >> >> >
> > > >> >> > 3 directories, 40 files
> > > >> >> >
> > > >> >> >
> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > > >> thomas@datatorrent.com>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> > > Chinmay,
> > > >> >> > >
> > > >> >> > > Before discussing where to put the files, let's make sure
> they
> > > are
> > > >> >> really
> > > >> >> > > needed for the operation of the CLI. As for names, anything
> new
> > > >> needs
> > > >> >> to
> > > >> >> > > reflect Apex in the name and should follow common
> conventions,
> > > >> >> especially
> > > >> >> > > in Bigtop where there are many existing integrations to look
> > at.
> > > >> >> > >
> > > >> >> > > Thanks,
> > > >> >> > > Thomas
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > > >> >> > > chinmay@datatorrent.com>
> > > >> >> > > wrote:
> > > >> >> > >
> > > >> >> > > > Really good points Pradeep and Aniruddha.
> > > >> >> > > >
> > > >> >> > > > 1. I believe we won't need to change the dtcli considering
> it
> > > >> works
> > > >> >> > with
> > > >> >> > > DT
> > > >> >> > > > Community edition. We can keep the directory structure
> > similar
> > > to
> > > >> >> that.
> > > >> >> > > > dt-env.sh has variables which contains information required
> > for
> > > >> >> dtcli
> > > >> >> > to
> > > >> >> > > > launch.
> > > >> >> > > >
> > > >> >> > > > 2. Let me check with bigtop community that whether they
> > > >> facilitate
> > > >> >> the
> > > >> >> > > > installation of rpms/debs before user is created. In either
> > > case,
> > > >> >> > current
> > > >> >> > > > dtcli creates a .dt folder in home directory. Also before
> > > putting
> > > >> >> > > anything
> > > >> >> > > > in /etc/skel we need to define what are the default
> contents
> > > that
> > > >> >> > should
> > > >> >> > > go
> > > >> >> > > > to ~/.dt/ folder. If there is no defaults, probably we
> should
> > > not
> > > >> >> > > > explicitly add it in /etc/skel.
> > > >> >> > > >
> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This way contents
> of
> > > >> >> dt-env.sh
> > > >> >> > > are
> > > >> >> > > > present as env variables. But I see a catch there. Adding
> > > >> dt-env.sh
> > > >> >> to
> > > >> >> > > > /etc/profile.d would make all the variables available at
> > > runtime
> > > >> all
> > > >> >> > the
> > > >> >> > > > time. I feel a little skeptical about that. Maybe a
> possible
> > > >> >> collision
> > > >> >> > > can
> > > >> >> > > > occur with other application vars.
> > > >> >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > > >> >> > > >
> > > >> >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > > >> >> > > Jira/repository.
> > > >> >> > > > Packaging related code usually goes there. (That's what all
> > the
> > > >> >> > > components
> > > >> >> > > > in bigtop does).
> > > >> >> > > > Having said that, once a PR is created, I'll be sharing the
> > > link
> > > >> of
> > > >> >> PR
> > > >> >> > > > here, so that apex community as well can review it.
> > > >> >> > > >
> > > >> >> > > > Thanks,
> > > >> >> > > > Chinmay.
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> > > >> >> > > >
> > > >> >> > > > > +1 on suggestions and approach.
> > > >> >> > > > > We may need to iron out details about exact paths etc.
> > > >> >> > > > > Which can be done on jira / PR comments.
> > > >> >> > > > > Is that right @dev?
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > > >> >> > apache@pradeepdalvi.com>
> > > >> >> > > > > wrote:
> > > >> >> > > > >
> > > >> >> > > > > > Inline comments...
> > > >> >> > > > > >
> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > >> >> > > > > > chinmay@datatorrent.com>
> > > >> >> > > > > > wrote:
> > > >> >> > > > > >
> > > >> >> > > > > > > @Thomas, Not all the jar present in DT community
> > edition
> > > >> will
> > > >> >> be
> > > >> >> > > > > included
> > > >> >> > > > > > > there.
> > > >> >> > > > > > >
> > > >> >> > > > > > > Here is the list of jars I found from common
> > dependencies
> > > >> >> between
> > > >> >> > > of
> > > >> >> > > > > apex
> > > >> >> > > > > > > and DT Community edition:
> > > >> >> > > > > > > netlet-1.2.0.jar
> > > >> >> > > > > > > kryo-2.24.0.jar
> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > >> >> > > > > > > async-http-client-1.7.20.jar
> > > >> >> > > > > > > netty-3.6.6.Final.jar
> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > > >> >> > > > > > > bval-jsr303-0.5.jar
> > > >> >> > > > > > > bval-core-0.5.jar
> > > >> >> > > > > > > commons-lang3-3.1.jar
> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > > >> >> > > > > > > httpclient-4.3.5.jar
> > > >> >> > > > > > > commons-codec-1.10.jar
> > > >> >> > > > > > > zip4j-1.3.2.jar
> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > > >> >> > > > > > > jline-2.11.jar
> > > >> >> > > > > > > ant-1.9.2.jar
> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> > > >> >> > > > > > > mbassador-1.1.9.jar
> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > > >> >> > > > > > >
> > > >> >> > > > > > > Ofcourse, I'll be running some tests do check that
> > dtcli
> > > >> works
> > > >> >> > > > properly
> > > >> >> > > > > > for
> > > >> >> > > > > > > launch+shutdown+kill of apps with only these
> libraries
> > > >> >> present in
> > > >> >> > > > > > isolation
> > > >> >> > > > > > > without dependency on local m2.
> > > >> >> > > > > > > I'm believe that there are unwanted jars which are
> used
> > > for
> > > >> >> > compile
> > > >> >> > > > > time
> > > >> >> > > > > > > dependency and not runtime in above list which I can
> > drop
> > > >> to
> > > >> >> keep
> > > >> >> > > > > package
> > > >> >> > > > > > > size minimal.
> > > >> >> > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > > @Anniruddha,
> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh
> is
> > > >> >> required.
> > > >> >> > > > Only
> > > >> >> > > > > > > required values I see in it are classpath. and dtcli
> is
> > > >> >> capable
> > > >> >> > of
> > > >> >> > > > > > building
> > > >> >> > > > > > > it on the fly.
> > > >> >> > > > > > >
> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli
> can
> > > find
> > > >> >> the
> > > >> >> > jar
> > > >> >> > > > > path,
> > > >> >> > > > > > > dt-env.sh and all such conf files. If we need a
> change
> > in
> > > >> >> dtcli,
> > > >> >> > > then
> > > >> >> > > > > > > community's opinion is required for whether dtcli
> > should
> > > >> >> change
> > > >> >> > in
> > > >> >> > > > our
> > > >> >> > > > > > repo
> > > >> >> > > > > > > or a copy of that with required changes exist in
> bigtop
> > > >> repo
> > > >> >> > until
> > > >> >> > > we
> > > >> >> > > > > > make
> > > >> >> > > > > > > the dtcli generic enough.
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > > Can we have these file paths set using environment
> > > variables
> > > >> >> with
> > > >> >> > > some
> > > >> >> > > > > > default values in dtcli?
> > > >> >> > > > > > Then we can set such params in dt-env.sh.
> > > >> >> > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file different from
> > > >> >> dt-site.xml?
> > > >> >> > Not
> > > >> >> > > > > sure
> > > >> >> > > > > > if
> > > >> >> > > > > > > adding in /etc/skel would help. Files & Dirs from
> > > /etc/skel
> > > >> >> are
> > > >> >> > > > copied
> > > >> >> > > > > to
> > > >> >> > > > > > > home of new user when useradd program is called. But
> > for
> > > >> >> existing
> > > >> >> > > > users
> > > >> >> > > > > > > this won't be of any use. Moreover, dtcli creates
> > ~/.dt/
> > > on
> > > >> >> the
> > > >> >> > fly
> > > >> >> > > > if
> > > >> >> > > > > > not
> > > >> >> > > > > > > encountered for the first time. Again correct me if
> I'm
> > > >> wrong.
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > > Yes, you are right. However this would probably be
> > > necessary
> > > >> >> step
> > > >> >> > for
> > > >> >> > > > > > rpm/deb, as they may also get installed during OS
> install
> > > and
> > > >> >> > before
> > > >> >> > > > user
> > > >> >> > > > > > accounts were created.
> > > >> >> > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its
> > the
> > > >> best
> > > >> >> > > idea.
> > > >> >> > > > > > We'll
> > > >> >> > > > > > > anyway put dtcli in location which is by default
> > present
> > > in
> > > >> >> path
> > > >> >> > > i.e.
> > > >> >> > > > > > > /usr/bin.
> > > >> >> > > > > >
> > > >> >> > > > > > For other env variables specific to apex, I'll prefer
> to
> > > use
> > > >> >> > > dt-env.sh
> > > >> >> > > > > and
> > > >> >> > > > > > > source it in dtcli rather than changing bashrc etc...
> > > >> >> > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be
> the
> > > >> >> cleaner
> > > >> >> > way
> > > >> >> > > > to
> > > >> >> > > > > > achieve the same.
> > > >> >> > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > Thanks,
> > > >> >> > > > > > > Chinmay.
> > > >> >> > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> > > >> >> > > > > > >
> > > >> >> > > > > > > > Hi,
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > @Chinmay,
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > We need to consider following:
> > > >> >> > > > > > > > System wide default config files can be located at
> > > >> following
> > > >> >> > > > > locations:
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> > > >> behaviour
> > > >> >> on
> > > >> >> > > how
> > > >> >> > > > it
> > > >> >> > > > > > > finds
> > > >> >> > > > > > > > those locations)
> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> > > >> >> behaviour on
> > > >> >> > > how
> > > >> >> > > > > it
> > > >> >> > > > > > > > finds those locations)
> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new
> > > users
> > > >> >> that
> > > >> >> > > will
> > > >> >> > > > > be
> > > >> >> > > > > > > > created in system in future.)
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > We may also have to modify bashrc / bashprofile for
> > > >> >> population
> > > >> >> > > any
> > > >> >> > > > > > > > variables that are required.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > @dev, please put in your comments / suggestion.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > Thanks,
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > Aniruddha
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > >> >> > > > > thomas@datatorrent.com>
> > > >> >> > > > > > > > wrote:
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > > -->
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> Kolhatkar <
> > > >> >> > > > > > > > chinmay@datatorrent.com
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > wrote:
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > > Hello Everyone!!
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of
> > apex,
> > > >> here
> > > >> >> > are
> > > >> >> > > > some
> > > >> >> > > > > > > > > proposals
> > > >> >> > > > > > > > > > about package structure etc..
> > > >> >> > > > > > > > > > Before posting it on bbigtop mailing list, I
> have
> > > >> some
> > > >> >> > > question
> > > >> >> > > > > for
> > > >> >> > > > > > > > apex
> > > >> >> > > > > > > > > > community.
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > Proposed Directory structure of apex package
> for
> > > both
> > > >> >> deb &
> > > >> >> > > > rpm:
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > >> >> > > > > > > > > >
> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > >> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > >> >> > > > > > > > > >
> /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> > > >> >> > > > > > > > > > <other dependent jars> = All the 3rd party jars
> > > which
> > > >> >> are
> > > >> >> > > > > required
> > > >> >> > > > > > > for
> > > >> >> > > > > > > > > apex
> > > >> >> > > > > > > > > > to run. Usually the dependencies are packaged
> as
> > > >> part of
> > > >> >> > > > rpm/deb
> > > >> >> > > > > by
> > > >> >> > > > > > > any
> > > >> >> > > > > > > > > > software in bigtop.
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > Can you specify what those jars are in Bigtop
> > > context?
> > > >> >> Same
> > > >> >> > as
> > > >> >> > > > > > shipped
> > > >> >> > > > > > > > with
> > > >> >> > > > > > > > > DT community addition under lib/ ?
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of corresponding
> of
> > > 3rd
> > > >> >> party
> > > >> >> > > jars
> > > >> >> > > > > > which
> > > >> >> > > > > > > > > needs
> > > >> >> > > > > > > > > > to included while packaging.
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > Questions related to this:
> > > >> >> > > > > > > > > > 1. Should we call the cli of apex as "apex"
> > instead
> > > >> of
> > > >> >> > > "dtcli"
> > > >> >> > > > in
> > > >> >> > > > > > > > bigtop
> > > >> >> > > > > > > > > > package?
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > I think we should keep the name until we are able
> > to
> > > >> >> change
> > > >> >> > it
> > > >> >> > > in
> > > >> >> > > > > > Apex.
> > > >> >> > > > > > > > > Otherwise this may get confusing.
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > > 2. I see that all softwares in bigtop have man
> > page
> > > >> for
> > > >> >> > their
> > > >> >> > > > > > > > > executables.
> > > >> >> > > > > > > > > > I think we should have it too for dtcli. Is
> there
> > > any
> > > >> >> > > > > documentation
> > > >> >> > > > > > > > > which I
> > > >> >> > > > > > > > > > can convert to man page? or can I use output of
> > > >> "dtcli
> > > >> >> > > --help"?
> > > >> >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop
> > as
> > > >> >> 3.3.0 OR
> > > >> >> > > > > > > > > > 3.3.0-incubating?
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > Has to be -incubating
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > > 4. Is it ok for apex package of bigtop to
> depend
> > on
> > > >> >> 2.7.1
> > > >> >> > > > > version
> > > >> >> > > > > > of
> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that we see with
> this
> > > >> >> > dependency?
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > Should work without changed. I thought we
> certified
> > > >> >> against
> > > >> >> > > > 2.7.0?
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > > 5. Following is the apache mirror from which
> > bigtop
> > > >> will
> > > >> >> > pick
> > > >> >> > > > the
> > > >> >> > > > > > > apex
> > > >> >> > > > > > > > > > source for compilation and packaging. Please
> > > correct
> > > >> if
> > > >> >> > > > > incorrect:
> > > >> >> > > > > > > > > >
> > > >> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used
> > by
> > > >> all
> > > >> >> the
> > > >> >> > > > > > softwares
> > > >> >> > > > > > > in
> > > >> >> > > > > > > > > > bigtop.
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > Please share your opinion.
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > Thanks,
> > > >> >> > > > > > > > > > Chinmay.
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> > Kolhatkar
> > > <
> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > > >> >> > > > > > > > > > wrote:
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > > > I've already started discussion on bigtop dev
> > > >> mailing
> > > >> >> > list
> > > >> >> > > > for
> > > >> >> > > > > > > > > > > integration. Also created a Jira for the
> same.
> > > For
> > > >> >> this
> > > >> >> > > > > > communities
> > > >> >> > > > > > > > > > > reference, here is the bigtop Jira:
> > BIGTOP-2313.
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise
> <
> > > >> >> > > > > > > thomas@datatorrent.com
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > > > wrote:
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > > >> Chinmay,
> > > >> >> > > > > > > > > > >>
> > > >> >> > > > > > > > > > >> I don't see anything under prerequisites
> that
> > > >> would
> > > >> >> be a
> > > >> >> > > > > > problem.
> > > >> >> > > > > > > We
> > > >> >> > > > > > > > > > >> looked
> > > >> >> > > > > > > > > > >> at the ASF licencing compatibility as part
> of
> > > >> >> becoming
> > > >> >> > an
> > > >> >> > > > > > > incubator
> > > >> >> > > > > > > > > > >> project.
> > > >> >> > > > > > > > > > >>
> > > >> >> > > > > > > > > > >> Please focus on the packaging during the
> next
> > > >> weeks.
> > > >> >> > Since
> > > >> >> > > > the
> > > >> >> > > > > > > work
> > > >> >> > > > > > > > > will
> > > >> >> > > > > > > > > > >> be
> > > >> >> > > > > > > > > > >> part of Bigtop, related discussions and
> JIRAs
> > > >> should
> > > >> >> > also
> > > >> >> > > be
> > > >> >> > > > > > > there.
> > > >> >> > > > > > > > > > >>
> > > >> >> > > > > > > > > > >> Would be good to have the packaging in place
> > by
> > > >> end
> > > >> >> Feb.
> > > >> >> > > > > > > > > > >>
> > > >> >> > > > > > > > > > >> Thanks,
> > > >> >> > > > > > > > > > >> Thomas
> > > >> >> > > > > > > > > > >>
> > > >> >> > > > > > > > > > >>
> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> > > >> Kolhatkar <
> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> > > >> >> > > > > > > > > > >> wrote:
> > > >> >> > > > > > > > > > >>
> > > >> >> > > > > > > > > > >> > Hi All,
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >> > We're planning a work on adding Apache
> Apex
> > > as a
> > > >> >> > > component
> > > >> >> > > > > to
> > > >> >> > > > > > > > Apache
> > > >> >> > > > > > > > > > >> > Bigtop.
> > > >> >> > > > > > > > > > >> > Bigtop is the packaging system for the
> > Apache
> > > >> big
> > > >> >> data
> > > >> >> > > > > > > ecosystem.
> > > >> >> > > > > > > > > > >> Several
> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE
> > for
> > > >> the
> > > >> >> > same:
> > > >> >> > > > > > > > > > >> >
> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >> > Proposed plan of execution is as follows:
> > > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> > > expectation
> > > >> >> for
> > > >> >> > new
> > > >> >> > > > > > > > components
> > > >> >> > > > > > > > > to
> > > >> >> > > > > > > > > > >> get
> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
> > > >> >> > > > > > > > > > >> > Here is the list of it:
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >>
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >> > Most of them seems to be standard ASF
> based
> > > >> >> > > requirements,
> > > >> >> > > > > but
> > > >> >> > > > > > > few
> > > >> >> > > > > > > > > need
> > > >> >> > > > > > > > > > >> to
> > > >> >> > > > > > > > > > >> > be checked for:
> > > >> >> > > > > > > > > > >> > 1. Software projects are expected to be
> > > Licensed
> > > >> >> under
> > > >> >> > > > > Apache
> > > >> >> > > > > > > > > License,
> > > >> >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
> > > expected
> > > >> >> to be
> > > >> >> > > > > > > compatible
> > > >> >> > > > > > > > > with
> > > >> >> > > > > > > > > > >> this
> > > >> >> > > > > > > > > > >> > license)
> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to
> > check
> > > if
> > > >> >> > > > > dependencies
> > > >> >> > > > > > of
> > > >> >> > > > > > > > > Apex
> > > >> >> > > > > > > > > > >> are
> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess
> would
> > > be a
> > > >> >> > > > > verification
> > > >> >> > > > > > > > check.
> > > >> >> > > > > > > > > > >> > 2. Software projects are expected to be
> > > >> compatible
> > > >> >> > with
> > > >> >> > > > all
> > > >> >> > > > > of
> > > >> >> > > > > > > the
> > > >> >> > > > > > > > > > >> > supported platforms that Bigtop
> distribution
> > > is
> > > >> >> > > targeting
> > > >> >> > > > > > > > > > >> >     - This needs verifying whether our
> > > software
> > > >> >> runs
> > > >> >> > > fine
> > > >> >> > > > in
> > > >> >> > > > > > > > > centos-6
> > > >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > > >> >> > opensuse-13.2.
> > > >> >> > > > > > > > > > >> > 3. What smoke tests that should be added
> for
> > > >> >> > deployment.
> > > >> >> > > > > > > > > > >> > 4. Identifying the test artifacts which
> goes
> > > >> beyond
> > > >> >> > > smoke
> > > >> >> > > > > test
> > > >> >> > > > > > > > > > >> >     - These are basically the integration
> > > tests
> > > >> for
> > > >> >> > > > > > verification
> > > >> >> > > > > > > > > after
> > > >> >> > > > > > > > > > >> the
> > > >> >> > > > > > > > > > >> > deployment. This is a soft requirement,
> but
> > > aim
> > > >> is
> > > >> >> to
> > > >> >> > > > > achieve
> > > >> >> > > > > > > this
> > > >> >> > > > > > > > > as
> > > >> >> > > > > > > > > > >> well
> > > >> >> > > > > > > > > > >> > or at least have explanation why not to
> > > include.
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >> > If there are any from the link which
> > > explicitly
> > > >> >> needs
> > > >> >> > to
> > > >> >> > > > be
> > > >> >> > > > > > > > checked
> > > >> >> > > > > > > > > > >> other
> > > >> >> > > > > > > > > > >> > than above 4, please let us know.
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as component to
> Bigtop*
> > > >> >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it
> > was
> > > >> >> learnt
> > > >> >> > > that
> > > >> >> > > > > the
> > > >> >> > > > > > > > > bigtop
> > > >> >> > > > > > > > > > >> > community want to see the addition of new
> > > >> >> components
> > > >> >> > in
> > > >> >> > > > > > phases.
> > > >> >> > > > > > > > Here
> > > >> >> > > > > > > > > > are
> > > >> >> > > > > > > > > > >> > the phases:
> > > >> >> > > > > > > > > > >> > 1. Packaging
> > > >> >> > > > > > > > > > >> >     - This needs creating of package i.e.
> > rpm
> > > &
> > > >> deb
> > > >> >> > > files.
> > > >> >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> > > >> DISCLAMER,
> > > >> >> > NOTES
> > > >> >> > > > etc
> > > >> >> > > > > if
> > > >> >> > > > > > > any
> > > >> >> > > > > > > > > > >> needed.
> > > >> >> > > > > > > > > > >> >     - Any documentation that need to be
> > added
> > > to
> > > >> >> > > > > distribution
> > > >> >> > > > > > of
> > > >> >> > > > > > > > our
> > > >> >> > > > > > > > > > >> > software.
> > > >> >> > > > > > > > > > >> >     - Any license information of
> > dependencies
> > > >> >> required
> > > >> >> > > to
> > > >> >> > > > be
> > > >> >> > > > > > > added
> > > >> >> > > > > > > > > to
> > > >> >> > > > > > > > > > >> > package
> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > >> >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment
> > and
> > > >> >> > > > configuration
> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for automatic
> > > >> deployment
> > > >> >> > > > > > > > > > >> > 4. Integration tests
> > > >> >> > > > > > > > > > >> >     - For verification of deployments.
> > > >> >> > > > > > > > > > >> > 5. license clearance:
> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new
> > > stuff
> > > >> is
> > > >> >> > > > compliant
> > > >> >> > > > > > > with
> > > >> >> > > > > > > > > ASF
> > > >> >> > > > > > > > > > >> > license requirements. If you add code
> > licenses
> > > >> >> under
> > > >> >> > > > > different
> > > >> >> > > > > > > > > > licenses,
> > > >> >> > > > > > > > > > >> > those would need to be listed in the
> NOTICE.
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >> > Please share your thoughts on the
> approach.
> > > >> >> > > > > > > > > > >> > We'll start corresponding communication on
> > > >> bigtop
> > > >> >> > > mailing
> > > >> >> > > > > list
> > > >> >> > > > > > > as
> > > >> >> > > > > > > > > > well.
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >> > We have some specific
> questions/suggestions
> > > >> >> related to
> > > >> >> > > > what
> > > >> >> > > > > > > should
> > > >> >> > > > > > > > > be
> > > >> >> > > > > > > > > > >> the
> > > >> >> > > > > > > > > > >> > content of the package and what should be
> > the
> > > >> smoke
> > > >> >> > > tests,
> > > >> >> > > > > but
> > > >> >> > > > > > > in
> > > >> >> > > > > > > > > the
> > > >> >> > > > > > > > > > >> > interest of not having too much content
> > here,
> > > >> we'll
> > > >> >> > put
> > > >> >> > > > the
> > > >> >> > > > > > > > > questions
> > > >> >> > > > > > > > > > >> as a
> > > >> >> > > > > > > > > > >> > separate mail in this mailthread.
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >> > Thanks,
> > > >> >> > > > > > > > > > >> > Chinmay.
> > > >> >> > > > > > > > > > >> >
> > > >> >> > > > > > > > > > >>
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

I think we already agreed that we will retain the current script name in
this bigtop iteration. There is a ticket for renaming, add you suggestions
there.

--
sent from mobile
On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <sa...@datatorrent.com> wrote:

> +1 for Symlink from apex to dtcli or start the transition process by making
> dtcli as the symlink to apex cli.
>
> Renaming is a breaking change should be done during major version number
> change.
>
> On Mon, Feb 15, 2016 at 9:57 AM David Yan <da...@datatorrent.com> wrote:
>
> > Apologies for hitting the "Send" button too early.
> >
> > What is the scope of the name change?  If we simply have a symlink from
> > apex to dtcli, then it should have no backward compatibility problem at
> > all.
> >
> > But the proper name change would involve:
> > - Change the name of DTCli class to ApexCli
> > - Change the prompt from dt> to apex>
> > - Change the initial greeting when running the CLI from something like
> "DT
> > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> >
> > The above changes may break existing scripts that rely on the the string
> > "DT".
> > But if folks are okay with this potential problem, I am okay with the
> name
> > change.
> >
> > For the man page, we can start with the content of this page:
> >
> > http://docs.datatorrent.com/dtcli/
> >
> > Plus the command line options when doing dtcli -h:
> >
> > usage: DTCli
> >  -e <arg>    Commands are read from the argument
> >  -f <arg>    Use the specified prompt at all time
> >  -h          Print this help
> >  -kp <arg>   Use the specified kerberos principal
> >  -kt <arg>   Use the specified kerberos keytab
> >  -p <arg>    JSONP padding function
> >  -r          JSON Raw mode
> >  -v          Verbose mode level 1
> >  -vv         Verbose mode level 2
> >  -vvv        Verbose mode level 3
> >  -vvvv       Verbose mode level 4
> >
> > David
> >
> >
> > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <da...@datatorrent.com>
> wrote:
> >
> > > What is the scope of the name change?  If we simply have a symlink from
> > > apex to dtcli, then it should have no backward compatibility problem at
> > all.
> > >
> > > But the proper name change would involve:
> > > - Change the name of DTCli class to ApexCli
> > > - Change the prompt from dt> to apex>
> > > - Change the initial greeting when running the CLI from something like
> > "DT
> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> > >
> > > The above changes may break existing scripts that rely on the the
> string
> > > "DT".
> > > But if folks are okay with this potential problem, I am okay with the
> > name
> > > change.
> > >
> > > For the man page, we can start with the content of this page:
> > >
> > >
> > >
> > >
> > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <chinmay@apache.org
> >
> > > wrote:
> > >
> > >> Here is a suggestion I have related to name change:
> > >>
> > >> 1. While packaging we change the name of "dtcli" script to "apex".
> This
> > is
> > >> done so that in future when we change the name to "apex", users of
> > bigtop
> > >> apex don't have to transition much.
> > >> 2. We also keep a symlink named dtcli which points to apex script.
> This
> > is
> > >> for backward compatibility.
> > >> 3. We still keep conf/dt-env.sh for first integration in bigtop. This
> is
> > >> in
> > >> the interest of not changing the content of dtcli or apex script.
> > >> 4. In apex 3.4.0, we take care of all these naming related changes and
> > >> update bigtop repository later to remove references to dtcli all
> > together.
> > >>
> > >> Please share your thoughts on above approach.
> > >>
> > >> Also, please share what could be the man page content for "apex".
> > >>
> > >> Thanks,
> > >> Chinmay.
> > >>
> > >>
> > >>
> > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> chinmay@apache.org>
> > >> wrote:
> > >>
> > >> > Yes.. It'll be build from source tar downloaded from one of the
> apache
> > >> > mirror: apache.osuosl.org
> > >> >
> > >> > The reason for dt-env.sh is it's sourced from dtcli. This leaves us
> 3
> > >> > options:
> > >> > If we should not have any dt-env.sh, dtcli would need a change while
> > >> > packaging.
> > >> >
> > >> > But if we want to use dtcli as it is, then we would need dt-env.sh
> > >> >
> > >> >
> > >> >
> > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> thomas@datatorrent.com
> > >
> > >> > wrote:
> > >> >
> > >> >> Looks good overall, though there shouldn't be any dt-env.sh
> > >> >>
> > >> >> What will the bigtop package be built from, the source tar?
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> > chinmay@apache.org
> > >> >
> > >> >> wrote:
> > >> >>
> > >> >> > Hi Thomas,
> > >> >> >
> > >> >> > Thanks for the feedback.
> > >> >> >
> > >> >> > First, we're not changing the name anywhere. We'll follow what
> > >> >> currently is
> > >> >> > in source tarball for 3.3.0-incubating version.
> > >> >> >
> > >> >> > Secondly, I've mentioned a directory structure below which is
> > inline
> > >> >> with
> > >> >> > other existing integrations with bigtop.
> > >> >> >
> > >> >> > The need for each file is as follows:
> > >> >> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/
> > of
> > >> >> source
> > >> >> > code apex 3.3.0-incubating.
> > >> >> >
> > >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for
> > >> hadoop
> > >> >> > binary path and exports an env variable for dtcli to use it.
> > >> >> >
> > >> >> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar
> > >> way to
> > >> >> > how DT Community Edition finds dependency jars.
> > >> >> > Please note that this is the first iteration list of jars. I'm
> > >> trying to
> > >> >> > narrow this down to only those which are really required.
> > >> >> > The test that I'm running to check if dtcli runs fine with given
> > >> >> dependency
> > >> >> > is to launch+shutdown+kill+status for pi demo.
> > >> >> >
> > >> >> > Above is the directory structure which is required for CLI to
> work.
> > >> >> > What aniruddha and pradeep has mentioned are some additional
> files
> > >> which
> > >> >> > makes the like of administrator easier. For eg. /etc/skel
> > >> >> >
> > >> >> > Please let me know if this is inline with what you're thinking.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > Chinmay.
> > >> >> >
> > >> >> >
> > >> >> > .
> > >> >> > |-- bin
> > >> >> > |   `-- dtcli
> > >> >> > |-- conf
> > >> >> > |   `-- dt-env.sh
> > >> >> > `-- lib
> > >> >> >     |-- ant-1.9.2.jar
> > >> >> >     |-- ant-launcher-1.9.2.jar
> > >> >> >     |-- apex-api-3.3.0-incubating.jar
> > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > >> >> >     |-- apex-common-3.3.0-incubating.jar
> > >> >> >     |-- apex-engine.jar
> > >> >> >     |-- apex-engine-tests.jar
> > >> >> >     |-- async-http-client-1.7.20.jar
> > >> >> >     |-- bval-core-0.5.jar
> > >> >> >     |-- bval-jsr303-0.5.jar
> > >> >> >     |-- commons-beanutils-1.8.3.jar
> > >> >> >     |-- commons-codec-1.10.jar
> > >> >> >     |-- commons-lang3-3.1.jar
> > >> >> >     |-- commons-logging-1.1.3.jar
> > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> > >> >> >     |-- httpclient-4.3.5.jar
> > >> >> >     |-- httpcore-4.3.2.jar
> > >> >> >     |-- jackson-core-asl-1.9.2.jar
> > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> > >> >> >     |-- javax.servlet-3.1.jar
> > >> >> >     |-- javax.servlet-api-3.0.1.jar
> > >> >> >     |-- jersey-apache-client4-1.9.jar
> > >> >> >     |-- jersey-client-1.9.jar
> > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > >> >> >     |-- jline-2.11.jar
> > >> >> >     |-- kryo-2.24.0.jar
> > >> >> >     |-- mbassador-1.1.9.jar
> > >> >> >     |-- minlog-1.2.jar
> > >> >> >     |-- netlet-1.2.0.jar
> > >> >> >     |-- netty-3.6.6.Final.jar
> > >> >> >     |-- objenesis-2.1.jar
> > >> >> >     |-- validation-api-1.1.0.Final.jar
> > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> > >> >> >     `-- zip4j-1.3.2.jar
> > >> >> >
> > >> >> > 3 directories, 40 files
> > >> >> >
> > >> >> >
> > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> > >> thomas@datatorrent.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Chinmay,
> > >> >> > >
> > >> >> > > Before discussing where to put the files, let's make sure they
> > are
> > >> >> really
> > >> >> > > needed for the operation of the CLI. As for names, anything new
> > >> needs
> > >> >> to
> > >> >> > > reflect Apex in the name and should follow common conventions,
> > >> >> especially
> > >> >> > > in Bigtop where there are many existing integrations to look
> at.
> > >> >> > >
> > >> >> > > Thanks,
> > >> >> > > Thomas
> > >> >> > >
> > >> >> > >
> > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > >> >> > > chinmay@datatorrent.com>
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > > Really good points Pradeep and Aniruddha.
> > >> >> > > >
> > >> >> > > > 1. I believe we won't need to change the dtcli considering it
> > >> works
> > >> >> > with
> > >> >> > > DT
> > >> >> > > > Community edition. We can keep the directory structure
> similar
> > to
> > >> >> that.
> > >> >> > > > dt-env.sh has variables which contains information required
> for
> > >> >> dtcli
> > >> >> > to
> > >> >> > > > launch.
> > >> >> > > >
> > >> >> > > > 2. Let me check with bigtop community that whether they
> > >> facilitate
> > >> >> the
> > >> >> > > > installation of rpms/debs before user is created. In either
> > case,
> > >> >> > current
> > >> >> > > > dtcli creates a .dt folder in home directory. Also before
> > putting
> > >> >> > > anything
> > >> >> > > > in /etc/skel we need to define what are the default contents
> > that
> > >> >> > should
> > >> >> > > go
> > >> >> > > > to ~/.dt/ folder. If there is no defaults, probably we should
> > not
> > >> >> > > > explicitly add it in /etc/skel.
> > >> >> > > >
> > >> >> > > > 3.  /etc/profile.d approach looks nice. This way contents of
> > >> >> dt-env.sh
> > >> >> > > are
> > >> >> > > > present as env variables. But I see a catch there. Adding
> > >> dt-env.sh
> > >> >> to
> > >> >> > > > /etc/profile.d would make all the variables available at
> > runtime
> > >> all
> > >> >> > the
> > >> >> > > > time. I feel a little skeptical about that. Maybe a possible
> > >> >> collision
> > >> >> > > can
> > >> >> > > > occur with other application vars.
> > >> >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > >> >> > > >
> > >> >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > >> >> > > Jira/repository.
> > >> >> > > > Packaging related code usually goes there. (That's what all
> the
> > >> >> > > components
> > >> >> > > > in bigtop does).
> > >> >> > > > Having said that, once a PR is created, I'll be sharing the
> > link
> > >> of
> > >> >> PR
> > >> >> > > > here, so that apex community as well can review it.
> > >> >> > > >
> > >> >> > > > Thanks,
> > >> >> > > > Chinmay.
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > >> >> > > > aniruddha@datatorrent.com> wrote:
> > >> >> > > >
> > >> >> > > > > +1 on suggestions and approach.
> > >> >> > > > > We may need to iron out details about exact paths etc.
> > >> >> > > > > Which can be done on jira / PR comments.
> > >> >> > > > > Is that right @dev?
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > >> >> > apache@pradeepdalvi.com>
> > >> >> > > > > wrote:
> > >> >> > > > >
> > >> >> > > > > > Inline comments...
> > >> >> > > > > >
> > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > >> >> > > > > > chinmay@datatorrent.com>
> > >> >> > > > > > wrote:
> > >> >> > > > > >
> > >> >> > > > > > > @Thomas, Not all the jar present in DT community
> edition
> > >> will
> > >> >> be
> > >> >> > > > > included
> > >> >> > > > > > > there.
> > >> >> > > > > > >
> > >> >> > > > > > > Here is the list of jars I found from common
> dependencies
> > >> >> between
> > >> >> > > of
> > >> >> > > > > apex
> > >> >> > > > > > > and DT Community edition:
> > >> >> > > > > > > netlet-1.2.0.jar
> > >> >> > > > > > > kryo-2.24.0.jar
> > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > >> >> > > > > > > async-http-client-1.7.20.jar
> > >> >> > > > > > > netty-3.6.6.Final.jar
> > >> >> > > > > > > validation-api-1.1.0.Final.jar
> > >> >> > > > > > > bval-jsr303-0.5.jar
> > >> >> > > > > > > bval-core-0.5.jar
> > >> >> > > > > > > commons-lang3-3.1.jar
> > >> >> > > > > > > commons-beanutils-1.8.3.jar
> > >> >> > > > > > > httpclient-4.3.5.jar
> > >> >> > > > > > > commons-codec-1.10.jar
> > >> >> > > > > > > zip4j-1.3.2.jar
> > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> > >> >> > > > > > > jersey-apache-client4-1.9.jar
> > >> >> > > > > > > jline-2.11.jar
> > >> >> > > > > > > ant-1.9.2.jar
> > >> >> > > > > > > ant-launcher-1.9.2.jar
> > >> >> > > > > > > mbassador-1.1.9.jar
> > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> > >> >> > > > > > > jackson-xc-1.9.2.jar
> > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> > >> >> > > > > > >
> > >> >> > > > > > > Ofcourse, I'll be running some tests do check that
> dtcli
> > >> works
> > >> >> > > > properly
> > >> >> > > > > > for
> > >> >> > > > > > > launch+shutdown+kill of apps with only these libraries
> > >> >> present in
> > >> >> > > > > > isolation
> > >> >> > > > > > > without dependency on local m2.
> > >> >> > > > > > > I'm believe that there are unwanted jars which are used
> > for
> > >> >> > compile
> > >> >> > > > > time
> > >> >> > > > > > > dependency and not runtime in above list which I can
> drop
> > >> to
> > >> >> keep
> > >> >> > > > > package
> > >> >> > > > > > > size minimal.
> > >> >> > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > > @Anniruddha,
> > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> > >> >> required.
> > >> >> > > > Only
> > >> >> > > > > > > required values I see in it are classpath. and dtcli is
> > >> >> capable
> > >> >> > of
> > >> >> > > > > > building
> > >> >> > > > > > > it on the fly.
> > >> >> > > > > > >
> > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can
> > find
> > >> >> the
> > >> >> > jar
> > >> >> > > > > path,
> > >> >> > > > > > > dt-env.sh and all such conf files. If we need a change
> in
> > >> >> dtcli,
> > >> >> > > then
> > >> >> > > > > > > community's opinion is required for whether dtcli
> should
> > >> >> change
> > >> >> > in
> > >> >> > > > our
> > >> >> > > > > > repo
> > >> >> > > > > > > or a copy of that with required changes exist in bigtop
> > >> repo
> > >> >> > until
> > >> >> > > we
> > >> >> > > > > > make
> > >> >> > > > > > > the dtcli generic enough.
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > > > Can we have these file paths set using environment
> > variables
> > >> >> with
> > >> >> > > some
> > >> >> > > > > > default values in dtcli?
> > >> >> > > > > > Then we can set such params in dt-env.sh.
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > > > > 3. dt-sited.xml -> How is this file different from
> > >> >> dt-site.xml?
> > >> >> > Not
> > >> >> > > > > sure
> > >> >> > > > > > if
> > >> >> > > > > > > adding in /etc/skel would help. Files & Dirs from
> > /etc/skel
> > >> >> are
> > >> >> > > > copied
> > >> >> > > > > to
> > >> >> > > > > > > home of new user when useradd program is called. But
> for
> > >> >> existing
> > >> >> > > > users
> > >> >> > > > > > > this won't be of any use. Moreover, dtcli creates
> ~/.dt/
> > on
> > >> >> the
> > >> >> > fly
> > >> >> > > > if
> > >> >> > > > > > not
> > >> >> > > > > > > encountered for the first time. Again correct me if I'm
> > >> wrong.
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > > > Yes, you are right. However this would probably be
> > necessary
> > >> >> step
> > >> >> > for
> > >> >> > > > > > rpm/deb, as they may also get installed during OS install
> > and
> > >> >> > before
> > >> >> > > > user
> > >> >> > > > > > accounts were created.
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its
> the
> > >> best
> > >> >> > > idea.
> > >> >> > > > > > We'll
> > >> >> > > > > > > anyway put dtcli in location which is by default
> present
> > in
> > >> >> path
> > >> >> > > i.e.
> > >> >> > > > > > > /usr/bin.
> > >> >> > > > > >
> > >> >> > > > > > For other env variables specific to apex, I'll prefer to
> > use
> > >> >> > > dt-env.sh
> > >> >> > > > > and
> > >> >> > > > > > > source it in dtcli rather than changing bashrc etc...
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> > >> >> cleaner
> > >> >> > way
> > >> >> > > > to
> > >> >> > > > > > achieve the same.
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > Thanks,
> > >> >> > > > > > > Chinmay.
> > >> >> > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> > >> >> > > > > > >
> > >> >> > > > > > > > Hi,
> > >> >> > > > > > > >
> > >> >> > > > > > > > @Chinmay,
> > >> >> > > > > > > >
> > >> >> > > > > > > > We need to consider following:
> > >> >> > > > > > > > System wide default config files can be located at
> > >> following
> > >> >> > > > > locations:
> > >> >> > > > > > > >
> > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> > >> behaviour
> > >> >> on
> > >> >> > > how
> > >> >> > > > it
> > >> >> > > > > > > finds
> > >> >> > > > > > > > those locations)
> > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> > >> >> behaviour on
> > >> >> > > how
> > >> >> > > > > it
> > >> >> > > > > > > > finds those locations)
> > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new
> > users
> > >> >> that
> > >> >> > > will
> > >> >> > > > > be
> > >> >> > > > > > > > created in system in future.)
> > >> >> > > > > > > >
> > >> >> > > > > > > > We may also have to modify bashrc / bashprofile for
> > >> >> population
> > >> >> > > any
> > >> >> > > > > > > > variables that are required.
> > >> >> > > > > > > >
> > >> >> > > > > > > > @dev, please put in your comments / suggestion.
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > Thanks,
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > Aniruddha
> > >> >> > > > > > > >
> > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > >> >> > > > > thomas@datatorrent.com>
> > >> >> > > > > > > > wrote:
> > >> >> > > > > > > >
> > >> >> > > > > > > > > -->
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > >> >> > > > > > > > chinmay@datatorrent.com
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > wrote:
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > > Hello Everyone!!
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of
> apex,
> > >> here
> > >> >> > are
> > >> >> > > > some
> > >> >> > > > > > > > > proposals
> > >> >> > > > > > > > > > about package structure etc..
> > >> >> > > > > > > > > > Before posting it on bbigtop mailing list, I have
> > >> some
> > >> >> > > question
> > >> >> > > > > for
> > >> >> > > > > > > > apex
> > >> >> > > > > > > > > > community.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Proposed Directory structure of apex package for
> > both
> > >> >> deb &
> > >> >> > > > rpm:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > >> >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > >> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > >> >> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> > >> >> > > > > > > > > > <other dependent jars> = All the 3rd party jars
> > which
> > >> >> are
> > >> >> > > > > required
> > >> >> > > > > > > for
> > >> >> > > > > > > > > apex
> > >> >> > > > > > > > > > to run. Usually the dependencies are packaged as
> > >> part of
> > >> >> > > > rpm/deb
> > >> >> > > > > by
> > >> >> > > > > > > any
> > >> >> > > > > > > > > > software in bigtop.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Can you specify what those jars are in Bigtop
> > context?
> > >> >> Same
> > >> >> > as
> > >> >> > > > > > shipped
> > >> >> > > > > > > > with
> > >> >> > > > > > > > > DT community addition under lib/ ?
> > >> >> > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > > <package LICENSE> = Licenses of corresponding of
> > 3rd
> > >> >> party
> > >> >> > > jars
> > >> >> > > > > > which
> > >> >> > > > > > > > > needs
> > >> >> > > > > > > > > > to included while packaging.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Questions related to this:
> > >> >> > > > > > > > > > 1. Should we call the cli of apex as "apex"
> instead
> > >> of
> > >> >> > > "dtcli"
> > >> >> > > > in
> > >> >> > > > > > > > bigtop
> > >> >> > > > > > > > > > package?
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > I think we should keep the name until we are able
> to
> > >> >> change
> > >> >> > it
> > >> >> > > in
> > >> >> > > > > > Apex.
> > >> >> > > > > > > > > Otherwise this may get confusing.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > > 2. I see that all softwares in bigtop have man
> page
> > >> for
> > >> >> > their
> > >> >> > > > > > > > > executables.
> > >> >> > > > > > > > > > I think we should have it too for dtcli. Is there
> > any
> > >> >> > > > > documentation
> > >> >> > > > > > > > > which I
> > >> >> > > > > > > > > > can convert to man page? or can I use output of
> > >> "dtcli
> > >> >> > > --help"?
> > >> >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop
> as
> > >> >> 3.3.0 OR
> > >> >> > > > > > > > > > 3.3.0-incubating?
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Has to be -incubating
> > >> >> > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend
> on
> > >> >> 2.7.1
> > >> >> > > > > version
> > >> >> > > > > > of
> > >> >> > > > > > > > > > bigtop hadoop? Any problems that we see with this
> > >> >> > dependency?
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Should work without changed. I thought we certified
> > >> >> against
> > >> >> > > > 2.7.0?
> > >> >> > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > > 5. Following is the apache mirror from which
> bigtop
> > >> will
> > >> >> > pick
> > >> >> > > > the
> > >> >> > > > > > > apex
> > >> >> > > > > > > > > > source for compilation and packaging. Please
> > correct
> > >> if
> > >> >> > > > > incorrect:
> > >> >> > > > > > > > > >
> > >> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used
> by
> > >> all
> > >> >> the
> > >> >> > > > > > softwares
> > >> >> > > > > > > in
> > >> >> > > > > > > > > > bigtop.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Please share your opinion.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Thanks,
> > >> >> > > > > > > > > > Chinmay.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay
> Kolhatkar
> > <
> > >> >> > > > > > > > > > chinmay@datatorrent.com>
> > >> >> > > > > > > > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > > I've already started discussion on bigtop dev
> > >> mailing
> > >> >> > list
> > >> >> > > > for
> > >> >> > > > > > > > > > > integration. Also created a Jira for the same.
> > For
> > >> >> this
> > >> >> > > > > > communities
> > >> >> > > > > > > > > > > reference, here is the bigtop Jira:
> BIGTOP-2313.
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > >> >> > > > > > > thomas@datatorrent.com
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > > > wrote:
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > >> Chinmay,
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> I don't see anything under prerequisites that
> > >> would
> > >> >> be a
> > >> >> > > > > > problem.
> > >> >> > > > > > > We
> > >> >> > > > > > > > > > >> looked
> > >> >> > > > > > > > > > >> at the ASF licencing compatibility as part of
> > >> >> becoming
> > >> >> > an
> > >> >> > > > > > > incubator
> > >> >> > > > > > > > > > >> project.
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> Please focus on the packaging during the next
> > >> weeks.
> > >> >> > Since
> > >> >> > > > the
> > >> >> > > > > > > work
> > >> >> > > > > > > > > will
> > >> >> > > > > > > > > > >> be
> > >> >> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> > >> should
> > >> >> > also
> > >> >> > > be
> > >> >> > > > > > > there.
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> Would be good to have the packaging in place
> by
> > >> end
> > >> >> Feb.
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> Thanks,
> > >> >> > > > > > > > > > >> Thomas
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> > >> Kolhatkar <
> > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> > >> >> > > > > > > > > > >> wrote:
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> > Hi All,
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >> > We're planning a work on adding Apache Apex
> > as a
> > >> >> > > component
> > >> >> > > > > to
> > >> >> > > > > > > > Apache
> > >> >> > > > > > > > > > >> > Bigtop.
> > >> >> > > > > > > > > > >> > Bigtop is the packaging system for the
> Apache
> > >> big
> > >> >> data
> > >> >> > > > > > > ecosystem.
> > >> >> > > > > > > > > > >> Several
> > >> >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE
> for
> > >> the
> > >> >> > same:
> > >> >> > > > > > > > > > >> >
> > >> https://issues.apache.org/jira/browse/APEXCORE-331
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >> > Proposed plan of execution is as follows:
> > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > >> >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> > expectation
> > >> >> for
> > >> >> > new
> > >> >> > > > > > > > components
> > >> >> > > > > > > > > to
> > >> >> > > > > > > > > > >> get
> > >> >> > > > > > > > > > >> > integrated into Bigtop.
> > >> >> > > > > > > > > > >> > Here is the list of it:
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >> > Most of them seems to be standard ASF based
> > >> >> > > requirements,
> > >> >> > > > > but
> > >> >> > > > > > > few
> > >> >> > > > > > > > > need
> > >> >> > > > > > > > > > >> to
> > >> >> > > > > > > > > > >> > be checked for:
> > >> >> > > > > > > > > > >> > 1. Software projects are expected to be
> > Licensed
> > >> >> under
> > >> >> > > > > Apache
> > >> >> > > > > > > > > License,
> > >> >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
> > expected
> > >> >> to be
> > >> >> > > > > > > compatible
> > >> >> > > > > > > > > with
> > >> >> > > > > > > > > > >> this
> > >> >> > > > > > > > > > >> > license)
> > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to
> check
> > if
> > >> >> > > > > dependencies
> > >> >> > > > > > of
> > >> >> > > > > > > > > Apex
> > >> >> > > > > > > > > > >> are
> > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would
> > be a
> > >> >> > > > > verification
> > >> >> > > > > > > > check.
> > >> >> > > > > > > > > > >> > 2. Software projects are expected to be
> > >> compatible
> > >> >> > with
> > >> >> > > > all
> > >> >> > > > > of
> > >> >> > > > > > > the
> > >> >> > > > > > > > > > >> > supported platforms that Bigtop distribution
> > is
> > >> >> > > targeting
> > >> >> > > > > > > > > > >> >     - This needs verifying whether our
> > software
> > >> >> runs
> > >> >> > > fine
> > >> >> > > > in
> > >> >> > > > > > > > > centos-6
> > >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > >> >> > opensuse-13.2.
> > >> >> > > > > > > > > > >> > 3. What smoke tests that should be added for
> > >> >> > deployment.
> > >> >> > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> > >> beyond
> > >> >> > > smoke
> > >> >> > > > > test
> > >> >> > > > > > > > > > >> >     - These are basically the integration
> > tests
> > >> for
> > >> >> > > > > > verification
> > >> >> > > > > > > > > after
> > >> >> > > > > > > > > > >> the
> > >> >> > > > > > > > > > >> > deployment. This is a soft requirement, but
> > aim
> > >> is
> > >> >> to
> > >> >> > > > > achieve
> > >> >> > > > > > > this
> > >> >> > > > > > > > > as
> > >> >> > > > > > > > > > >> well
> > >> >> > > > > > > > > > >> > or at least have explanation why not to
> > include.
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >> > If there are any from the link which
> > explicitly
> > >> >> needs
> > >> >> > to
> > >> >> > > > be
> > >> >> > > > > > > > checked
> > >> >> > > > > > > > > > >> other
> > >> >> > > > > > > > > > >> > than above 4, please let us know.
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > >> >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it
> was
> > >> >> learnt
> > >> >> > > that
> > >> >> > > > > the
> > >> >> > > > > > > > > bigtop
> > >> >> > > > > > > > > > >> > community want to see the addition of new
> > >> >> components
> > >> >> > in
> > >> >> > > > > > phases.
> > >> >> > > > > > > > Here
> > >> >> > > > > > > > > > are
> > >> >> > > > > > > > > > >> > the phases:
> > >> >> > > > > > > > > > >> > 1. Packaging
> > >> >> > > > > > > > > > >> >     - This needs creating of package i.e.
> rpm
> > &
> > >> deb
> > >> >> > > files.
> > >> >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> > >> DISCLAMER,
> > >> >> > NOTES
> > >> >> > > > etc
> > >> >> > > > > if
> > >> >> > > > > > > any
> > >> >> > > > > > > > > > >> needed.
> > >> >> > > > > > > > > > >> >     - Any documentation that need to be
> added
> > to
> > >> >> > > > > distribution
> > >> >> > > > > > of
> > >> >> > > > > > > > our
> > >> >> > > > > > > > > > >> > software.
> > >> >> > > > > > > > > > >> >     - Any license information of
> dependencies
> > >> >> required
> > >> >> > > to
> > >> >> > > > be
> > >> >> > > > > > > added
> > >> >> > > > > > > > > to
> > >> >> > > > > > > > > > >> > package
> > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > >> >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> > >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment
> and
> > >> >> > > > configuration
> > >> >> > > > > > > > > > >> >     - Add puppet recipes for automatic
> > >> deployment
> > >> >> > > > > > > > > > >> > 4. Integration tests
> > >> >> > > > > > > > > > >> >     - For verification of deployments.
> > >> >> > > > > > > > > > >> > 5. license clearance:
> > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new
> > stuff
> > >> is
> > >> >> > > > compliant
> > >> >> > > > > > > with
> > >> >> > > > > > > > > ASF
> > >> >> > > > > > > > > > >> > license requirements. If you add code
> licenses
> > >> >> under
> > >> >> > > > > different
> > >> >> > > > > > > > > > licenses,
> > >> >> > > > > > > > > > >> > those would need to be listed in the NOTICE.
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >> > Please share your thoughts on the approach.
> > >> >> > > > > > > > > > >> > We'll start corresponding communication on
> > >> bigtop
> > >> >> > > mailing
> > >> >> > > > > list
> > >> >> > > > > > > as
> > >> >> > > > > > > > > > well.
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >> > We have some specific questions/suggestions
> > >> >> related to
> > >> >> > > > what
> > >> >> > > > > > > should
> > >> >> > > > > > > > > be
> > >> >> > > > > > > > > > >> the
> > >> >> > > > > > > > > > >> > content of the package and what should be
> the
> > >> smoke
> > >> >> > > tests,
> > >> >> > > > > but
> > >> >> > > > > > > in
> > >> >> > > > > > > > > the
> > >> >> > > > > > > > > > >> > interest of not having too much content
> here,
> > >> we'll
> > >> >> > put
> > >> >> > > > the
> > >> >> > > > > > > > > questions
> > >> >> > > > > > > > > > >> as a
> > >> >> > > > > > > > > > >> > separate mail in this mailthread.
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >> > Thanks,
> > >> >> > > > > > > > > > >> > Chinmay.
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Sandesh Hegde <sa...@datatorrent.com>]

+1 for Symlink from apex to dtcli or start the transition process by making
dtcli as the symlink to apex cli.

Renaming is a breaking change should be done during major version number
change.

On Mon, Feb 15, 2016 at 9:57 AM David Yan <da...@datatorrent.com> wrote:

> Apologies for hitting the "Send" button too early.
>
> What is the scope of the name change?  If we simply have a symlink from
> apex to dtcli, then it should have no backward compatibility problem at
> all.
>
> But the proper name change would involve:
> - Change the name of DTCli class to ApexCli
> - Change the prompt from dt> to apex>
> - Change the initial greeting when running the CLI from something like "DT
> CLI 3.3.0-incubating ..." to "Apex CLI ...".
>
> The above changes may break existing scripts that rely on the the string
> "DT".
> But if folks are okay with this potential problem, I am okay with the name
> change.
>
> For the man page, we can start with the content of this page:
>
> http://docs.datatorrent.com/dtcli/
>
> Plus the command line options when doing dtcli -h:
>
> usage: DTCli
>  -e <arg>    Commands are read from the argument
>  -f <arg>    Use the specified prompt at all time
>  -h          Print this help
>  -kp <arg>   Use the specified kerberos principal
>  -kt <arg>   Use the specified kerberos keytab
>  -p <arg>    JSONP padding function
>  -r          JSON Raw mode
>  -v          Verbose mode level 1
>  -vv         Verbose mode level 2
>  -vvv        Verbose mode level 3
>  -vvvv       Verbose mode level 4
>
> David
>
>
> On Mon, Feb 15, 2016 at 9:56 AM, David Yan <da...@datatorrent.com> wrote:
>
> > What is the scope of the name change?  If we simply have a symlink from
> > apex to dtcli, then it should have no backward compatibility problem at
> all.
> >
> > But the proper name change would involve:
> > - Change the name of DTCli class to ApexCli
> > - Change the prompt from dt> to apex>
> > - Change the initial greeting when running the CLI from something like
> "DT
> > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> >
> > The above changes may break existing scripts that rely on the the string
> > "DT".
> > But if folks are okay with this potential problem, I am okay with the
> name
> > change.
> >
> > For the man page, we can start with the content of this page:
> >
> >
> >
> >
> > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <ch...@apache.org>
> > wrote:
> >
> >> Here is a suggestion I have related to name change:
> >>
> >> 1. While packaging we change the name of "dtcli" script to "apex". This
> is
> >> done so that in future when we change the name to "apex", users of
> bigtop
> >> apex don't have to transition much.
> >> 2. We also keep a symlink named dtcli which points to apex script. This
> is
> >> for backward compatibility.
> >> 3. We still keep conf/dt-env.sh for first integration in bigtop. This is
> >> in
> >> the interest of not changing the content of dtcli or apex script.
> >> 4. In apex 3.4.0, we take care of all these naming related changes and
> >> update bigtop repository later to remove references to dtcli all
> together.
> >>
> >> Please share your thoughts on above approach.
> >>
> >> Also, please share what could be the man page content for "apex".
> >>
> >> Thanks,
> >> Chinmay.
> >>
> >>
> >>
> >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <ch...@apache.org>
> >> wrote:
> >>
> >> > Yes.. It'll be build from source tar downloaded from one of the apache
> >> > mirror: apache.osuosl.org
> >> >
> >> > The reason for dt-env.sh is it's sourced from dtcli. This leaves us 3
> >> > options:
> >> > If we should not have any dt-env.sh, dtcli would need a change while
> >> > packaging.
> >> >
> >> > But if we want to use dtcli as it is, then we would need dt-env.sh
> >> >
> >> >
> >> >
> >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <thomas@datatorrent.com
> >
> >> > wrote:
> >> >
> >> >> Looks good overall, though there shouldn't be any dt-env.sh
> >> >>
> >> >> What will the bigtop package be built from, the source tar?
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> chinmay@apache.org
> >> >
> >> >> wrote:
> >> >>
> >> >> > Hi Thomas,
> >> >> >
> >> >> > Thanks for the feedback.
> >> >> >
> >> >> > First, we're not changing the name anywhere. We'll follow what
> >> >> currently is
> >> >> > in source tarball for 3.3.0-incubating version.
> >> >> >
> >> >> > Secondly, I've mentioned a directory structure below which is
> inline
> >> >> with
> >> >> > other existing integrations with bigtop.
> >> >> >
> >> >> > The need for each file is as follows:
> >> >> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/
> of
> >> >> source
> >> >> > code apex 3.3.0-incubating.
> >> >> >
> >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for
> >> hadoop
> >> >> > binary path and exports an env variable for dtcli to use it.
> >> >> >
> >> >> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar
> >> way to
> >> >> > how DT Community Edition finds dependency jars.
> >> >> > Please note that this is the first iteration list of jars. I'm
> >> trying to
> >> >> > narrow this down to only those which are really required.
> >> >> > The test that I'm running to check if dtcli runs fine with given
> >> >> dependency
> >> >> > is to launch+shutdown+kill+status for pi demo.
> >> >> >
> >> >> > Above is the directory structure which is required for CLI to work.
> >> >> > What aniruddha and pradeep has mentioned are some additional files
> >> which
> >> >> > makes the like of administrator easier. For eg. /etc/skel
> >> >> >
> >> >> > Please let me know if this is inline with what you're thinking.
> >> >> >
> >> >> > Thanks,
> >> >> > Chinmay.
> >> >> >
> >> >> >
> >> >> > .
> >> >> > |-- bin
> >> >> > |   `-- dtcli
> >> >> > |-- conf
> >> >> > |   `-- dt-env.sh
> >> >> > `-- lib
> >> >> >     |-- ant-1.9.2.jar
> >> >> >     |-- ant-launcher-1.9.2.jar
> >> >> >     |-- apex-api-3.3.0-incubating.jar
> >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> >> >> >     |-- apex-common-3.3.0-incubating.jar
> >> >> >     |-- apex-engine.jar
> >> >> >     |-- apex-engine-tests.jar
> >> >> >     |-- async-http-client-1.7.20.jar
> >> >> >     |-- bval-core-0.5.jar
> >> >> >     |-- bval-jsr303-0.5.jar
> >> >> >     |-- commons-beanutils-1.8.3.jar
> >> >> >     |-- commons-codec-1.10.jar
> >> >> >     |-- commons-lang3-3.1.jar
> >> >> >     |-- commons-logging-1.1.3.jar
> >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> >> >> >     |-- hadoop-common-2.2.0-tests.jar
> >> >> >     |-- httpclient-4.3.5.jar
> >> >> >     |-- httpcore-4.3.2.jar
> >> >> >     |-- jackson-core-asl-1.9.2.jar
> >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> >> >> >     |-- javax.servlet-3.1.jar
> >> >> >     |-- javax.servlet-api-3.0.1.jar
> >> >> >     |-- jersey-apache-client4-1.9.jar
> >> >> >     |-- jersey-client-1.9.jar
> >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> >> >> >     |-- jline-2.11.jar
> >> >> >     |-- kryo-2.24.0.jar
> >> >> >     |-- mbassador-1.1.9.jar
> >> >> >     |-- minlog-1.2.jar
> >> >> >     |-- netlet-1.2.0.jar
> >> >> >     |-- netty-3.6.6.Final.jar
> >> >> >     |-- objenesis-2.1.jar
> >> >> >     |-- validation-api-1.1.0.Final.jar
> >> >> >     |-- xbean-asm5-shaded-4.3.jar
> >> >> >     `-- zip4j-1.3.2.jar
> >> >> >
> >> >> > 3 directories, 40 files
> >> >> >
> >> >> >
> >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> >> thomas@datatorrent.com>
> >> >> > wrote:
> >> >> >
> >> >> > > Chinmay,
> >> >> > >
> >> >> > > Before discussing where to put the files, let's make sure they
> are
> >> >> really
> >> >> > > needed for the operation of the CLI. As for names, anything new
> >> needs
> >> >> to
> >> >> > > reflect Apex in the name and should follow common conventions,
> >> >> especially
> >> >> > > in Bigtop where there are many existing integrations to look at.
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Thomas
> >> >> > >
> >> >> > >
> >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> >> >> > > chinmay@datatorrent.com>
> >> >> > > wrote:
> >> >> > >
> >> >> > > > Really good points Pradeep and Aniruddha.
> >> >> > > >
> >> >> > > > 1. I believe we won't need to change the dtcli considering it
> >> works
> >> >> > with
> >> >> > > DT
> >> >> > > > Community edition. We can keep the directory structure similar
> to
> >> >> that.
> >> >> > > > dt-env.sh has variables which contains information required for
> >> >> dtcli
> >> >> > to
> >> >> > > > launch.
> >> >> > > >
> >> >> > > > 2. Let me check with bigtop community that whether they
> >> facilitate
> >> >> the
> >> >> > > > installation of rpms/debs before user is created. In either
> case,
> >> >> > current
> >> >> > > > dtcli creates a .dt folder in home directory. Also before
> putting
> >> >> > > anything
> >> >> > > > in /etc/skel we need to define what are the default contents
> that
> >> >> > should
> >> >> > > go
> >> >> > > > to ~/.dt/ folder. If there is no defaults, probably we should
> not
> >> >> > > > explicitly add it in /etc/skel.
> >> >> > > >
> >> >> > > > 3.  /etc/profile.d approach looks nice. This way contents of
> >> >> dt-env.sh
> >> >> > > are
> >> >> > > > present as env variables. But I see a catch there. Adding
> >> dt-env.sh
> >> >> to
> >> >> > > > /etc/profile.d would make all the variables available at
> runtime
> >> all
> >> >> > the
> >> >> > > > time. I feel a little skeptical about that. Maybe a possible
> >> >> collision
> >> >> > > can
> >> >> > > > occur with other application vars.
> >> >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> >> >> > > >
> >> >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> >> >> > > Jira/repository.
> >> >> > > > Packaging related code usually goes there. (That's what all the
> >> >> > > components
> >> >> > > > in bigtop does).
> >> >> > > > Having said that, once a PR is created, I'll be sharing the
> link
> >> of
> >> >> PR
> >> >> > > > here, so that apex community as well can review it.
> >> >> > > >
> >> >> > > > Thanks,
> >> >> > > > Chinmay.
> >> >> > > >
> >> >> > > >
> >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> >> >> > > > aniruddha@datatorrent.com> wrote:
> >> >> > > >
> >> >> > > > > +1 on suggestions and approach.
> >> >> > > > > We may need to iron out details about exact paths etc.
> >> >> > > > > Which can be done on jira / PR comments.
> >> >> > > > > Is that right @dev?
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> >> >> > apache@pradeepdalvi.com>
> >> >> > > > > wrote:
> >> >> > > > >
> >> >> > > > > > Inline comments...
> >> >> > > > > >
> >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> >> >> > > > > > chinmay@datatorrent.com>
> >> >> > > > > > wrote:
> >> >> > > > > >
> >> >> > > > > > > @Thomas, Not all the jar present in DT community edition
> >> will
> >> >> be
> >> >> > > > > included
> >> >> > > > > > > there.
> >> >> > > > > > >
> >> >> > > > > > > Here is the list of jars I found from common dependencies
> >> >> between
> >> >> > > of
> >> >> > > > > apex
> >> >> > > > > > > and DT Community edition:
> >> >> > > > > > > netlet-1.2.0.jar
> >> >> > > > > > > kryo-2.24.0.jar
> >> >> > > > > > > jackson-core-asl-1.9.2.jar
> >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> >> >> > > > > > > async-http-client-1.7.20.jar
> >> >> > > > > > > netty-3.6.6.Final.jar
> >> >> > > > > > > validation-api-1.1.0.Final.jar
> >> >> > > > > > > bval-jsr303-0.5.jar
> >> >> > > > > > > bval-core-0.5.jar
> >> >> > > > > > > commons-lang3-3.1.jar
> >> >> > > > > > > commons-beanutils-1.8.3.jar
> >> >> > > > > > > httpclient-4.3.5.jar
> >> >> > > > > > > commons-codec-1.10.jar
> >> >> > > > > > > zip4j-1.3.2.jar
> >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> >> >> > > > > > > jersey-apache-client4-1.9.jar
> >> >> > > > > > > jline-2.11.jar
> >> >> > > > > > > ant-1.9.2.jar
> >> >> > > > > > > ant-launcher-1.9.2.jar
> >> >> > > > > > > mbassador-1.1.9.jar
> >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> >> >> > > > > > > jackson-xc-1.9.2.jar
> >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> >> >> > > > > > >
> >> >> > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> >> works
> >> >> > > > properly
> >> >> > > > > > for
> >> >> > > > > > > launch+shutdown+kill of apps with only these libraries
> >> >> present in
> >> >> > > > > > isolation
> >> >> > > > > > > without dependency on local m2.
> >> >> > > > > > > I'm believe that there are unwanted jars which are used
> for
> >> >> > compile
> >> >> > > > > time
> >> >> > > > > > > dependency and not runtime in above list which I can drop
> >> to
> >> >> keep
> >> >> > > > > package
> >> >> > > > > > > size minimal.
> >> >> > > > > > >
> >> >> > > > > > >
> >> >> > > > > > > @Anniruddha,
> >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> >> >> required.
> >> >> > > > Only
> >> >> > > > > > > required values I see in it are classpath. and dtcli is
> >> >> capable
> >> >> > of
> >> >> > > > > > building
> >> >> > > > > > > it on the fly.
> >> >> > > > > > >
> >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can
> find
> >> >> the
> >> >> > jar
> >> >> > > > > path,
> >> >> > > > > > > dt-env.sh and all such conf files. If we need a change in
> >> >> dtcli,
> >> >> > > then
> >> >> > > > > > > community's opinion is required for whether dtcli should
> >> >> change
> >> >> > in
> >> >> > > > our
> >> >> > > > > > repo
> >> >> > > > > > > or a copy of that with required changes exist in bigtop
> >> repo
> >> >> > until
> >> >> > > we
> >> >> > > > > > make
> >> >> > > > > > > the dtcli generic enough.
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > > > Can we have these file paths set using environment
> variables
> >> >> with
> >> >> > > some
> >> >> > > > > > default values in dtcli?
> >> >> > > > > > Then we can set such params in dt-env.sh.
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > > > 3. dt-sited.xml -> How is this file different from
> >> >> dt-site.xml?
> >> >> > Not
> >> >> > > > > sure
> >> >> > > > > > if
> >> >> > > > > > > adding in /etc/skel would help. Files & Dirs from
> /etc/skel
> >> >> are
> >> >> > > > copied
> >> >> > > > > to
> >> >> > > > > > > home of new user when useradd program is called. But for
> >> >> existing
> >> >> > > > users
> >> >> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/
> on
> >> >> the
> >> >> > fly
> >> >> > > > if
> >> >> > > > > > not
> >> >> > > > > > > encountered for the first time. Again correct me if I'm
> >> wrong.
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > > > Yes, you are right. However this would probably be
> necessary
> >> >> step
> >> >> > for
> >> >> > > > > > rpm/deb, as they may also get installed during OS install
> and
> >> >> > before
> >> >> > > > user
> >> >> > > > > > accounts were created.
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
> >> best
> >> >> > > idea.
> >> >> > > > > > We'll
> >> >> > > > > > > anyway put dtcli in location which is by default present
> in
> >> >> path
> >> >> > > i.e.
> >> >> > > > > > > /usr/bin.
> >> >> > > > > >
> >> >> > > > > > For other env variables specific to apex, I'll prefer to
> use
> >> >> > > dt-env.sh
> >> >> > > > > and
> >> >> > > > > > > source it in dtcli rather than changing bashrc etc...
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> >> >> cleaner
> >> >> > way
> >> >> > > > to
> >> >> > > > > > achieve the same.
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > > >
> >> >> > > > > > Thanks,
> >> >> > > > > > > Chinmay.
> >> >> > > > > > >
> >> >> > > > > > >
> >> >> > > > > > >
> >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> >> >> > > > > > >
> >> >> > > > > > > > Hi,
> >> >> > > > > > > >
> >> >> > > > > > > > @Chinmay,
> >> >> > > > > > > >
> >> >> > > > > > > > We need to consider following:
> >> >> > > > > > > > System wide default config files can be located at
> >> following
> >> >> > > > > locations:
> >> >> > > > > > > >
> >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> >> behaviour
> >> >> on
> >> >> > > how
> >> >> > > > it
> >> >> > > > > > > finds
> >> >> > > > > > > > those locations)
> >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> >> >> behaviour on
> >> >> > > how
> >> >> > > > > it
> >> >> > > > > > > > finds those locations)
> >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new
> users
> >> >> that
> >> >> > > will
> >> >> > > > > be
> >> >> > > > > > > > created in system in future.)
> >> >> > > > > > > >
> >> >> > > > > > > > We may also have to modify bashrc / bashprofile for
> >> >> population
> >> >> > > any
> >> >> > > > > > > > variables that are required.
> >> >> > > > > > > >
> >> >> > > > > > > > @dev, please put in your comments / suggestion.
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > > Thanks,
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > > Aniruddha
> >> >> > > > > > > >
> >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> >> >> > > > > thomas@datatorrent.com>
> >> >> > > > > > > > wrote:
> >> >> > > > > > > >
> >> >> > > > > > > > > -->
> >> >> > > > > > > > >
> >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> >> >> > > > > > > > chinmay@datatorrent.com
> >> >> > > > > > > > > >
> >> >> > > > > > > > > wrote:
> >> >> > > > > > > > >
> >> >> > > > > > > > > > Hello Everyone!!
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
> >> here
> >> >> > are
> >> >> > > > some
> >> >> > > > > > > > > proposals
> >> >> > > > > > > > > > about package structure etc..
> >> >> > > > > > > > > > Before posting it on bbigtop mailing list, I have
> >> some
> >> >> > > question
> >> >> > > > > for
> >> >> > > > > > > > apex
> >> >> > > > > > > > > > community.
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > Proposed Directory structure of apex package for
> both
> >> >> deb &
> >> >> > > > rpm:
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> >> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> >> >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> >> >> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >> >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> >> >> > > > > > > > >
> >> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> >> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> >> >> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> >> >> > > > > > > > > >
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> >> >> > > > > > > > > > <other dependent jars> = All the 3rd party jars
> which
> >> >> are
> >> >> > > > > required
> >> >> > > > > > > for
> >> >> > > > > > > > > apex
> >> >> > > > > > > > > > to run. Usually the dependencies are packaged as
> >> part of
> >> >> > > > rpm/deb
> >> >> > > > > by
> >> >> > > > > > > any
> >> >> > > > > > > > > > software in bigtop.
> >> >> > > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > > > Can you specify what those jars are in Bigtop
> context?
> >> >> Same
> >> >> > as
> >> >> > > > > > shipped
> >> >> > > > > > > > with
> >> >> > > > > > > > > DT community addition under lib/ ?
> >> >> > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > > > > <package LICENSE> = Licenses of corresponding of
> 3rd
> >> >> party
> >> >> > > jars
> >> >> > > > > > which
> >> >> > > > > > > > > needs
> >> >> > > > > > > > > > to included while packaging.
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > Questions related to this:
> >> >> > > > > > > > > > 1. Should we call the cli of apex as "apex" instead
> >> of
> >> >> > > "dtcli"
> >> >> > > > in
> >> >> > > > > > > > bigtop
> >> >> > > > > > > > > > package?
> >> >> > > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > > > I think we should keep the name until we are able to
> >> >> change
> >> >> > it
> >> >> > > in
> >> >> > > > > > Apex.
> >> >> > > > > > > > > Otherwise this may get confusing.
> >> >> > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > > > > 2. I see that all softwares in bigtop have man page
> >> for
> >> >> > their
> >> >> > > > > > > > > executables.
> >> >> > > > > > > > > > I think we should have it too for dtcli. Is there
> any
> >> >> > > > > documentation
> >> >> > > > > > > > > which I
> >> >> > > > > > > > > > can convert to man page? or can I use output of
> >> "dtcli
> >> >> > > --help"?
> >> >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> >> >> 3.3.0 OR
> >> >> > > > > > > > > > 3.3.0-incubating?
> >> >> > > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > > > Has to be -incubating
> >> >> > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
> >> >> 2.7.1
> >> >> > > > > version
> >> >> > > > > > of
> >> >> > > > > > > > > > bigtop hadoop? Any problems that we see with this
> >> >> > dependency?
> >> >> > > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > > > Should work without changed. I thought we certified
> >> >> against
> >> >> > > > 2.7.0?
> >> >> > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > > > > 5. Following is the apache mirror from which bigtop
> >> will
> >> >> > pick
> >> >> > > > the
> >> >> > > > > > > apex
> >> >> > > > > > > > > > source for compilation and packaging. Please
> correct
> >> if
> >> >> > > > > incorrect:
> >> >> > > > > > > > > >
> >> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
> >> all
> >> >> the
> >> >> > > > > > softwares
> >> >> > > > > > > in
> >> >> > > > > > > > > > bigtop.
> >> >> > > > > > > > > >
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > Please share your opinion.
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > Thanks,
> >> >> > > > > > > > > > Chinmay.
> >> >> > > > > > > > > >
> >> >> > > > > > > > > >
> >> >> > > > > > > > > >
> >> >> > > > > > > > > >
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar
> <
> >> >> > > > > > > > > > chinmay@datatorrent.com>
> >> >> > > > > > > > > > wrote:
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > > > I've already started discussion on bigtop dev
> >> mailing
> >> >> > list
> >> >> > > > for
> >> >> > > > > > > > > > > integration. Also created a Jira for the same.
> For
> >> >> this
> >> >> > > > > > communities
> >> >> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> >> >> > > > > > > thomas@datatorrent.com
> >> >> > > > > > > > >
> >> >> > > > > > > > > > > wrote:
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > > >> Chinmay,
> >> >> > > > > > > > > > >>
> >> >> > > > > > > > > > >> I don't see anything under prerequisites that
> >> would
> >> >> be a
> >> >> > > > > > problem.
> >> >> > > > > > > We
> >> >> > > > > > > > > > >> looked
> >> >> > > > > > > > > > >> at the ASF licencing compatibility as part of
> >> >> becoming
> >> >> > an
> >> >> > > > > > > incubator
> >> >> > > > > > > > > > >> project.
> >> >> > > > > > > > > > >>
> >> >> > > > > > > > > > >> Please focus on the packaging during the next
> >> weeks.
> >> >> > Since
> >> >> > > > the
> >> >> > > > > > > work
> >> >> > > > > > > > > will
> >> >> > > > > > > > > > >> be
> >> >> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> >> should
> >> >> > also
> >> >> > > be
> >> >> > > > > > > there.
> >> >> > > > > > > > > > >>
> >> >> > > > > > > > > > >> Would be good to have the packaging in place by
> >> end
> >> >> Feb.
> >> >> > > > > > > > > > >>
> >> >> > > > > > > > > > >> Thanks,
> >> >> > > > > > > > > > >> Thomas
> >> >> > > > > > > > > > >>
> >> >> > > > > > > > > > >>
> >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> >> Kolhatkar <
> >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> >> >> > > > > > > > > > >> wrote:
> >> >> > > > > > > > > > >>
> >> >> > > > > > > > > > >> > Hi All,
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >> > We're planning a work on adding Apache Apex
> as a
> >> >> > > component
> >> >> > > > > to
> >> >> > > > > > > > Apache
> >> >> > > > > > > > > > >> > Bigtop.
> >> >> > > > > > > > > > >> > Bigtop is the packaging system for the Apache
> >> big
> >> >> data
> >> >> > > > > > > ecosystem.
> >> >> > > > > > > > > > >> Several
> >> >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for
> >> the
> >> >> > same:
> >> >> > > > > > > > > > >> >
> >> https://issues.apache.org/jira/browse/APEXCORE-331
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >> > Proposed plan of execution is as follows:
> >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> >> >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> expectation
> >> >> for
> >> >> > new
> >> >> > > > > > > > components
> >> >> > > > > > > > > to
> >> >> > > > > > > > > > >> get
> >> >> > > > > > > > > > >> > integrated into Bigtop.
> >> >> > > > > > > > > > >> > Here is the list of it:
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >>
> >> >> > > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >> > Most of them seems to be standard ASF based
> >> >> > > requirements,
> >> >> > > > > but
> >> >> > > > > > > few
> >> >> > > > > > > > > need
> >> >> > > > > > > > > > >> to
> >> >> > > > > > > > > > >> > be checked for:
> >> >> > > > > > > > > > >> > 1. Software projects are expected to be
> Licensed
> >> >> under
> >> >> > > > > Apache
> >> >> > > > > > > > > License,
> >> >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
> expected
> >> >> to be
> >> >> > > > > > > compatible
> >> >> > > > > > > > > with
> >> >> > > > > > > > > > >> this
> >> >> > > > > > > > > > >> > license)
> >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check
> if
> >> >> > > > > dependencies
> >> >> > > > > > of
> >> >> > > > > > > > > Apex
> >> >> > > > > > > > > > >> are
> >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would
> be a
> >> >> > > > > verification
> >> >> > > > > > > > check.
> >> >> > > > > > > > > > >> > 2. Software projects are expected to be
> >> compatible
> >> >> > with
> >> >> > > > all
> >> >> > > > > of
> >> >> > > > > > > the
> >> >> > > > > > > > > > >> > supported platforms that Bigtop distribution
> is
> >> >> > > targeting
> >> >> > > > > > > > > > >> >     - This needs verifying whether our
> software
> >> >> runs
> >> >> > > fine
> >> >> > > > in
> >> >> > > > > > > > > centos-6
> >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> >> >> > opensuse-13.2.
> >> >> > > > > > > > > > >> > 3. What smoke tests that should be added for
> >> >> > deployment.
> >> >> > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> >> beyond
> >> >> > > smoke
> >> >> > > > > test
> >> >> > > > > > > > > > >> >     - These are basically the integration
> tests
> >> for
> >> >> > > > > > verification
> >> >> > > > > > > > > after
> >> >> > > > > > > > > > >> the
> >> >> > > > > > > > > > >> > deployment. This is a soft requirement, but
> aim
> >> is
> >> >> to
> >> >> > > > > achieve
> >> >> > > > > > > this
> >> >> > > > > > > > > as
> >> >> > > > > > > > > > >> well
> >> >> > > > > > > > > > >> > or at least have explanation why not to
> include.
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >> > If there are any from the link which
> explicitly
> >> >> needs
> >> >> > to
> >> >> > > > be
> >> >> > > > > > > > checked
> >> >> > > > > > > > > > >> other
> >> >> > > > > > > > > > >> > than above 4, please let us know.
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> >> >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
> >> >> learnt
> >> >> > > that
> >> >> > > > > the
> >> >> > > > > > > > > bigtop
> >> >> > > > > > > > > > >> > community want to see the addition of new
> >> >> components
> >> >> > in
> >> >> > > > > > phases.
> >> >> > > > > > > > Here
> >> >> > > > > > > > > > are
> >> >> > > > > > > > > > >> > the phases:
> >> >> > > > > > > > > > >> > 1. Packaging
> >> >> > > > > > > > > > >> >     - This needs creating of package i.e. rpm
> &
> >> deb
> >> >> > > files.
> >> >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> >> DISCLAMER,
> >> >> > NOTES
> >> >> > > > etc
> >> >> > > > > if
> >> >> > > > > > > any
> >> >> > > > > > > > > > >> needed.
> >> >> > > > > > > > > > >> >     - Any documentation that need to be added
> to
> >> >> > > > > distribution
> >> >> > > > > > of
> >> >> > > > > > > > our
> >> >> > > > > > > > > > >> > software.
> >> >> > > > > > > > > > >> >     - Any license information of dependencies
> >> >> required
> >> >> > > to
> >> >> > > > be
> >> >> > > > > > > added
> >> >> > > > > > > > > to
> >> >> > > > > > > > > > >> > package
> >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> >> >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> >> >> > > > configuration
> >> >> > > > > > > > > > >> >     - Add puppet recipes for automatic
> >> deployment
> >> >> > > > > > > > > > >> > 4. Integration tests
> >> >> > > > > > > > > > >> >     - For verification of deployments.
> >> >> > > > > > > > > > >> > 5. license clearance:
> >> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new
> stuff
> >> is
> >> >> > > > compliant
> >> >> > > > > > > with
> >> >> > > > > > > > > ASF
> >> >> > > > > > > > > > >> > license requirements. If you add code licenses
> >> >> under
> >> >> > > > > different
> >> >> > > > > > > > > > licenses,
> >> >> > > > > > > > > > >> > those would need to be listed in the NOTICE.
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >> > Please share your thoughts on the approach.
> >> >> > > > > > > > > > >> > We'll start corresponding communication on
> >> bigtop
> >> >> > > mailing
> >> >> > > > > list
> >> >> > > > > > > as
> >> >> > > > > > > > > > well.
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >> > We have some specific questions/suggestions
> >> >> related to
> >> >> > > > what
> >> >> > > > > > > should
> >> >> > > > > > > > > be
> >> >> > > > > > > > > > >> the
> >> >> > > > > > > > > > >> > content of the package and what should be the
> >> smoke
> >> >> > > tests,
> >> >> > > > > but
> >> >> > > > > > > in
> >> >> > > > > > > > > the
> >> >> > > > > > > > > > >> > interest of not having too much content here,
> >> we'll
> >> >> > put
> >> >> > > > the
> >> >> > > > > > > > > questions
> >> >> > > > > > > > > > >> as a
> >> >> > > > > > > > > > >> > separate mail in this mailthread.
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >> > Thanks,
> >> >> > > > > > > > > > >> > Chinmay.
> >> >> > > > > > > > > > >> >
> >> >> > > > > > > > > > >>
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>

Re: Apache Apex as component in Apache bigtop

[David Yan <da...@datatorrent.com>]

Apologies for hitting the "Send" button too early.

What is the scope of the name change?  If we simply have a symlink from
apex to dtcli, then it should have no backward compatibility problem at all.

But the proper name change would involve:
- Change the name of DTCli class to ApexCli
- Change the prompt from dt> to apex>
- Change the initial greeting when running the CLI from something like "DT
CLI 3.3.0-incubating ..." to "Apex CLI ...".

The above changes may break existing scripts that rely on the the string
"DT".
But if folks are okay with this potential problem, I am okay with the name
change.

For the man page, we can start with the content of this page:

http://docs.datatorrent.com/dtcli/

Plus the command line options when doing dtcli -h:

usage: DTCli
 -e <arg>    Commands are read from the argument
 -f <arg>    Use the specified prompt at all time
 -h          Print this help
 -kp <arg>   Use the specified kerberos principal
 -kt <arg>   Use the specified kerberos keytab
 -p <arg>    JSONP padding function
 -r          JSON Raw mode
 -v          Verbose mode level 1
 -vv         Verbose mode level 2
 -vvv        Verbose mode level 3
 -vvvv       Verbose mode level 4

David


On Mon, Feb 15, 2016 at 9:56 AM, David Yan <da...@datatorrent.com> wrote:

> What is the scope of the name change?  If we simply have a symlink from
> apex to dtcli, then it should have no backward compatibility problem at all.
>
> But the proper name change would involve:
> - Change the name of DTCli class to ApexCli
> - Change the prompt from dt> to apex>
> - Change the initial greeting when running the CLI from something like "DT
> CLI 3.3.0-incubating ..." to "Apex CLI ...".
>
> The above changes may break existing scripts that rely on the the string
> "DT".
> But if folks are okay with this potential problem, I am okay with the name
> change.
>
> For the man page, we can start with the content of this page:
>
>
>
>
> On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
>> Here is a suggestion I have related to name change:
>>
>> 1. While packaging we change the name of "dtcli" script to "apex". This is
>> done so that in future when we change the name to "apex", users of bigtop
>> apex don't have to transition much.
>> 2. We also keep a symlink named dtcli which points to apex script. This is
>> for backward compatibility.
>> 3. We still keep conf/dt-env.sh for first integration in bigtop. This is
>> in
>> the interest of not changing the content of dtcli or apex script.
>> 4. In apex 3.4.0, we take care of all these naming related changes and
>> update bigtop repository later to remove references to dtcli all together.
>>
>> Please share your thoughts on above approach.
>>
>> Also, please share what could be the man page content for "apex".
>>
>> Thanks,
>> Chinmay.
>>
>>
>>
>> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <ch...@apache.org>
>> wrote:
>>
>> > Yes.. It'll be build from source tar downloaded from one of the apache
>> > mirror: apache.osuosl.org
>> >
>> > The reason for dt-env.sh is it's sourced from dtcli. This leaves us 3
>> > options:
>> > If we should not have any dt-env.sh, dtcli would need a change while
>> > packaging.
>> >
>> > But if we want to use dtcli as it is, then we would need dt-env.sh
>> >
>> >
>> >
>> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <th...@datatorrent.com>
>> > wrote:
>> >
>> >> Looks good overall, though there shouldn't be any dt-env.sh
>> >>
>> >> What will the bigtop package be built from, the source tar?
>> >>
>> >>
>> >>
>> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <chinmay@apache.org
>> >
>> >> wrote:
>> >>
>> >> > Hi Thomas,
>> >> >
>> >> > Thanks for the feedback.
>> >> >
>> >> > First, we're not changing the name anywhere. We'll follow what
>> >> currently is
>> >> > in source tarball for 3.3.0-incubating version.
>> >> >
>> >> > Secondly, I've mentioned a directory structure below which is inline
>> >> with
>> >> > other existing integrations with bigtop.
>> >> >
>> >> > The need for each file is as follows:
>> >> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of
>> >> source
>> >> > code apex 3.3.0-incubating.
>> >> >
>> >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for
>> hadoop
>> >> > binary path and exports an env variable for dtcli to use it.
>> >> >
>> >> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar
>> way to
>> >> > how DT Community Edition finds dependency jars.
>> >> > Please note that this is the first iteration list of jars. I'm
>> trying to
>> >> > narrow this down to only those which are really required.
>> >> > The test that I'm running to check if dtcli runs fine with given
>> >> dependency
>> >> > is to launch+shutdown+kill+status for pi demo.
>> >> >
>> >> > Above is the directory structure which is required for CLI to work.
>> >> > What aniruddha and pradeep has mentioned are some additional files
>> which
>> >> > makes the like of administrator easier. For eg. /etc/skel
>> >> >
>> >> > Please let me know if this is inline with what you're thinking.
>> >> >
>> >> > Thanks,
>> >> > Chinmay.
>> >> >
>> >> >
>> >> > .
>> >> > |-- bin
>> >> > |   `-- dtcli
>> >> > |-- conf
>> >> > |   `-- dt-env.sh
>> >> > `-- lib
>> >> >     |-- ant-1.9.2.jar
>> >> >     |-- ant-launcher-1.9.2.jar
>> >> >     |-- apex-api-3.3.0-incubating.jar
>> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
>> >> >     |-- apex-common-3.3.0-incubating.jar
>> >> >     |-- apex-engine.jar
>> >> >     |-- apex-engine-tests.jar
>> >> >     |-- async-http-client-1.7.20.jar
>> >> >     |-- bval-core-0.5.jar
>> >> >     |-- bval-jsr303-0.5.jar
>> >> >     |-- commons-beanutils-1.8.3.jar
>> >> >     |-- commons-codec-1.10.jar
>> >> >     |-- commons-lang3-3.1.jar
>> >> >     |-- commons-logging-1.1.3.jar
>> >> >     |-- grizzly-http-servlet-2.1.2.jar
>> >> >     |-- hadoop-common-2.2.0-tests.jar
>> >> >     |-- httpclient-4.3.5.jar
>> >> >     |-- httpcore-4.3.2.jar
>> >> >     |-- jackson-core-asl-1.9.2.jar
>> >> >     |-- jackson-mapper-asl-1.9.2.jar
>> >> >     |-- javax.servlet-3.1.jar
>> >> >     |-- javax.servlet-api-3.0.1.jar
>> >> >     |-- jersey-apache-client4-1.9.jar
>> >> >     |-- jersey-client-1.9.jar
>> >> >     |-- jetty-http-8.1.10.v20130312.jar
>> >> >     |-- jetty-io-8.1.10.v20130312.jar
>> >> >     |-- jetty-util-8.1.10.v20130312.jar
>> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
>> >> >     |-- jline-2.11.jar
>> >> >     |-- kryo-2.24.0.jar
>> >> >     |-- mbassador-1.1.9.jar
>> >> >     |-- minlog-1.2.jar
>> >> >     |-- netlet-1.2.0.jar
>> >> >     |-- netty-3.6.6.Final.jar
>> >> >     |-- objenesis-2.1.jar
>> >> >     |-- validation-api-1.1.0.Final.jar
>> >> >     |-- xbean-asm5-shaded-4.3.jar
>> >> >     `-- zip4j-1.3.2.jar
>> >> >
>> >> > 3 directories, 40 files
>> >> >
>> >> >
>> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
>> thomas@datatorrent.com>
>> >> > wrote:
>> >> >
>> >> > > Chinmay,
>> >> > >
>> >> > > Before discussing where to put the files, let's make sure they are
>> >> really
>> >> > > needed for the operation of the CLI. As for names, anything new
>> needs
>> >> to
>> >> > > reflect Apex in the name and should follow common conventions,
>> >> especially
>> >> > > in Bigtop where there are many existing integrations to look at.
>> >> > >
>> >> > > Thanks,
>> >> > > Thomas
>> >> > >
>> >> > >
>> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
>> >> > > chinmay@datatorrent.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Really good points Pradeep and Aniruddha.
>> >> > > >
>> >> > > > 1. I believe we won't need to change the dtcli considering it
>> works
>> >> > with
>> >> > > DT
>> >> > > > Community edition. We can keep the directory structure similar to
>> >> that.
>> >> > > > dt-env.sh has variables which contains information required for
>> >> dtcli
>> >> > to
>> >> > > > launch.
>> >> > > >
>> >> > > > 2. Let me check with bigtop community that whether they
>> facilitate
>> >> the
>> >> > > > installation of rpms/debs before user is created. In either case,
>> >> > current
>> >> > > > dtcli creates a .dt folder in home directory. Also before putting
>> >> > > anything
>> >> > > > in /etc/skel we need to define what are the default contents that
>> >> > should
>> >> > > go
>> >> > > > to ~/.dt/ folder. If there is no defaults, probably we should not
>> >> > > > explicitly add it in /etc/skel.
>> >> > > >
>> >> > > > 3.  /etc/profile.d approach looks nice. This way contents of
>> >> dt-env.sh
>> >> > > are
>> >> > > > present as env variables. But I see a catch there. Adding
>> dt-env.sh
>> >> to
>> >> > > > /etc/profile.d would make all the variables available at runtime
>> all
>> >> > the
>> >> > > > time. I feel a little skeptical about that. Maybe a possible
>> >> collision
>> >> > > can
>> >> > > > occur with other application vars.
>> >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
>> >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
>> >> > > >
>> >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
>> >> > > Jira/repository.
>> >> > > > Packaging related code usually goes there. (That's what all the
>> >> > > components
>> >> > > > in bigtop does).
>> >> > > > Having said that, once a PR is created, I'll be sharing the link
>> of
>> >> PR
>> >> > > > here, so that apex community as well can review it.
>> >> > > >
>> >> > > > Thanks,
>> >> > > > Chinmay.
>> >> > > >
>> >> > > >
>> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
>> >> > > > aniruddha@datatorrent.com> wrote:
>> >> > > >
>> >> > > > > +1 on suggestions and approach.
>> >> > > > > We may need to iron out details about exact paths etc.
>> >> > > > > Which can be done on jira / PR comments.
>> >> > > > > Is that right @dev?
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
>> >> > apache@pradeepdalvi.com>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > Inline comments...
>> >> > > > > >
>> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
>> >> > > > > > chinmay@datatorrent.com>
>> >> > > > > > wrote:
>> >> > > > > >
>> >> > > > > > > @Thomas, Not all the jar present in DT community edition
>> will
>> >> be
>> >> > > > > included
>> >> > > > > > > there.
>> >> > > > > > >
>> >> > > > > > > Here is the list of jars I found from common dependencies
>> >> between
>> >> > > of
>> >> > > > > apex
>> >> > > > > > > and DT Community edition:
>> >> > > > > > > netlet-1.2.0.jar
>> >> > > > > > > kryo-2.24.0.jar
>> >> > > > > > > jackson-core-asl-1.9.2.jar
>> >> > > > > > > jackson-mapper-asl-1.9.2.jar
>> >> > > > > > > async-http-client-1.7.20.jar
>> >> > > > > > > netty-3.6.6.Final.jar
>> >> > > > > > > validation-api-1.1.0.Final.jar
>> >> > > > > > > bval-jsr303-0.5.jar
>> >> > > > > > > bval-core-0.5.jar
>> >> > > > > > > commons-lang3-3.1.jar
>> >> > > > > > > commons-beanutils-1.8.3.jar
>> >> > > > > > > httpclient-4.3.5.jar
>> >> > > > > > > commons-codec-1.10.jar
>> >> > > > > > > zip4j-1.3.2.jar
>> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
>> >> > > > > > > xbean-asm5-shaded-4.3.jar
>> >> > > > > > > jersey-apache-client4-1.9.jar
>> >> > > > > > > jline-2.11.jar
>> >> > > > > > > ant-1.9.2.jar
>> >> > > > > > > ant-launcher-1.9.2.jar
>> >> > > > > > > mbassador-1.1.9.jar
>> >> > > > > > > jackson-jaxrs-1.9.2.jar
>> >> > > > > > > jackson-xc-1.9.2.jar
>> >> > > > > > > hadoop-common-2.2.0-tests.jar
>> >> > > > > > >
>> >> > > > > > > Ofcourse, I'll be running some tests do check that dtcli
>> works
>> >> > > > properly
>> >> > > > > > for
>> >> > > > > > > launch+shutdown+kill of apps with only these libraries
>> >> present in
>> >> > > > > > isolation
>> >> > > > > > > without dependency on local m2.
>> >> > > > > > > I'm believe that there are unwanted jars which are used for
>> >> > compile
>> >> > > > > time
>> >> > > > > > > dependency and not runtime in above list which I can drop
>> to
>> >> keep
>> >> > > > > package
>> >> > > > > > > size minimal.
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > @Anniruddha,
>> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
>> >> required.
>> >> > > > Only
>> >> > > > > > > required values I see in it are classpath. and dtcli is
>> >> capable
>> >> > of
>> >> > > > > > building
>> >> > > > > > > it on the fly.
>> >> > > > > > >
>> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find
>> >> the
>> >> > jar
>> >> > > > > path,
>> >> > > > > > > dt-env.sh and all such conf files. If we need a change in
>> >> dtcli,
>> >> > > then
>> >> > > > > > > community's opinion is required for whether dtcli should
>> >> change
>> >> > in
>> >> > > > our
>> >> > > > > > repo
>> >> > > > > > > or a copy of that with required changes exist in bigtop
>> repo
>> >> > until
>> >> > > we
>> >> > > > > > make
>> >> > > > > > > the dtcli generic enough.
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > > Can we have these file paths set using environment variables
>> >> with
>> >> > > some
>> >> > > > > > default values in dtcli?
>> >> > > > > > Then we can set such params in dt-env.sh.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > > 3. dt-sited.xml -> How is this file different from
>> >> dt-site.xml?
>> >> > Not
>> >> > > > > sure
>> >> > > > > > if
>> >> > > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel
>> >> are
>> >> > > > copied
>> >> > > > > to
>> >> > > > > > > home of new user when useradd program is called. But for
>> >> existing
>> >> > > > users
>> >> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on
>> >> the
>> >> > fly
>> >> > > > if
>> >> > > > > > not
>> >> > > > > > > encountered for the first time. Again correct me if I'm
>> wrong.
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > > Yes, you are right. However this would probably be necessary
>> >> step
>> >> > for
>> >> > > > > > rpm/deb, as they may also get installed during OS install and
>> >> > before
>> >> > > > user
>> >> > > > > > accounts were created.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
>> best
>> >> > > idea.
>> >> > > > > > We'll
>> >> > > > > > > anyway put dtcli in location which is by default present in
>> >> path
>> >> > > i.e.
>> >> > > > > > > /usr/bin.
>> >> > > > > >
>> >> > > > > > For other env variables specific to apex, I'll prefer to use
>> >> > > dt-env.sh
>> >> > > > > and
>> >> > > > > > > source it in dtcli rather than changing bashrc etc...
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
>> >> cleaner
>> >> > way
>> >> > > > to
>> >> > > > > > achieve the same.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > >
>> >> > > > > > Thanks,
>> >> > > > > > > Chinmay.
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
>> >> > > > > > > aniruddha@datatorrent.com> wrote:
>> >> > > > > > >
>> >> > > > > > > > Hi,
>> >> > > > > > > >
>> >> > > > > > > > @Chinmay,
>> >> > > > > > > >
>> >> > > > > > > > We need to consider following:
>> >> > > > > > > > System wide default config files can be located at
>> following
>> >> > > > > locations:
>> >> > > > > > > >
>> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
>> behaviour
>> >> on
>> >> > > how
>> >> > > > it
>> >> > > > > > > finds
>> >> > > > > > > > those locations)
>> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
>> >> behaviour on
>> >> > > how
>> >> > > > > it
>> >> > > > > > > > finds those locations)
>> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users
>> >> that
>> >> > > will
>> >> > > > > be
>> >> > > > > > > > created in system in future.)
>> >> > > > > > > >
>> >> > > > > > > > We may also have to modify bashrc / bashprofile for
>> >> population
>> >> > > any
>> >> > > > > > > > variables that are required.
>> >> > > > > > > >
>> >> > > > > > > > @dev, please put in your comments / suggestion.
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > Thanks,
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > Aniruddha
>> >> > > > > > > >
>> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
>> >> > > > > thomas@datatorrent.com>
>> >> > > > > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > -->
>> >> > > > > > > > >
>> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
>> >> > > > > > > > chinmay@datatorrent.com
>> >> > > > > > > > > >
>> >> > > > > > > > > wrote:
>> >> > > > > > > > >
>> >> > > > > > > > > > Hello Everyone!!
>> >> > > > > > > > > >
>> >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
>> here
>> >> > are
>> >> > > > some
>> >> > > > > > > > > proposals
>> >> > > > > > > > > > about package structure etc..
>> >> > > > > > > > > > Before posting it on bbigtop mailing list, I have
>> some
>> >> > > question
>> >> > > > > for
>> >> > > > > > > > apex
>> >> > > > > > > > > > community.
>> >> > > > > > > > > >
>> >> > > > > > > > > > Proposed Directory structure of apex package for both
>> >> deb &
>> >> > > > rpm:
>> >> > > > > > > > > >
>> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
>> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
>> >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
>> >> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
>> >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
>> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
>> >> > > > > > > > >
>> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
>> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
>> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
>> >> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
>> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
>> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > <version> = 3.3.0-incubating.
>> >> > > > > > > > > > <other dependent jars> = All the 3rd party jars which
>> >> are
>> >> > > > > required
>> >> > > > > > > for
>> >> > > > > > > > > apex
>> >> > > > > > > > > > to run. Usually the dependencies are packaged as
>> part of
>> >> > > > rpm/deb
>> >> > > > > by
>> >> > > > > > > any
>> >> > > > > > > > > > software in bigtop.
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > Can you specify what those jars are in Bigtop context?
>> >> Same
>> >> > as
>> >> > > > > > shipped
>> >> > > > > > > > with
>> >> > > > > > > > > DT community addition under lib/ ?
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
>> >> party
>> >> > > jars
>> >> > > > > > which
>> >> > > > > > > > > needs
>> >> > > > > > > > > > to included while packaging.
>> >> > > > > > > > > >
>> >> > > > > > > > > > Questions related to this:
>> >> > > > > > > > > > 1. Should we call the cli of apex as "apex" instead
>> of
>> >> > > "dtcli"
>> >> > > > in
>> >> > > > > > > > bigtop
>> >> > > > > > > > > > package?
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > I think we should keep the name until we are able to
>> >> change
>> >> > it
>> >> > > in
>> >> > > > > > Apex.
>> >> > > > > > > > > Otherwise this may get confusing.
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > > 2. I see that all softwares in bigtop have man page
>> for
>> >> > their
>> >> > > > > > > > > executables.
>> >> > > > > > > > > > I think we should have it too for dtcli. Is there any
>> >> > > > > documentation
>> >> > > > > > > > > which I
>> >> > > > > > > > > > can convert to man page? or can I use output of
>> "dtcli
>> >> > > --help"?
>> >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
>> >> 3.3.0 OR
>> >> > > > > > > > > > 3.3.0-incubating?
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > Has to be -incubating
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
>> >> 2.7.1
>> >> > > > > version
>> >> > > > > > of
>> >> > > > > > > > > > bigtop hadoop? Any problems that we see with this
>> >> > dependency?
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > Should work without changed. I thought we certified
>> >> against
>> >> > > > 2.7.0?
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > > 5. Following is the apache mirror from which bigtop
>> will
>> >> > pick
>> >> > > > the
>> >> > > > > > > apex
>> >> > > > > > > > > > source for compilation and packaging. Please correct
>> if
>> >> > > > > incorrect:
>> >> > > > > > > > > >
>> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>> >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
>> all
>> >> the
>> >> > > > > > softwares
>> >> > > > > > > in
>> >> > > > > > > > > > bigtop.
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > Please share your opinion.
>> >> > > > > > > > > >
>> >> > > > > > > > > > Thanks,
>> >> > > > > > > > > > Chinmay.
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
>> >> > > > > > > > > > chinmay@datatorrent.com>
>> >> > > > > > > > > > wrote:
>> >> > > > > > > > > >
>> >> > > > > > > > > > > Yes.. Starting to work on the packaging.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > I've already started discussion on bigtop dev
>> mailing
>> >> > list
>> >> > > > for
>> >> > > > > > > > > > > integration. Also created a Jira for the same. For
>> >> this
>> >> > > > > > communities
>> >> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
>> >> > > > > > > thomas@datatorrent.com
>> >> > > > > > > > >
>> >> > > > > > > > > > > wrote:
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >> Chinmay,
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> I don't see anything under prerequisites that
>> would
>> >> be a
>> >> > > > > > problem.
>> >> > > > > > > We
>> >> > > > > > > > > > >> looked
>> >> > > > > > > > > > >> at the ASF licencing compatibility as part of
>> >> becoming
>> >> > an
>> >> > > > > > > incubator
>> >> > > > > > > > > > >> project.
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> Please focus on the packaging during the next
>> weeks.
>> >> > Since
>> >> > > > the
>> >> > > > > > > work
>> >> > > > > > > > > will
>> >> > > > > > > > > > >> be
>> >> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
>> should
>> >> > also
>> >> > > be
>> >> > > > > > > there.
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> Would be good to have the packaging in place by
>> end
>> >> Feb.
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> Thanks,
>> >> > > > > > > > > > >> Thomas
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
>> Kolhatkar <
>> >> > > > > > > > > > >> chinmay@datatorrent.com>
>> >> > > > > > > > > > >> wrote:
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> > Hi All,
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > We're planning a work on adding Apache Apex as a
>> >> > > component
>> >> > > > > to
>> >> > > > > > > > Apache
>> >> > > > > > > > > > >> > Bigtop.
>> >> > > > > > > > > > >> > Bigtop is the packaging system for the Apache
>> big
>> >> data
>> >> > > > > > > ecosystem.
>> >> > > > > > > > > > >> Several
>> >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for
>> the
>> >> > same:
>> >> > > > > > > > > > >> >
>> https://issues.apache.org/jira/browse/APEXCORE-331
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > Proposed plan of execution is as follows:
>> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
>> >> > > > > > > > > > >> > Apache bigtop has some hard and soft expectation
>> >> for
>> >> > new
>> >> > > > > > > > components
>> >> > > > > > > > > to
>> >> > > > > > > > > > >> get
>> >> > > > > > > > > > >> > integrated into Bigtop.
>> >> > > > > > > > > > >> > Here is the list of it:
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >>
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > Most of them seems to be standard ASF based
>> >> > > requirements,
>> >> > > > > but
>> >> > > > > > > few
>> >> > > > > > > > > need
>> >> > > > > > > > > > >> to
>> >> > > > > > > > > > >> > be checked for:
>> >> > > > > > > > > > >> > 1. Software projects are expected to be Licensed
>> >> under
>> >> > > > > Apache
>> >> > > > > > > > > License,
>> >> > > > > > > > > > >> > Version 2.0 (and their dependencies are expected
>> >> to be
>> >> > > > > > > compatible
>> >> > > > > > > > > with
>> >> > > > > > > > > > >> this
>> >> > > > > > > > > > >> > license)
>> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
>> >> > > > > dependencies
>> >> > > > > > of
>> >> > > > > > > > > Apex
>> >> > > > > > > > > > >> are
>> >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
>> >> > > > > verification
>> >> > > > > > > > check.
>> >> > > > > > > > > > >> > 2. Software projects are expected to be
>> compatible
>> >> > with
>> >> > > > all
>> >> > > > > of
>> >> > > > > > > the
>> >> > > > > > > > > > >> > supported platforms that Bigtop distribution is
>> >> > > targeting
>> >> > > > > > > > > > >> >     - This needs verifying whether our software
>> >> runs
>> >> > > fine
>> >> > > > in
>> >> > > > > > > > > centos-6
>> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
>> >> > opensuse-13.2.
>> >> > > > > > > > > > >> > 3. What smoke tests that should be added for
>> >> > deployment.
>> >> > > > > > > > > > >> > 4. Identifying the test artifacts which goes
>> beyond
>> >> > > smoke
>> >> > > > > test
>> >> > > > > > > > > > >> >     - These are basically the integration tests
>> for
>> >> > > > > > verification
>> >> > > > > > > > > after
>> >> > > > > > > > > > >> the
>> >> > > > > > > > > > >> > deployment. This is a soft requirement, but aim
>> is
>> >> to
>> >> > > > > achieve
>> >> > > > > > > this
>> >> > > > > > > > > as
>> >> > > > > > > > > > >> well
>> >> > > > > > > > > > >> > or at least have explanation why not to include.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > If there are any from the link which explicitly
>> >> needs
>> >> > to
>> >> > > > be
>> >> > > > > > > > checked
>> >> > > > > > > > > > >> other
>> >> > > > > > > > > > >> > than above 4, please let us know.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
>> >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
>> >> learnt
>> >> > > that
>> >> > > > > the
>> >> > > > > > > > > bigtop
>> >> > > > > > > > > > >> > community want to see the addition of new
>> >> components
>> >> > in
>> >> > > > > > phases.
>> >> > > > > > > > Here
>> >> > > > > > > > > > are
>> >> > > > > > > > > > >> > the phases:
>> >> > > > > > > > > > >> > 1. Packaging
>> >> > > > > > > > > > >> >     - This needs creating of package i.e. rpm &
>> deb
>> >> > > files.
>> >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
>> DISCLAMER,
>> >> > NOTES
>> >> > > > etc
>> >> > > > > if
>> >> > > > > > > any
>> >> > > > > > > > > > >> needed.
>> >> > > > > > > > > > >> >     - Any documentation that need to be added to
>> >> > > > > distribution
>> >> > > > > > of
>> >> > > > > > > > our
>> >> > > > > > > > > > >> > software.
>> >> > > > > > > > > > >> >     - Any license information of dependencies
>> >> required
>> >> > > to
>> >> > > > be
>> >> > > > > > > added
>> >> > > > > > > > > to
>> >> > > > > > > > > > >> > package
>> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
>> >> > > > > > > > > > >> >     - Adding smoke test for packaging.
>> >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
>> >> > > > configuration
>> >> > > > > > > > > > >> >     - Add puppet recipes for automatic
>> deployment
>> >> > > > > > > > > > >> > 4. Integration tests
>> >> > > > > > > > > > >> >     - For verification of deployments.
>> >> > > > > > > > > > >> > 5. license clearance:
>> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff
>> is
>> >> > > > compliant
>> >> > > > > > > with
>> >> > > > > > > > > ASF
>> >> > > > > > > > > > >> > license requirements. If you add code licenses
>> >> under
>> >> > > > > different
>> >> > > > > > > > > > licenses,
>> >> > > > > > > > > > >> > those would need to be listed in the NOTICE.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > Please share your thoughts on the approach.
>> >> > > > > > > > > > >> > We'll start corresponding communication on
>> bigtop
>> >> > > mailing
>> >> > > > > list
>> >> > > > > > > as
>> >> > > > > > > > > > well.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > We have some specific questions/suggestions
>> >> related to
>> >> > > > what
>> >> > > > > > > should
>> >> > > > > > > > > be
>> >> > > > > > > > > > >> the
>> >> > > > > > > > > > >> > content of the package and what should be the
>> smoke
>> >> > > tests,
>> >> > > > > but
>> >> > > > > > > in
>> >> > > > > > > > > the
>> >> > > > > > > > > > >> > interest of not having too much content here,
>> we'll
>> >> > put
>> >> > > > the
>> >> > > > > > > > > questions
>> >> > > > > > > > > > >> as a
>> >> > > > > > > > > > >> > separate mail in this mailthread.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > Thanks,
>> >> > > > > > > > > > >> > Chinmay.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>

Re: Apache Apex as component in Apache bigtop

[David Yan <da...@datatorrent.com>]

What is the scope of the name change?  If we simply have a symlink from
apex to dtcli, then it should have no backward compatibility problem at all.

But the proper name change would involve:
- Change the name of DTCli class to ApexCli
- Change the prompt from dt> to apex>
- Change the initial greeting when running the CLI from something like "DT
CLI 3.3.0-incubating ..." to "Apex CLI ...".

The above changes may break existing scripts that rely on the the string
"DT".
But if folks are okay with this potential problem, I am okay with the name
change.

For the man page, we can start with the content of this page:




On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Here is a suggestion I have related to name change:
>
> 1. While packaging we change the name of "dtcli" script to "apex". This is
> done so that in future when we change the name to "apex", users of bigtop
> apex don't have to transition much.
> 2. We also keep a symlink named dtcli which points to apex script. This is
> for backward compatibility.
> 3. We still keep conf/dt-env.sh for first integration in bigtop. This is in
> the interest of not changing the content of dtcli or apex script.
> 4. In apex 3.4.0, we take care of all these naming related changes and
> update bigtop repository later to remove references to dtcli all together.
>
> Please share your thoughts on above approach.
>
> Also, please share what could be the man page content for "apex".
>
> Thanks,
> Chinmay.
>
>
>
> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Yes.. It'll be build from source tar downloaded from one of the apache
> > mirror: apache.osuosl.org
> >
> > The reason for dt-env.sh is it's sourced from dtcli. This leaves us 3
> > options:
> > If we should not have any dt-env.sh, dtcli would need a change while
> > packaging.
> >
> > But if we want to use dtcli as it is, then we would need dt-env.sh
> >
> >
> >
> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> >> Looks good overall, though there shouldn't be any dt-env.sh
> >>
> >> What will the bigtop package be built from, the source tar?
> >>
> >>
> >>
> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <ch...@apache.org>
> >> wrote:
> >>
> >> > Hi Thomas,
> >> >
> >> > Thanks for the feedback.
> >> >
> >> > First, we're not changing the name anywhere. We'll follow what
> >> currently is
> >> > in source tarball for 3.3.0-incubating version.
> >> >
> >> > Secondly, I've mentioned a directory structure below which is inline
> >> with
> >> > other existing integrations with bigtop.
> >> >
> >> > The need for each file is as follows:
> >> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of
> >> source
> >> > code apex 3.3.0-incubating.
> >> >
> >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for
> hadoop
> >> > binary path and exports an env variable for dtcli to use it.
> >> >
> >> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar way
> to
> >> > how DT Community Edition finds dependency jars.
> >> > Please note that this is the first iteration list of jars. I'm trying
> to
> >> > narrow this down to only those which are really required.
> >> > The test that I'm running to check if dtcli runs fine with given
> >> dependency
> >> > is to launch+shutdown+kill+status for pi demo.
> >> >
> >> > Above is the directory structure which is required for CLI to work.
> >> > What aniruddha and pradeep has mentioned are some additional files
> which
> >> > makes the like of administrator easier. For eg. /etc/skel
> >> >
> >> > Please let me know if this is inline with what you're thinking.
> >> >
> >> > Thanks,
> >> > Chinmay.
> >> >
> >> >
> >> > .
> >> > |-- bin
> >> > |   `-- dtcli
> >> > |-- conf
> >> > |   `-- dt-env.sh
> >> > `-- lib
> >> >     |-- ant-1.9.2.jar
> >> >     |-- ant-launcher-1.9.2.jar
> >> >     |-- apex-api-3.3.0-incubating.jar
> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> >> >     |-- apex-common-3.3.0-incubating.jar
> >> >     |-- apex-engine.jar
> >> >     |-- apex-engine-tests.jar
> >> >     |-- async-http-client-1.7.20.jar
> >> >     |-- bval-core-0.5.jar
> >> >     |-- bval-jsr303-0.5.jar
> >> >     |-- commons-beanutils-1.8.3.jar
> >> >     |-- commons-codec-1.10.jar
> >> >     |-- commons-lang3-3.1.jar
> >> >     |-- commons-logging-1.1.3.jar
> >> >     |-- grizzly-http-servlet-2.1.2.jar
> >> >     |-- hadoop-common-2.2.0-tests.jar
> >> >     |-- httpclient-4.3.5.jar
> >> >     |-- httpcore-4.3.2.jar
> >> >     |-- jackson-core-asl-1.9.2.jar
> >> >     |-- jackson-mapper-asl-1.9.2.jar
> >> >     |-- javax.servlet-3.1.jar
> >> >     |-- javax.servlet-api-3.0.1.jar
> >> >     |-- jersey-apache-client4-1.9.jar
> >> >     |-- jersey-client-1.9.jar
> >> >     |-- jetty-http-8.1.10.v20130312.jar
> >> >     |-- jetty-io-8.1.10.v20130312.jar
> >> >     |-- jetty-util-8.1.10.v20130312.jar
> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> >> >     |-- jline-2.11.jar
> >> >     |-- kryo-2.24.0.jar
> >> >     |-- mbassador-1.1.9.jar
> >> >     |-- minlog-1.2.jar
> >> >     |-- netlet-1.2.0.jar
> >> >     |-- netty-3.6.6.Final.jar
> >> >     |-- objenesis-2.1.jar
> >> >     |-- validation-api-1.1.0.Final.jar
> >> >     |-- xbean-asm5-shaded-4.3.jar
> >> >     `-- zip4j-1.3.2.jar
> >> >
> >> > 3 directories, 40 files
> >> >
> >> >
> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <thomas@datatorrent.com
> >
> >> > wrote:
> >> >
> >> > > Chinmay,
> >> > >
> >> > > Before discussing where to put the files, let's make sure they are
> >> really
> >> > > needed for the operation of the CLI. As for names, anything new
> needs
> >> to
> >> > > reflect Apex in the name and should follow common conventions,
> >> especially
> >> > > in Bigtop where there are many existing integrations to look at.
> >> > >
> >> > > Thanks,
> >> > > Thomas
> >> > >
> >> > >
> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> >> > > chinmay@datatorrent.com>
> >> > > wrote:
> >> > >
> >> > > > Really good points Pradeep and Aniruddha.
> >> > > >
> >> > > > 1. I believe we won't need to change the dtcli considering it
> works
> >> > with
> >> > > DT
> >> > > > Community edition. We can keep the directory structure similar to
> >> that.
> >> > > > dt-env.sh has variables which contains information required for
> >> dtcli
> >> > to
> >> > > > launch.
> >> > > >
> >> > > > 2. Let me check with bigtop community that whether they facilitate
> >> the
> >> > > > installation of rpms/debs before user is created. In either case,
> >> > current
> >> > > > dtcli creates a .dt folder in home directory. Also before putting
> >> > > anything
> >> > > > in /etc/skel we need to define what are the default contents that
> >> > should
> >> > > go
> >> > > > to ~/.dt/ folder. If there is no defaults, probably we should not
> >> > > > explicitly add it in /etc/skel.
> >> > > >
> >> > > > 3.  /etc/profile.d approach looks nice. This way contents of
> >> dt-env.sh
> >> > > are
> >> > > > present as env variables. But I see a catch there. Adding
> dt-env.sh
> >> to
> >> > > > /etc/profile.d would make all the variables available at runtime
> all
> >> > the
> >> > > > time. I feel a little skeptical about that. Maybe a possible
> >> collision
> >> > > can
> >> > > > occur with other application vars.
> >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> >> > > >
> >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> >> > > Jira/repository.
> >> > > > Packaging related code usually goes there. (That's what all the
> >> > > components
> >> > > > in bigtop does).
> >> > > > Having said that, once a PR is created, I'll be sharing the link
> of
> >> PR
> >> > > > here, so that apex community as well can review it.
> >> > > >
> >> > > > Thanks,
> >> > > > Chinmay.
> >> > > >
> >> > > >
> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> >> > > > aniruddha@datatorrent.com> wrote:
> >> > > >
> >> > > > > +1 on suggestions and approach.
> >> > > > > We may need to iron out details about exact paths etc.
> >> > > > > Which can be done on jira / PR comments.
> >> > > > > Is that right @dev?
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> >> > apache@pradeepdalvi.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Inline comments...
> >> > > > > >
> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> >> > > > > > chinmay@datatorrent.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > @Thomas, Not all the jar present in DT community edition
> will
> >> be
> >> > > > > included
> >> > > > > > > there.
> >> > > > > > >
> >> > > > > > > Here is the list of jars I found from common dependencies
> >> between
> >> > > of
> >> > > > > apex
> >> > > > > > > and DT Community edition:
> >> > > > > > > netlet-1.2.0.jar
> >> > > > > > > kryo-2.24.0.jar
> >> > > > > > > jackson-core-asl-1.9.2.jar
> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> >> > > > > > > async-http-client-1.7.20.jar
> >> > > > > > > netty-3.6.6.Final.jar
> >> > > > > > > validation-api-1.1.0.Final.jar
> >> > > > > > > bval-jsr303-0.5.jar
> >> > > > > > > bval-core-0.5.jar
> >> > > > > > > commons-lang3-3.1.jar
> >> > > > > > > commons-beanutils-1.8.3.jar
> >> > > > > > > httpclient-4.3.5.jar
> >> > > > > > > commons-codec-1.10.jar
> >> > > > > > > zip4j-1.3.2.jar
> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> >> > > > > > > xbean-asm5-shaded-4.3.jar
> >> > > > > > > jersey-apache-client4-1.9.jar
> >> > > > > > > jline-2.11.jar
> >> > > > > > > ant-1.9.2.jar
> >> > > > > > > ant-launcher-1.9.2.jar
> >> > > > > > > mbassador-1.1.9.jar
> >> > > > > > > jackson-jaxrs-1.9.2.jar
> >> > > > > > > jackson-xc-1.9.2.jar
> >> > > > > > > hadoop-common-2.2.0-tests.jar
> >> > > > > > >
> >> > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> works
> >> > > > properly
> >> > > > > > for
> >> > > > > > > launch+shutdown+kill of apps with only these libraries
> >> present in
> >> > > > > > isolation
> >> > > > > > > without dependency on local m2.
> >> > > > > > > I'm believe that there are unwanted jars which are used for
> >> > compile
> >> > > > > time
> >> > > > > > > dependency and not runtime in above list which I can drop to
> >> keep
> >> > > > > package
> >> > > > > > > size minimal.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > @Anniruddha,
> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> >> required.
> >> > > > Only
> >> > > > > > > required values I see in it are classpath. and dtcli is
> >> capable
> >> > of
> >> > > > > > building
> >> > > > > > > it on the fly.
> >> > > > > > >
> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find
> >> the
> >> > jar
> >> > > > > path,
> >> > > > > > > dt-env.sh and all such conf files. If we need a change in
> >> dtcli,
> >> > > then
> >> > > > > > > community's opinion is required for whether dtcli should
> >> change
> >> > in
> >> > > > our
> >> > > > > > repo
> >> > > > > > > or a copy of that with required changes exist in bigtop repo
> >> > until
> >> > > we
> >> > > > > > make
> >> > > > > > > the dtcli generic enough.
> >> > > > > > >
> >> > > > > >
> >> > > > > > Can we have these file paths set using environment variables
> >> with
> >> > > some
> >> > > > > > default values in dtcli?
> >> > > > > > Then we can set such params in dt-env.sh.
> >> > > > > >
> >> > > > > >
> >> > > > > > > 3. dt-sited.xml -> How is this file different from
> >> dt-site.xml?
> >> > Not
> >> > > > > sure
> >> > > > > > if
> >> > > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel
> >> are
> >> > > > copied
> >> > > > > to
> >> > > > > > > home of new user when useradd program is called. But for
> >> existing
> >> > > > users
> >> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on
> >> the
> >> > fly
> >> > > > if
> >> > > > > > not
> >> > > > > > > encountered for the first time. Again correct me if I'm
> wrong.
> >> > > > > > >
> >> > > > > >
> >> > > > > > Yes, you are right. However this would probably be necessary
> >> step
> >> > for
> >> > > > > > rpm/deb, as they may also get installed during OS install and
> >> > before
> >> > > > user
> >> > > > > > accounts were created.
> >> > > > > >
> >> > > > > >
> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
> best
> >> > > idea.
> >> > > > > > We'll
> >> > > > > > > anyway put dtcli in location which is by default present in
> >> path
> >> > > i.e.
> >> > > > > > > /usr/bin.
> >> > > > > >
> >> > > > > > For other env variables specific to apex, I'll prefer to use
> >> > > dt-env.sh
> >> > > > > and
> >> > > > > > > source it in dtcli rather than changing bashrc etc...
> >> > > > > >
> >> > > > > >
> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> >> cleaner
> >> > way
> >> > > > to
> >> > > > > > achieve the same.
> >> > > > > >
> >> > > > > >
> >> > > > > > >
> >> > > > > > Thanks,
> >> > > > > > > Chinmay.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> >> > > > > > > aniruddha@datatorrent.com> wrote:
> >> > > > > > >
> >> > > > > > > > Hi,
> >> > > > > > > >
> >> > > > > > > > @Chinmay,
> >> > > > > > > >
> >> > > > > > > > We need to consider following:
> >> > > > > > > > System wide default config files can be located at
> following
> >> > > > > locations:
> >> > > > > > > >
> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> behaviour
> >> on
> >> > > how
> >> > > > it
> >> > > > > > > finds
> >> > > > > > > > those locations)
> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> >> behaviour on
> >> > > how
> >> > > > > it
> >> > > > > > > > finds those locations)
> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users
> >> that
> >> > > will
> >> > > > > be
> >> > > > > > > > created in system in future.)
> >> > > > > > > >
> >> > > > > > > > We may also have to modify bashrc / bashprofile for
> >> population
> >> > > any
> >> > > > > > > > variables that are required.
> >> > > > > > > >
> >> > > > > > > > @dev, please put in your comments / suggestion.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Aniruddha
> >> > > > > > > >
> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> >> > > > > thomas@datatorrent.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > -->
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> >> > > > > > > > chinmay@datatorrent.com
> >> > > > > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hello Everyone!!
> >> > > > > > > > > >
> >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
> here
> >> > are
> >> > > > some
> >> > > > > > > > > proposals
> >> > > > > > > > > > about package structure etc..
> >> > > > > > > > > > Before posting it on bbigtop mailing list, I have some
> >> > > question
> >> > > > > for
> >> > > > > > > > apex
> >> > > > > > > > > > community.
> >> > > > > > > > > >
> >> > > > > > > > > > Proposed Directory structure of apex package for both
> >> deb &
> >> > > > rpm:
> >> > > > > > > > > >
> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> >> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> >> > > > > > > > >
> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> >> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > <version> = 3.3.0-incubating.
> >> > > > > > > > > > <other dependent jars> = All the 3rd party jars which
> >> are
> >> > > > > required
> >> > > > > > > for
> >> > > > > > > > > apex
> >> > > > > > > > > > to run. Usually the dependencies are packaged as part
> of
> >> > > > rpm/deb
> >> > > > > by
> >> > > > > > > any
> >> > > > > > > > > > software in bigtop.
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Can you specify what those jars are in Bigtop context?
> >> Same
> >> > as
> >> > > > > > shipped
> >> > > > > > > > with
> >> > > > > > > > > DT community addition under lib/ ?
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
> >> party
> >> > > jars
> >> > > > > > which
> >> > > > > > > > > needs
> >> > > > > > > > > > to included while packaging.
> >> > > > > > > > > >
> >> > > > > > > > > > Questions related to this:
> >> > > > > > > > > > 1. Should we call the cli of apex as "apex" instead of
> >> > > "dtcli"
> >> > > > in
> >> > > > > > > > bigtop
> >> > > > > > > > > > package?
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > I think we should keep the name until we are able to
> >> change
> >> > it
> >> > > in
> >> > > > > > Apex.
> >> > > > > > > > > Otherwise this may get confusing.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > > 2. I see that all softwares in bigtop have man page
> for
> >> > their
> >> > > > > > > > > executables.
> >> > > > > > > > > > I think we should have it too for dtcli. Is there any
> >> > > > > documentation
> >> > > > > > > > > which I
> >> > > > > > > > > > can convert to man page? or can I use output of "dtcli
> >> > > --help"?
> >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> >> 3.3.0 OR
> >> > > > > > > > > > 3.3.0-incubating?
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Has to be -incubating
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
> >> 2.7.1
> >> > > > > version
> >> > > > > > of
> >> > > > > > > > > > bigtop hadoop? Any problems that we see with this
> >> > dependency?
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Should work without changed. I thought we certified
> >> against
> >> > > > 2.7.0?
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > > 5. Following is the apache mirror from which bigtop
> will
> >> > pick
> >> > > > the
> >> > > > > > > apex
> >> > > > > > > > > > source for compilation and packaging. Please correct
> if
> >> > > > > incorrect:
> >> > > > > > > > > >
> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by all
> >> the
> >> > > > > > softwares
> >> > > > > > > in
> >> > > > > > > > > > bigtop.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Please share your opinion.
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Chinmay.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> >> > > > > > > > > > chinmay@datatorrent.com>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> >> > > > > > > > > > >
> >> > > > > > > > > > > I've already started discussion on bigtop dev
> mailing
> >> > list
> >> > > > for
> >> > > > > > > > > > > integration. Also created a Jira for the same. For
> >> this
> >> > > > > > communities
> >> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> >> > > > > > > thomas@datatorrent.com
> >> > > > > > > > >
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > >> Chinmay,
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> I don't see anything under prerequisites that would
> >> be a
> >> > > > > > problem.
> >> > > > > > > We
> >> > > > > > > > > > >> looked
> >> > > > > > > > > > >> at the ASF licencing compatibility as part of
> >> becoming
> >> > an
> >> > > > > > > incubator
> >> > > > > > > > > > >> project.
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Please focus on the packaging during the next
> weeks.
> >> > Since
> >> > > > the
> >> > > > > > > work
> >> > > > > > > > > will
> >> > > > > > > > > > >> be
> >> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> should
> >> > also
> >> > > be
> >> > > > > > > there.
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Would be good to have the packaging in place by end
> >> Feb.
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Thanks,
> >> > > > > > > > > > >> Thomas
> >> > > > > > > > > > >>
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar
> <
> >> > > > > > > > > > >> chinmay@datatorrent.com>
> >> > > > > > > > > > >> wrote:
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> > Hi All,
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > We're planning a work on adding Apache Apex as a
> >> > > component
> >> > > > > to
> >> > > > > > > > Apache
> >> > > > > > > > > > >> > Bigtop.
> >> > > > > > > > > > >> > Bigtop is the packaging system for the Apache big
> >> data
> >> > > > > > > ecosystem.
> >> > > > > > > > > > >> Several
> >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for
> the
> >> > same:
> >> > > > > > > > > > >> >
> https://issues.apache.org/jira/browse/APEXCORE-331
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Proposed plan of execution is as follows:
> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> >> > > > > > > > > > >> > Apache bigtop has some hard and soft expectation
> >> for
> >> > new
> >> > > > > > > > components
> >> > > > > > > > > to
> >> > > > > > > > > > >> get
> >> > > > > > > > > > >> > integrated into Bigtop.
> >> > > > > > > > > > >> > Here is the list of it:
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Most of them seems to be standard ASF based
> >> > > requirements,
> >> > > > > but
> >> > > > > > > few
> >> > > > > > > > > need
> >> > > > > > > > > > >> to
> >> > > > > > > > > > >> > be checked for:
> >> > > > > > > > > > >> > 1. Software projects are expected to be Licensed
> >> under
> >> > > > > Apache
> >> > > > > > > > > License,
> >> > > > > > > > > > >> > Version 2.0 (and their dependencies are expected
> >> to be
> >> > > > > > > compatible
> >> > > > > > > > > with
> >> > > > > > > > > > >> this
> >> > > > > > > > > > >> > license)
> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> >> > > > > dependencies
> >> > > > > > of
> >> > > > > > > > > Apex
> >> > > > > > > > > > >> are
> >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> >> > > > > verification
> >> > > > > > > > check.
> >> > > > > > > > > > >> > 2. Software projects are expected to be
> compatible
> >> > with
> >> > > > all
> >> > > > > of
> >> > > > > > > the
> >> > > > > > > > > > >> > supported platforms that Bigtop distribution is
> >> > > targeting
> >> > > > > > > > > > >> >     - This needs verifying whether our software
> >> runs
> >> > > fine
> >> > > > in
> >> > > > > > > > > centos-6
> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> >> > opensuse-13.2.
> >> > > > > > > > > > >> > 3. What smoke tests that should be added for
> >> > deployment.
> >> > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> beyond
> >> > > smoke
> >> > > > > test
> >> > > > > > > > > > >> >     - These are basically the integration tests
> for
> >> > > > > > verification
> >> > > > > > > > > after
> >> > > > > > > > > > >> the
> >> > > > > > > > > > >> > deployment. This is a soft requirement, but aim
> is
> >> to
> >> > > > > achieve
> >> > > > > > > this
> >> > > > > > > > > as
> >> > > > > > > > > > >> well
> >> > > > > > > > > > >> > or at least have explanation why not to include.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > If there are any from the link which explicitly
> >> needs
> >> > to
> >> > > > be
> >> > > > > > > > checked
> >> > > > > > > > > > >> other
> >> > > > > > > > > > >> > than above 4, please let us know.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
> >> learnt
> >> > > that
> >> > > > > the
> >> > > > > > > > > bigtop
> >> > > > > > > > > > >> > community want to see the addition of new
> >> components
> >> > in
> >> > > > > > phases.
> >> > > > > > > > Here
> >> > > > > > > > > > are
> >> > > > > > > > > > >> > the phases:
> >> > > > > > > > > > >> > 1. Packaging
> >> > > > > > > > > > >> >     - This needs creating of package i.e. rpm &
> deb
> >> > > files.
> >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER,
> >> > NOTES
> >> > > > etc
> >> > > > > if
> >> > > > > > > any
> >> > > > > > > > > > >> needed.
> >> > > > > > > > > > >> >     - Any documentation that need to be added to
> >> > > > > distribution
> >> > > > > > of
> >> > > > > > > > our
> >> > > > > > > > > > >> > software.
> >> > > > > > > > > > >> >     - Any license information of dependencies
> >> required
> >> > > to
> >> > > > be
> >> > > > > > > added
> >> > > > > > > > > to
> >> > > > > > > > > > >> > package
> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> >> > > > configuration
> >> > > > > > > > > > >> >     - Add puppet recipes for automatic deployment
> >> > > > > > > > > > >> > 4. Integration tests
> >> > > > > > > > > > >> >     - For verification of deployments.
> >> > > > > > > > > > >> > 5. license clearance:
> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff
> is
> >> > > > compliant
> >> > > > > > > with
> >> > > > > > > > > ASF
> >> > > > > > > > > > >> > license requirements. If you add code licenses
> >> under
> >> > > > > different
> >> > > > > > > > > > licenses,
> >> > > > > > > > > > >> > those would need to be listed in the NOTICE.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Please share your thoughts on the approach.
> >> > > > > > > > > > >> > We'll start corresponding communication on bigtop
> >> > > mailing
> >> > > > > list
> >> > > > > > > as
> >> > > > > > > > > > well.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > We have some specific questions/suggestions
> >> related to
> >> > > > what
> >> > > > > > > should
> >> > > > > > > > > be
> >> > > > > > > > > > >> the
> >> > > > > > > > > > >> > content of the package and what should be the
> smoke
> >> > > tests,
> >> > > > > but
> >> > > > > > > in
> >> > > > > > > > > the
> >> > > > > > > > > > >> > interest of not having too much content here,
> we'll
> >> > put
> >> > > > the
> >> > > > > > > > > questions
> >> > > > > > > > > > >> as a
> >> > > > > > > > > > >> > separate mail in this mailthread.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Thanks,
> >> > > > > > > > > > >> > Chinmay.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Here is a suggestion I have related to name change:

1. While packaging we change the name of "dtcli" script to "apex". This is
done so that in future when we change the name to "apex", users of bigtop
apex don't have to transition much.
2. We also keep a symlink named dtcli which points to apex script. This is
for backward compatibility.
3. We still keep conf/dt-env.sh for first integration in bigtop. This is in
the interest of not changing the content of dtcli or apex script.
4. In apex 3.4.0, we take care of all these naming related changes and
update bigtop repository later to remove references to dtcli all together.

Please share your thoughts on above approach.

Also, please share what could be the man page content for "apex".

Thanks,
Chinmay.



On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Yes.. It'll be build from source tar downloaded from one of the apache
> mirror: apache.osuosl.org
>
> The reason for dt-env.sh is it's sourced from dtcli. This leaves us 3
> options:
> If we should not have any dt-env.sh, dtcli would need a change while
> packaging.
>
> But if we want to use dtcli as it is, then we would need dt-env.sh
>
>
>
> On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
>> Looks good overall, though there shouldn't be any dt-env.sh
>>
>> What will the bigtop package be built from, the source tar?
>>
>>
>>
>> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <ch...@apache.org>
>> wrote:
>>
>> > Hi Thomas,
>> >
>> > Thanks for the feedback.
>> >
>> > First, we're not changing the name anywhere. We'll follow what
>> currently is
>> > in source tarball for 3.3.0-incubating version.
>> >
>> > Secondly, I've mentioned a directory structure below which is inline
>> with
>> > other existing integrations with bigtop.
>> >
>> > The need for each file is as follows:
>> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of
>> source
>> > code apex 3.3.0-incubating.
>> >
>> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for hadoop
>> > binary path and exports an env variable for dtcli to use it.
>> >
>> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar way to
>> > how DT Community Edition finds dependency jars.
>> > Please note that this is the first iteration list of jars. I'm trying to
>> > narrow this down to only those which are really required.
>> > The test that I'm running to check if dtcli runs fine with given
>> dependency
>> > is to launch+shutdown+kill+status for pi demo.
>> >
>> > Above is the directory structure which is required for CLI to work.
>> > What aniruddha and pradeep has mentioned are some additional files which
>> > makes the like of administrator easier. For eg. /etc/skel
>> >
>> > Please let me know if this is inline with what you're thinking.
>> >
>> > Thanks,
>> > Chinmay.
>> >
>> >
>> > .
>> > |-- bin
>> > |   `-- dtcli
>> > |-- conf
>> > |   `-- dt-env.sh
>> > `-- lib
>> >     |-- ant-1.9.2.jar
>> >     |-- ant-launcher-1.9.2.jar
>> >     |-- apex-api-3.3.0-incubating.jar
>> >     |-- apex-bufferserver-3.3.0-incubating.jar
>> >     |-- apex-common-3.3.0-incubating.jar
>> >     |-- apex-engine.jar
>> >     |-- apex-engine-tests.jar
>> >     |-- async-http-client-1.7.20.jar
>> >     |-- bval-core-0.5.jar
>> >     |-- bval-jsr303-0.5.jar
>> >     |-- commons-beanutils-1.8.3.jar
>> >     |-- commons-codec-1.10.jar
>> >     |-- commons-lang3-3.1.jar
>> >     |-- commons-logging-1.1.3.jar
>> >     |-- grizzly-http-servlet-2.1.2.jar
>> >     |-- hadoop-common-2.2.0-tests.jar
>> >     |-- httpclient-4.3.5.jar
>> >     |-- httpcore-4.3.2.jar
>> >     |-- jackson-core-asl-1.9.2.jar
>> >     |-- jackson-mapper-asl-1.9.2.jar
>> >     |-- javax.servlet-3.1.jar
>> >     |-- javax.servlet-api-3.0.1.jar
>> >     |-- jersey-apache-client4-1.9.jar
>> >     |-- jersey-client-1.9.jar
>> >     |-- jetty-http-8.1.10.v20130312.jar
>> >     |-- jetty-io-8.1.10.v20130312.jar
>> >     |-- jetty-util-8.1.10.v20130312.jar
>> >     |-- jetty-websocket-8.1.10.v20130312.jar
>> >     |-- jline-2.11.jar
>> >     |-- kryo-2.24.0.jar
>> >     |-- mbassador-1.1.9.jar
>> >     |-- minlog-1.2.jar
>> >     |-- netlet-1.2.0.jar
>> >     |-- netty-3.6.6.Final.jar
>> >     |-- objenesis-2.1.jar
>> >     |-- validation-api-1.1.0.Final.jar
>> >     |-- xbean-asm5-shaded-4.3.jar
>> >     `-- zip4j-1.3.2.jar
>> >
>> > 3 directories, 40 files
>> >
>> >
>> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <th...@datatorrent.com>
>> > wrote:
>> >
>> > > Chinmay,
>> > >
>> > > Before discussing where to put the files, let's make sure they are
>> really
>> > > needed for the operation of the CLI. As for names, anything new needs
>> to
>> > > reflect Apex in the name and should follow common conventions,
>> especially
>> > > in Bigtop where there are many existing integrations to look at.
>> > >
>> > > Thanks,
>> > > Thomas
>> > >
>> > >
>> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
>> > > chinmay@datatorrent.com>
>> > > wrote:
>> > >
>> > > > Really good points Pradeep and Aniruddha.
>> > > >
>> > > > 1. I believe we won't need to change the dtcli considering it works
>> > with
>> > > DT
>> > > > Community edition. We can keep the directory structure similar to
>> that.
>> > > > dt-env.sh has variables which contains information required for
>> dtcli
>> > to
>> > > > launch.
>> > > >
>> > > > 2. Let me check with bigtop community that whether they facilitate
>> the
>> > > > installation of rpms/debs before user is created. In either case,
>> > current
>> > > > dtcli creates a .dt folder in home directory. Also before putting
>> > > anything
>> > > > in /etc/skel we need to define what are the default contents that
>> > should
>> > > go
>> > > > to ~/.dt/ folder. If there is no defaults, probably we should not
>> > > > explicitly add it in /etc/skel.
>> > > >
>> > > > 3.  /etc/profile.d approach looks nice. This way contents of
>> dt-env.sh
>> > > are
>> > > > present as env variables. But I see a catch there. Adding dt-env.sh
>> to
>> > > > /etc/profile.d would make all the variables available at runtime all
>> > the
>> > > > time. I feel a little skeptical about that. Maybe a possible
>> collision
>> > > can
>> > > > occur with other application vars.
>> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
>> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
>> > > >
>> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
>> > > Jira/repository.
>> > > > Packaging related code usually goes there. (That's what all the
>> > > components
>> > > > in bigtop does).
>> > > > Having said that, once a PR is created, I'll be sharing the link of
>> PR
>> > > > here, so that apex community as well can review it.
>> > > >
>> > > > Thanks,
>> > > > Chinmay.
>> > > >
>> > > >
>> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
>> > > > aniruddha@datatorrent.com> wrote:
>> > > >
>> > > > > +1 on suggestions and approach.
>> > > > > We may need to iron out details about exact paths etc.
>> > > > > Which can be done on jira / PR comments.
>> > > > > Is that right @dev?
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
>> > apache@pradeepdalvi.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Inline comments...
>> > > > > >
>> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
>> > > > > > chinmay@datatorrent.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > @Thomas, Not all the jar present in DT community edition will
>> be
>> > > > > included
>> > > > > > > there.
>> > > > > > >
>> > > > > > > Here is the list of jars I found from common dependencies
>> between
>> > > of
>> > > > > apex
>> > > > > > > and DT Community edition:
>> > > > > > > netlet-1.2.0.jar
>> > > > > > > kryo-2.24.0.jar
>> > > > > > > jackson-core-asl-1.9.2.jar
>> > > > > > > jackson-mapper-asl-1.9.2.jar
>> > > > > > > async-http-client-1.7.20.jar
>> > > > > > > netty-3.6.6.Final.jar
>> > > > > > > validation-api-1.1.0.Final.jar
>> > > > > > > bval-jsr303-0.5.jar
>> > > > > > > bval-core-0.5.jar
>> > > > > > > commons-lang3-3.1.jar
>> > > > > > > commons-beanutils-1.8.3.jar
>> > > > > > > httpclient-4.3.5.jar
>> > > > > > > commons-codec-1.10.jar
>> > > > > > > zip4j-1.3.2.jar
>> > > > > > > jetty-websocket-8.1.10.v20130312.jar
>> > > > > > > xbean-asm5-shaded-4.3.jar
>> > > > > > > jersey-apache-client4-1.9.jar
>> > > > > > > jline-2.11.jar
>> > > > > > > ant-1.9.2.jar
>> > > > > > > ant-launcher-1.9.2.jar
>> > > > > > > mbassador-1.1.9.jar
>> > > > > > > jackson-jaxrs-1.9.2.jar
>> > > > > > > jackson-xc-1.9.2.jar
>> > > > > > > hadoop-common-2.2.0-tests.jar
>> > > > > > >
>> > > > > > > Ofcourse, I'll be running some tests do check that dtcli works
>> > > > properly
>> > > > > > for
>> > > > > > > launch+shutdown+kill of apps with only these libraries
>> present in
>> > > > > > isolation
>> > > > > > > without dependency on local m2.
>> > > > > > > I'm believe that there are unwanted jars which are used for
>> > compile
>> > > > > time
>> > > > > > > dependency and not runtime in above list which I can drop to
>> keep
>> > > > > package
>> > > > > > > size minimal.
>> > > > > > >
>> > > > > > >
>> > > > > > > @Anniruddha,
>> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
>> required.
>> > > > Only
>> > > > > > > required values I see in it are classpath. and dtcli is
>> capable
>> > of
>> > > > > > building
>> > > > > > > it on the fly.
>> > > > > > >
>> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find
>> the
>> > jar
>> > > > > path,
>> > > > > > > dt-env.sh and all such conf files. If we need a change in
>> dtcli,
>> > > then
>> > > > > > > community's opinion is required for whether dtcli should
>> change
>> > in
>> > > > our
>> > > > > > repo
>> > > > > > > or a copy of that with required changes exist in bigtop repo
>> > until
>> > > we
>> > > > > > make
>> > > > > > > the dtcli generic enough.
>> > > > > > >
>> > > > > >
>> > > > > > Can we have these file paths set using environment variables
>> with
>> > > some
>> > > > > > default values in dtcli?
>> > > > > > Then we can set such params in dt-env.sh.
>> > > > > >
>> > > > > >
>> > > > > > > 3. dt-sited.xml -> How is this file different from
>> dt-site.xml?
>> > Not
>> > > > > sure
>> > > > > > if
>> > > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel
>> are
>> > > > copied
>> > > > > to
>> > > > > > > home of new user when useradd program is called. But for
>> existing
>> > > > users
>> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on
>> the
>> > fly
>> > > > if
>> > > > > > not
>> > > > > > > encountered for the first time. Again correct me if I'm wrong.
>> > > > > > >
>> > > > > >
>> > > > > > Yes, you are right. However this would probably be necessary
>> step
>> > for
>> > > > > > rpm/deb, as they may also get installed during OS install and
>> > before
>> > > > user
>> > > > > > accounts were created.
>> > > > > >
>> > > > > >
>> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best
>> > > idea.
>> > > > > > We'll
>> > > > > > > anyway put dtcli in location which is by default present in
>> path
>> > > i.e.
>> > > > > > > /usr/bin.
>> > > > > >
>> > > > > > For other env variables specific to apex, I'll prefer to use
>> > > dt-env.sh
>> > > > > and
>> > > > > > > source it in dtcli rather than changing bashrc etc...
>> > > > > >
>> > > > > >
>> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
>> cleaner
>> > way
>> > > > to
>> > > > > > achieve the same.
>> > > > > >
>> > > > > >
>> > > > > > >
>> > > > > > Thanks,
>> > > > > > > Chinmay.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
>> > > > > > > aniruddha@datatorrent.com> wrote:
>> > > > > > >
>> > > > > > > > Hi,
>> > > > > > > >
>> > > > > > > > @Chinmay,
>> > > > > > > >
>> > > > > > > > We need to consider following:
>> > > > > > > > System wide default config files can be located at following
>> > > > > locations:
>> > > > > > > >
>> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour
>> on
>> > > how
>> > > > it
>> > > > > > > finds
>> > > > > > > > those locations)
>> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
>> behaviour on
>> > > how
>> > > > > it
>> > > > > > > > finds those locations)
>> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users
>> that
>> > > will
>> > > > > be
>> > > > > > > > created in system in future.)
>> > > > > > > >
>> > > > > > > > We may also have to modify bashrc / bashprofile for
>> population
>> > > any
>> > > > > > > > variables that are required.
>> > > > > > > >
>> > > > > > > > @dev, please put in your comments / suggestion.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Aniruddha
>> > > > > > > >
>> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
>> > > > > thomas@datatorrent.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > -->
>> > > > > > > > >
>> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
>> > > > > > > > chinmay@datatorrent.com
>> > > > > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hello Everyone!!
>> > > > > > > > > >
>> > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex, here
>> > are
>> > > > some
>> > > > > > > > > proposals
>> > > > > > > > > > about package structure etc..
>> > > > > > > > > > Before posting it on bbigtop mailing list, I have some
>> > > question
>> > > > > for
>> > > > > > > > apex
>> > > > > > > > > > community.
>> > > > > > > > > >
>> > > > > > > > > > Proposed Directory structure of apex package for both
>> deb &
>> > > > rpm:
>> > > > > > > > > >
>> > > > > > > > > > /usr/lib/apex/bin/dtcli
>> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
>> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
>> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
>> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
>> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
>> > > > > > > > >
>> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
>> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
>> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
>> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
>> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
>> > > > > > > > > > /usr/share/doc/apex/NOTICE
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > <version> = 3.3.0-incubating.
>> > > > > > > > > > <other dependent jars> = All the 3rd party jars which
>> are
>> > > > > required
>> > > > > > > for
>> > > > > > > > > apex
>> > > > > > > > > > to run. Usually the dependencies are packaged as part of
>> > > > rpm/deb
>> > > > > by
>> > > > > > > any
>> > > > > > > > > > software in bigtop.
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Can you specify what those jars are in Bigtop context?
>> Same
>> > as
>> > > > > > shipped
>> > > > > > > > with
>> > > > > > > > > DT community addition under lib/ ?
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
>> party
>> > > jars
>> > > > > > which
>> > > > > > > > > needs
>> > > > > > > > > > to included while packaging.
>> > > > > > > > > >
>> > > > > > > > > > Questions related to this:
>> > > > > > > > > > 1. Should we call the cli of apex as "apex" instead of
>> > > "dtcli"
>> > > > in
>> > > > > > > > bigtop
>> > > > > > > > > > package?
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > I think we should keep the name until we are able to
>> change
>> > it
>> > > in
>> > > > > > Apex.
>> > > > > > > > > Otherwise this may get confusing.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > > 2. I see that all softwares in bigtop have man page for
>> > their
>> > > > > > > > > executables.
>> > > > > > > > > > I think we should have it too for dtcli. Is there any
>> > > > > documentation
>> > > > > > > > > which I
>> > > > > > > > > > can convert to man page? or can I use output of "dtcli
>> > > --help"?
>> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
>> 3.3.0 OR
>> > > > > > > > > > 3.3.0-incubating?
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Has to be -incubating
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
>> 2.7.1
>> > > > > version
>> > > > > > of
>> > > > > > > > > > bigtop hadoop? Any problems that we see with this
>> > dependency?
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Should work without changed. I thought we certified
>> against
>> > > > 2.7.0?
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > > 5. Following is the apache mirror from which bigtop will
>> > pick
>> > > > the
>> > > > > > > apex
>> > > > > > > > > > source for compilation and packaging. Please correct if
>> > > > > incorrect:
>> > > > > > > > > >
>> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by all
>> the
>> > > > > > softwares
>> > > > > > > in
>> > > > > > > > > > bigtop.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Please share your opinion.
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Chinmay.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
>> > > > > > > > > > chinmay@datatorrent.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Yes.. Starting to work on the packaging.
>> > > > > > > > > > >
>> > > > > > > > > > > I've already started discussion on bigtop dev mailing
>> > list
>> > > > for
>> > > > > > > > > > > integration. Also created a Jira for the same. For
>> this
>> > > > > > communities
>> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
>> > > > > > > thomas@datatorrent.com
>> > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > >> Chinmay,
>> > > > > > > > > > >>
>> > > > > > > > > > >> I don't see anything under prerequisites that would
>> be a
>> > > > > > problem.
>> > > > > > > We
>> > > > > > > > > > >> looked
>> > > > > > > > > > >> at the ASF licencing compatibility as part of
>> becoming
>> > an
>> > > > > > > incubator
>> > > > > > > > > > >> project.
>> > > > > > > > > > >>
>> > > > > > > > > > >> Please focus on the packaging during the next weeks.
>> > Since
>> > > > the
>> > > > > > > work
>> > > > > > > > > will
>> > > > > > > > > > >> be
>> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs should
>> > also
>> > > be
>> > > > > > > there.
>> > > > > > > > > > >>
>> > > > > > > > > > >> Would be good to have the packaging in place by end
>> Feb.
>> > > > > > > > > > >>
>> > > > > > > > > > >> Thanks,
>> > > > > > > > > > >> Thomas
>> > > > > > > > > > >>
>> > > > > > > > > > >>
>> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
>> > > > > > > > > > >> chinmay@datatorrent.com>
>> > > > > > > > > > >> wrote:
>> > > > > > > > > > >>
>> > > > > > > > > > >> > Hi All,
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > We're planning a work on adding Apache Apex as a
>> > > component
>> > > > > to
>> > > > > > > > Apache
>> > > > > > > > > > >> > Bigtop.
>> > > > > > > > > > >> > Bigtop is the packaging system for the Apache big
>> data
>> > > > > > > ecosystem.
>> > > > > > > > > > >> Several
>> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for the
>> > same:
>> > > > > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > Proposed plan of execution is as follows:
>> > > > > > > > > > >> > *Step 1) Handle prerequisites*
>> > > > > > > > > > >> > Apache bigtop has some hard and soft expectation
>> for
>> > new
>> > > > > > > > components
>> > > > > > > > > to
>> > > > > > > > > > >> get
>> > > > > > > > > > >> > integrated into Bigtop.
>> > > > > > > > > > >> > Here is the list of it:
>> > > > > > > > > > >> >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > Most of them seems to be standard ASF based
>> > > requirements,
>> > > > > but
>> > > > > > > few
>> > > > > > > > > need
>> > > > > > > > > > >> to
>> > > > > > > > > > >> > be checked for:
>> > > > > > > > > > >> > 1. Software projects are expected to be Licensed
>> under
>> > > > > Apache
>> > > > > > > > > License,
>> > > > > > > > > > >> > Version 2.0 (and their dependencies are expected
>> to be
>> > > > > > > compatible
>> > > > > > > > > with
>> > > > > > > > > > >> this
>> > > > > > > > > > >> > license)
>> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
>> > > > > dependencies
>> > > > > > of
>> > > > > > > > > Apex
>> > > > > > > > > > >> are
>> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
>> > > > > verification
>> > > > > > > > check.
>> > > > > > > > > > >> > 2. Software projects are expected to be compatible
>> > with
>> > > > all
>> > > > > of
>> > > > > > > the
>> > > > > > > > > > >> > supported platforms that Bigtop distribution is
>> > > targeting
>> > > > > > > > > > >> >     - This needs verifying whether our software
>> runs
>> > > fine
>> > > > in
>> > > > > > > > > centos-6
>> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
>> > opensuse-13.2.
>> > > > > > > > > > >> > 3. What smoke tests that should be added for
>> > deployment.
>> > > > > > > > > > >> > 4. Identifying the test artifacts which goes beyond
>> > > smoke
>> > > > > test
>> > > > > > > > > > >> >     - These are basically the integration tests for
>> > > > > > verification
>> > > > > > > > > after
>> > > > > > > > > > >> the
>> > > > > > > > > > >> > deployment. This is a soft requirement, but aim is
>> to
>> > > > > achieve
>> > > > > > > this
>> > > > > > > > > as
>> > > > > > > > > > >> well
>> > > > > > > > > > >> > or at least have explanation why not to include.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > If there are any from the link which explicitly
>> needs
>> > to
>> > > > be
>> > > > > > > > checked
>> > > > > > > > > > >> other
>> > > > > > > > > > >> > than above 4, please let us know.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
>> > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
>> learnt
>> > > that
>> > > > > the
>> > > > > > > > > bigtop
>> > > > > > > > > > >> > community want to see the addition of new
>> components
>> > in
>> > > > > > phases.
>> > > > > > > > Here
>> > > > > > > > > > are
>> > > > > > > > > > >> > the phases:
>> > > > > > > > > > >> > 1. Packaging
>> > > > > > > > > > >> >     - This needs creating of package i.e. rpm & deb
>> > > files.
>> > > > > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER,
>> > NOTES
>> > > > etc
>> > > > > if
>> > > > > > > any
>> > > > > > > > > > >> needed.
>> > > > > > > > > > >> >     - Any documentation that need to be added to
>> > > > > distribution
>> > > > > > of
>> > > > > > > > our
>> > > > > > > > > > >> > software.
>> > > > > > > > > > >> >     - Any license information of dependencies
>> required
>> > > to
>> > > > be
>> > > > > > > added
>> > > > > > > > > to
>> > > > > > > > > > >> > package
>> > > > > > > > > > >> > 2. Smoke tests (at very least)
>> > > > > > > > > > >> >     - Adding smoke test for packaging.
>> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
>> > > > configuration
>> > > > > > > > > > >> >     - Add puppet recipes for automatic deployment
>> > > > > > > > > > >> > 4. Integration tests
>> > > > > > > > > > >> >     - For verification of deployments.
>> > > > > > > > > > >> > 5. license clearance:
>> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff is
>> > > > compliant
>> > > > > > > with
>> > > > > > > > > ASF
>> > > > > > > > > > >> > license requirements. If you add code licenses
>> under
>> > > > > different
>> > > > > > > > > > licenses,
>> > > > > > > > > > >> > those would need to be listed in the NOTICE.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > Please share your thoughts on the approach.
>> > > > > > > > > > >> > We'll start corresponding communication on bigtop
>> > > mailing
>> > > > > list
>> > > > > > > as
>> > > > > > > > > > well.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > We have some specific questions/suggestions
>> related to
>> > > > what
>> > > > > > > should
>> > > > > > > > > be
>> > > > > > > > > > >> the
>> > > > > > > > > > >> > content of the package and what should be the smoke
>> > > tests,
>> > > > > but
>> > > > > > > in
>> > > > > > > > > the
>> > > > > > > > > > >> > interest of not having too much content here, we'll
>> > put
>> > > > the
>> > > > > > > > > questions
>> > > > > > > > > > >> as a
>> > > > > > > > > > >> > separate mail in this mailthread.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > Thanks,
>> > > > > > > > > > >> > Chinmay.
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Yes.. It'll be build from source tar downloaded from one of the apache
mirror: apache.osuosl.org

The reason for dt-env.sh is it's sourced from dtcli. This leaves us 3
options:
If we should not have any dt-env.sh, dtcli would need a change while
packaging.

But if we want to use dtcli as it is, then we would need dt-env.sh



On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <th...@datatorrent.com>
wrote:

> Looks good overall, though there shouldn't be any dt-env.sh
>
> What will the bigtop package be built from, the source tar?
>
>
>
> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <ch...@apache.org>
> wrote:
>
> > Hi Thomas,
> >
> > Thanks for the feedback.
> >
> > First, we're not changing the name anywhere. We'll follow what currently
> is
> > in source tarball for 3.3.0-incubating version.
> >
> > Secondly, I've mentioned a directory structure below which is inline with
> > other existing integrations with bigtop.
> >
> > The need for each file is as follows:
> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of
> source
> > code apex 3.3.0-incubating.
> >
> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for hadoop
> > binary path and exports an env variable for dtcli to use it.
> >
> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar way to
> > how DT Community Edition finds dependency jars.
> > Please note that this is the first iteration list of jars. I'm trying to
> > narrow this down to only those which are really required.
> > The test that I'm running to check if dtcli runs fine with given
> dependency
> > is to launch+shutdown+kill+status for pi demo.
> >
> > Above is the directory structure which is required for CLI to work.
> > What aniruddha and pradeep has mentioned are some additional files which
> > makes the like of administrator easier. For eg. /etc/skel
> >
> > Please let me know if this is inline with what you're thinking.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > .
> > |-- bin
> > |   `-- dtcli
> > |-- conf
> > |   `-- dt-env.sh
> > `-- lib
> >     |-- ant-1.9.2.jar
> >     |-- ant-launcher-1.9.2.jar
> >     |-- apex-api-3.3.0-incubating.jar
> >     |-- apex-bufferserver-3.3.0-incubating.jar
> >     |-- apex-common-3.3.0-incubating.jar
> >     |-- apex-engine.jar
> >     |-- apex-engine-tests.jar
> >     |-- async-http-client-1.7.20.jar
> >     |-- bval-core-0.5.jar
> >     |-- bval-jsr303-0.5.jar
> >     |-- commons-beanutils-1.8.3.jar
> >     |-- commons-codec-1.10.jar
> >     |-- commons-lang3-3.1.jar
> >     |-- commons-logging-1.1.3.jar
> >     |-- grizzly-http-servlet-2.1.2.jar
> >     |-- hadoop-common-2.2.0-tests.jar
> >     |-- httpclient-4.3.5.jar
> >     |-- httpcore-4.3.2.jar
> >     |-- jackson-core-asl-1.9.2.jar
> >     |-- jackson-mapper-asl-1.9.2.jar
> >     |-- javax.servlet-3.1.jar
> >     |-- javax.servlet-api-3.0.1.jar
> >     |-- jersey-apache-client4-1.9.jar
> >     |-- jersey-client-1.9.jar
> >     |-- jetty-http-8.1.10.v20130312.jar
> >     |-- jetty-io-8.1.10.v20130312.jar
> >     |-- jetty-util-8.1.10.v20130312.jar
> >     |-- jetty-websocket-8.1.10.v20130312.jar
> >     |-- jline-2.11.jar
> >     |-- kryo-2.24.0.jar
> >     |-- mbassador-1.1.9.jar
> >     |-- minlog-1.2.jar
> >     |-- netlet-1.2.0.jar
> >     |-- netty-3.6.6.Final.jar
> >     |-- objenesis-2.1.jar
> >     |-- validation-api-1.1.0.Final.jar
> >     |-- xbean-asm5-shaded-4.3.jar
> >     `-- zip4j-1.3.2.jar
> >
> > 3 directories, 40 files
> >
> >
> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> > > Chinmay,
> > >
> > > Before discussing where to put the files, let's make sure they are
> really
> > > needed for the operation of the CLI. As for names, anything new needs
> to
> > > reflect Apex in the name and should follow common conventions,
> especially
> > > in Bigtop where there are many existing integrations to look at.
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com>
> > > wrote:
> > >
> > > > Really good points Pradeep and Aniruddha.
> > > >
> > > > 1. I believe we won't need to change the dtcli considering it works
> > with
> > > DT
> > > > Community edition. We can keep the directory structure similar to
> that.
> > > > dt-env.sh has variables which contains information required for dtcli
> > to
> > > > launch.
> > > >
> > > > 2. Let me check with bigtop community that whether they facilitate
> the
> > > > installation of rpms/debs before user is created. In either case,
> > current
> > > > dtcli creates a .dt folder in home directory. Also before putting
> > > anything
> > > > in /etc/skel we need to define what are the default contents that
> > should
> > > go
> > > > to ~/.dt/ folder. If there is no defaults, probably we should not
> > > > explicitly add it in /etc/skel.
> > > >
> > > > 3.  /etc/profile.d approach looks nice. This way contents of
> dt-env.sh
> > > are
> > > > present as env variables. But I see a catch there. Adding dt-env.sh
> to
> > > > /etc/profile.d would make all the variables available at runtime all
> > the
> > > > time. I feel a little skeptical about that. Maybe a possible
> collision
> > > can
> > > > occur with other application vars.
> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > > >
> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > > Jira/repository.
> > > > Packaging related code usually goes there. (That's what all the
> > > components
> > > > in bigtop does).
> > > > Having said that, once a PR is created, I'll be sharing the link of
> PR
> > > > here, so that apex community as well can review it.
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > > aniruddha@datatorrent.com> wrote:
> > > >
> > > > > +1 on suggestions and approach.
> > > > > We may need to iron out details about exact paths etc.
> > > > > Which can be done on jira / PR comments.
> > > > > Is that right @dev?
> > > > >
> > > > >
> > > > >
> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > apache@pradeepdalvi.com>
> > > > > wrote:
> > > > >
> > > > > > Inline comments...
> > > > > >
> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > > > chinmay@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > @Thomas, Not all the jar present in DT community edition will
> be
> > > > > included
> > > > > > > there.
> > > > > > >
> > > > > > > Here is the list of jars I found from common dependencies
> between
> > > of
> > > > > apex
> > > > > > > and DT Community edition:
> > > > > > > netlet-1.2.0.jar
> > > > > > > kryo-2.24.0.jar
> > > > > > > jackson-core-asl-1.9.2.jar
> > > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > > > async-http-client-1.7.20.jar
> > > > > > > netty-3.6.6.Final.jar
> > > > > > > validation-api-1.1.0.Final.jar
> > > > > > > bval-jsr303-0.5.jar
> > > > > > > bval-core-0.5.jar
> > > > > > > commons-lang3-3.1.jar
> > > > > > > commons-beanutils-1.8.3.jar
> > > > > > > httpclient-4.3.5.jar
> > > > > > > commons-codec-1.10.jar
> > > > > > > zip4j-1.3.2.jar
> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > > jersey-apache-client4-1.9.jar
> > > > > > > jline-2.11.jar
> > > > > > > ant-1.9.2.jar
> > > > > > > ant-launcher-1.9.2.jar
> > > > > > > mbassador-1.1.9.jar
> > > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > > jackson-xc-1.9.2.jar
> > > > > > > hadoop-common-2.2.0-tests.jar
> > > > > > >
> > > > > > > Ofcourse, I'll be running some tests do check that dtcli works
> > > > properly
> > > > > > for
> > > > > > > launch+shutdown+kill of apps with only these libraries present
> in
> > > > > > isolation
> > > > > > > without dependency on local m2.
> > > > > > > I'm believe that there are unwanted jars which are used for
> > compile
> > > > > time
> > > > > > > dependency and not runtime in above list which I can drop to
> keep
> > > > > package
> > > > > > > size minimal.
> > > > > > >
> > > > > > >
> > > > > > > @Anniruddha,
> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> required.
> > > > Only
> > > > > > > required values I see in it are classpath. and dtcli is capable
> > of
> > > > > > building
> > > > > > > it on the fly.
> > > > > > >
> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the
> > jar
> > > > > path,
> > > > > > > dt-env.sh and all such conf files. If we need a change in
> dtcli,
> > > then
> > > > > > > community's opinion is required for whether dtcli should change
> > in
> > > > our
> > > > > > repo
> > > > > > > or a copy of that with required changes exist in bigtop repo
> > until
> > > we
> > > > > > make
> > > > > > > the dtcli generic enough.
> > > > > > >
> > > > > >
> > > > > > Can we have these file paths set using environment variables with
> > > some
> > > > > > default values in dtcli?
> > > > > > Then we can set such params in dt-env.sh.
> > > > > >
> > > > > >
> > > > > > > 3. dt-sited.xml -> How is this file different from dt-site.xml?
> > Not
> > > > > sure
> > > > > > if
> > > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel are
> > > > copied
> > > > > to
> > > > > > > home of new user when useradd program is called. But for
> existing
> > > > users
> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the
> > fly
> > > > if
> > > > > > not
> > > > > > > encountered for the first time. Again correct me if I'm wrong.
> > > > > > >
> > > > > >
> > > > > > Yes, you are right. However this would probably be necessary step
> > for
> > > > > > rpm/deb, as they may also get installed during OS install and
> > before
> > > > user
> > > > > > accounts were created.
> > > > > >
> > > > > >
> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best
> > > idea.
> > > > > > We'll
> > > > > > > anyway put dtcli in location which is by default present in
> path
> > > i.e.
> > > > > > > /usr/bin.
> > > > > >
> > > > > > For other env variables specific to apex, I'll prefer to use
> > > dt-env.sh
> > > > > and
> > > > > > > source it in dtcli rather than changing bashrc etc...
> > > > > >
> > > > > >
> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner
> > way
> > > > to
> > > > > > achieve the same.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > Thanks,
> > > > > > > Chinmay.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > > > aniruddha@datatorrent.com> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > @Chinmay,
> > > > > > > >
> > > > > > > > We need to consider following:
> > > > > > > > System wide default config files can be located at following
> > > > > locations:
> > > > > > > >
> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour
> on
> > > how
> > > > it
> > > > > > > finds
> > > > > > > > those locations)
> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour
> on
> > > how
> > > > > it
> > > > > > > > finds those locations)
> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users
> that
> > > will
> > > > > be
> > > > > > > > created in system in future.)
> > > > > > > >
> > > > > > > > We may also have to modify bashrc / bashprofile for
> population
> > > any
> > > > > > > > variables that are required.
> > > > > > > >
> > > > > > > > @dev, please put in your comments / suggestion.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > >
> > > > > > > > Aniruddha
> > > > > > > >
> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > > > thomas@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > -->
> > > > > > > > >
> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > > > chinmay@datatorrent.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello Everyone!!
> > > > > > > > > >
> > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex, here
> > are
> > > > some
> > > > > > > > > proposals
> > > > > > > > > > about package structure etc..
> > > > > > > > > > Before posting it on bbigtop mailing list, I have some
> > > question
> > > > > for
> > > > > > > > apex
> > > > > > > > > > community.
> > > > > > > > > >
> > > > > > > > > > Proposed Directory structure of apex package for both
> deb &
> > > > rpm:
> > > > > > > > > >
> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > > > >
> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > > > <other dependent jars> = All the 3rd party jars which are
> > > > > required
> > > > > > > for
> > > > > > > > > apex
> > > > > > > > > > to run. Usually the dependencies are packaged as part of
> > > > rpm/deb
> > > > > by
> > > > > > > any
> > > > > > > > > > software in bigtop.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Can you specify what those jars are in Bigtop context? Same
> > as
> > > > > > shipped
> > > > > > > > with
> > > > > > > > > DT community addition under lib/ ?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
> party
> > > jars
> > > > > > which
> > > > > > > > > needs
> > > > > > > > > > to included while packaging.
> > > > > > > > > >
> > > > > > > > > > Questions related to this:
> > > > > > > > > > 1. Should we call the cli of apex as "apex" instead of
> > > "dtcli"
> > > > in
> > > > > > > > bigtop
> > > > > > > > > > package?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I think we should keep the name until we are able to change
> > it
> > > in
> > > > > > Apex.
> > > > > > > > > Otherwise this may get confusing.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > 2. I see that all softwares in bigtop have man page for
> > their
> > > > > > > > > executables.
> > > > > > > > > > I think we should have it too for dtcli. Is there any
> > > > > documentation
> > > > > > > > > which I
> > > > > > > > > > can convert to man page? or can I use output of "dtcli
> > > --help"?
> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0
> OR
> > > > > > > > > > 3.3.0-incubating?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Has to be -incubating
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1
> > > > > version
> > > > > > of
> > > > > > > > > > bigtop hadoop? Any problems that we see with this
> > dependency?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Should work without changed. I thought we certified against
> > > > 2.7.0?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > 5. Following is the apache mirror from which bigtop will
> > pick
> > > > the
> > > > > > > apex
> > > > > > > > > > source for compilation and packaging. Please correct if
> > > > > incorrect:
> > > > > > > > > >
> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by all
> the
> > > > > > softwares
> > > > > > > in
> > > > > > > > > > bigtop.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Please share your opinion.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Chinmay.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > > > > chinmay@datatorrent.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > > > >
> > > > > > > > > > > I've already started discussion on bigtop dev mailing
> > list
> > > > for
> > > > > > > > > > > integration. Also created a Jira for the same. For this
> > > > > > communities
> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > > > > thomas@datatorrent.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Chinmay,
> > > > > > > > > > >>
> > > > > > > > > > >> I don't see anything under prerequisites that would
> be a
> > > > > > problem.
> > > > > > > We
> > > > > > > > > > >> looked
> > > > > > > > > > >> at the ASF licencing compatibility as part of becoming
> > an
> > > > > > > incubator
> > > > > > > > > > >> project.
> > > > > > > > > > >>
> > > > > > > > > > >> Please focus on the packaging during the next weeks.
> > Since
> > > > the
> > > > > > > work
> > > > > > > > > will
> > > > > > > > > > >> be
> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs should
> > also
> > > be
> > > > > > > there.
> > > > > > > > > > >>
> > > > > > > > > > >> Would be good to have the packaging in place by end
> Feb.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> Thomas
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi All,
> > > > > > > > > > >> >
> > > > > > > > > > >> > We're planning a work on adding Apache Apex as a
> > > component
> > > > > to
> > > > > > > > Apache
> > > > > > > > > > >> > Bigtop.
> > > > > > > > > > >> > Bigtop is the packaging system for the Apache big
> data
> > > > > > > ecosystem.
> > > > > > > > > > >> Several
> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for the
> > same:
> > > > > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > > > >> >
> > > > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > > > >> > Apache bigtop has some hard and soft expectation for
> > new
> > > > > > > > components
> > > > > > > > > to
> > > > > > > > > > >> get
> > > > > > > > > > >> > integrated into Bigtop.
> > > > > > > > > > >> > Here is the list of it:
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > > > >> >
> > > > > > > > > > >> > Most of them seems to be standard ASF based
> > > requirements,
> > > > > but
> > > > > > > few
> > > > > > > > > need
> > > > > > > > > > >> to
> > > > > > > > > > >> > be checked for:
> > > > > > > > > > >> > 1. Software projects are expected to be Licensed
> under
> > > > > Apache
> > > > > > > > > License,
> > > > > > > > > > >> > Version 2.0 (and their dependencies are expected to
> be
> > > > > > > compatible
> > > > > > > > > with
> > > > > > > > > > >> this
> > > > > > > > > > >> > license)
> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > > > > dependencies
> > > > > > of
> > > > > > > > > Apex
> > > > > > > > > > >> are
> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > > > > verification
> > > > > > > > check.
> > > > > > > > > > >> > 2. Software projects are expected to be compatible
> > with
> > > > all
> > > > > of
> > > > > > > the
> > > > > > > > > > >> > supported platforms that Bigtop distribution is
> > > targeting
> > > > > > > > > > >> >     - This needs verifying whether our software runs
> > > fine
> > > > in
> > > > > > > > > centos-6
> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > opensuse-13.2.
> > > > > > > > > > >> > 3. What smoke tests that should be added for
> > deployment.
> > > > > > > > > > >> > 4. Identifying the test artifacts which goes beyond
> > > smoke
> > > > > test
> > > > > > > > > > >> >     - These are basically the integration tests for
> > > > > > verification
> > > > > > > > > after
> > > > > > > > > > >> the
> > > > > > > > > > >> > deployment. This is a soft requirement, but aim is
> to
> > > > > achieve
> > > > > > > this
> > > > > > > > > as
> > > > > > > > > > >> well
> > > > > > > > > > >> > or at least have explanation why not to include.
> > > > > > > > > > >> >
> > > > > > > > > > >> > If there are any from the link which explicitly
> needs
> > to
> > > > be
> > > > > > > > checked
> > > > > > > > > > >> other
> > > > > > > > > > >> > than above 4, please let us know.
> > > > > > > > > > >> >
> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
> learnt
> > > that
> > > > > the
> > > > > > > > > bigtop
> > > > > > > > > > >> > community want to see the addition of new components
> > in
> > > > > > phases.
> > > > > > > > Here
> > > > > > > > > > are
> > > > > > > > > > >> > the phases:
> > > > > > > > > > >> > 1. Packaging
> > > > > > > > > > >> >     - This needs creating of package i.e. rpm & deb
> > > files.
> > > > > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER,
> > NOTES
> > > > etc
> > > > > if
> > > > > > > any
> > > > > > > > > > >> needed.
> > > > > > > > > > >> >     - Any documentation that need to be added to
> > > > > distribution
> > > > > > of
> > > > > > > > our
> > > > > > > > > > >> > software.
> > > > > > > > > > >> >     - Any license information of dependencies
> required
> > > to
> > > > be
> > > > > > > added
> > > > > > > > > to
> > > > > > > > > > >> > package
> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > > > configuration
> > > > > > > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > > > > > > >> > 4. Integration tests
> > > > > > > > > > >> >     - For verification of deployments.
> > > > > > > > > > >> > 5. license clearance:
> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff is
> > > > compliant
> > > > > > > with
> > > > > > > > > ASF
> > > > > > > > > > >> > license requirements. If you add code licenses under
> > > > > different
> > > > > > > > > > licenses,
> > > > > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > > > >> > We'll start corresponding communication on bigtop
> > > mailing
> > > > > list
> > > > > > > as
> > > > > > > > > > well.
> > > > > > > > > > >> >
> > > > > > > > > > >> > We have some specific questions/suggestions related
> to
> > > > what
> > > > > > > should
> > > > > > > > > be
> > > > > > > > > > >> the
> > > > > > > > > > >> > content of the package and what should be the smoke
> > > tests,
> > > > > but
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >> > interest of not having too much content here, we'll
> > put
> > > > the
> > > > > > > > > questions
> > > > > > > > > > >> as a
> > > > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks,
> > > > > > > > > > >> > Chinmay.
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

Looks good overall, though there shouldn't be any dt-env.sh

What will the bigtop package be built from, the source tar?



On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <ch...@apache.org>
wrote:

> Hi Thomas,
>
> Thanks for the feedback.
>
> First, we're not changing the name anywhere. We'll follow what currently is
> in source tarball for 3.3.0-incubating version.
>
> Secondly, I've mentioned a directory structure below which is inline with
> other existing integrations with bigtop.
>
> The need for each file is as follows:
> 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of source
> code apex 3.3.0-incubating.
>
> 2. conf/dt-env.sh -> This is source from dtcli. This searches for hadoop
> binary path and exports an env variable for dtcli to use it.
>
> 3. lib/ -> Libraries for CLI. This list is extracted in a similar way to
> how DT Community Edition finds dependency jars.
> Please note that this is the first iteration list of jars. I'm trying to
> narrow this down to only those which are really required.
> The test that I'm running to check if dtcli runs fine with given dependency
> is to launch+shutdown+kill+status for pi demo.
>
> Above is the directory structure which is required for CLI to work.
> What aniruddha and pradeep has mentioned are some additional files which
> makes the like of administrator easier. For eg. /etc/skel
>
> Please let me know if this is inline with what you're thinking.
>
> Thanks,
> Chinmay.
>
>
> .
> |-- bin
> |   `-- dtcli
> |-- conf
> |   `-- dt-env.sh
> `-- lib
>     |-- ant-1.9.2.jar
>     |-- ant-launcher-1.9.2.jar
>     |-- apex-api-3.3.0-incubating.jar
>     |-- apex-bufferserver-3.3.0-incubating.jar
>     |-- apex-common-3.3.0-incubating.jar
>     |-- apex-engine.jar
>     |-- apex-engine-tests.jar
>     |-- async-http-client-1.7.20.jar
>     |-- bval-core-0.5.jar
>     |-- bval-jsr303-0.5.jar
>     |-- commons-beanutils-1.8.3.jar
>     |-- commons-codec-1.10.jar
>     |-- commons-lang3-3.1.jar
>     |-- commons-logging-1.1.3.jar
>     |-- grizzly-http-servlet-2.1.2.jar
>     |-- hadoop-common-2.2.0-tests.jar
>     |-- httpclient-4.3.5.jar
>     |-- httpcore-4.3.2.jar
>     |-- jackson-core-asl-1.9.2.jar
>     |-- jackson-mapper-asl-1.9.2.jar
>     |-- javax.servlet-3.1.jar
>     |-- javax.servlet-api-3.0.1.jar
>     |-- jersey-apache-client4-1.9.jar
>     |-- jersey-client-1.9.jar
>     |-- jetty-http-8.1.10.v20130312.jar
>     |-- jetty-io-8.1.10.v20130312.jar
>     |-- jetty-util-8.1.10.v20130312.jar
>     |-- jetty-websocket-8.1.10.v20130312.jar
>     |-- jline-2.11.jar
>     |-- kryo-2.24.0.jar
>     |-- mbassador-1.1.9.jar
>     |-- minlog-1.2.jar
>     |-- netlet-1.2.0.jar
>     |-- netty-3.6.6.Final.jar
>     |-- objenesis-2.1.jar
>     |-- validation-api-1.1.0.Final.jar
>     |-- xbean-asm5-shaded-4.3.jar
>     `-- zip4j-1.3.2.jar
>
> 3 directories, 40 files
>
>
> On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
> > Chinmay,
> >
> > Before discussing where to put the files, let's make sure they are really
> > needed for the operation of the CLI. As for names, anything new needs to
> > reflect Apex in the name and should follow common conventions, especially
> > in Bigtop where there are many existing integrations to look at.
> >
> > Thanks,
> > Thomas
> >
> >
> > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com>
> > wrote:
> >
> > > Really good points Pradeep and Aniruddha.
> > >
> > > 1. I believe we won't need to change the dtcli considering it works
> with
> > DT
> > > Community edition. We can keep the directory structure similar to that.
> > > dt-env.sh has variables which contains information required for dtcli
> to
> > > launch.
> > >
> > > 2. Let me check with bigtop community that whether they facilitate the
> > > installation of rpms/debs before user is created. In either case,
> current
> > > dtcli creates a .dt folder in home directory. Also before putting
> > anything
> > > in /etc/skel we need to define what are the default contents that
> should
> > go
> > > to ~/.dt/ folder. If there is no defaults, probably we should not
> > > explicitly add it in /etc/skel.
> > >
> > > 3.  /etc/profile.d approach looks nice. This way contents of dt-env.sh
> > are
> > > present as env variables. But I see a catch there. Adding dt-env.sh to
> > > /etc/profile.d would make all the variables available at runtime all
> the
> > > time. I feel a little skeptical about that. Maybe a possible collision
> > can
> > > occur with other application vars.
> > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > >
> > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > Jira/repository.
> > > Packaging related code usually goes there. (That's what all the
> > components
> > > in bigtop does).
> > > Having said that, once a PR is created, I'll be sharing the link of PR
> > > here, so that apex community as well can review it.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > > aniruddha@datatorrent.com> wrote:
> > >
> > > > +1 on suggestions and approach.
> > > > We may need to iron out details about exact paths etc.
> > > > Which can be done on jira / PR comments.
> > > > Is that right @dev?
> > > >
> > > >
> > > >
> > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> apache@pradeepdalvi.com>
> > > > wrote:
> > > >
> > > > > Inline comments...
> > > > >
> > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > > chinmay@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > @Thomas, Not all the jar present in DT community edition will be
> > > > included
> > > > > > there.
> > > > > >
> > > > > > Here is the list of jars I found from common dependencies between
> > of
> > > > apex
> > > > > > and DT Community edition:
> > > > > > netlet-1.2.0.jar
> > > > > > kryo-2.24.0.jar
> > > > > > jackson-core-asl-1.9.2.jar
> > > > > > jackson-mapper-asl-1.9.2.jar
> > > > > > async-http-client-1.7.20.jar
> > > > > > netty-3.6.6.Final.jar
> > > > > > validation-api-1.1.0.Final.jar
> > > > > > bval-jsr303-0.5.jar
> > > > > > bval-core-0.5.jar
> > > > > > commons-lang3-3.1.jar
> > > > > > commons-beanutils-1.8.3.jar
> > > > > > httpclient-4.3.5.jar
> > > > > > commons-codec-1.10.jar
> > > > > > zip4j-1.3.2.jar
> > > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > > xbean-asm5-shaded-4.3.jar
> > > > > > jersey-apache-client4-1.9.jar
> > > > > > jline-2.11.jar
> > > > > > ant-1.9.2.jar
> > > > > > ant-launcher-1.9.2.jar
> > > > > > mbassador-1.1.9.jar
> > > > > > jackson-jaxrs-1.9.2.jar
> > > > > > jackson-xc-1.9.2.jar
> > > > > > hadoop-common-2.2.0-tests.jar
> > > > > >
> > > > > > Ofcourse, I'll be running some tests do check that dtcli works
> > > properly
> > > > > for
> > > > > > launch+shutdown+kill of apps with only these libraries present in
> > > > > isolation
> > > > > > without dependency on local m2.
> > > > > > I'm believe that there are unwanted jars which are used for
> compile
> > > > time
> > > > > > dependency and not runtime in above list which I can drop to keep
> > > > package
> > > > > > size minimal.
> > > > > >
> > > > > >
> > > > > > @Anniruddha,
> > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required.
> > > Only
> > > > > > required values I see in it are classpath. and dtcli is capable
> of
> > > > > building
> > > > > > it on the fly.
> > > > > >
> > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the
> jar
> > > > path,
> > > > > > dt-env.sh and all such conf files. If we need a change in dtcli,
> > then
> > > > > > community's opinion is required for whether dtcli should change
> in
> > > our
> > > > > repo
> > > > > > or a copy of that with required changes exist in bigtop repo
> until
> > we
> > > > > make
> > > > > > the dtcli generic enough.
> > > > > >
> > > > >
> > > > > Can we have these file paths set using environment variables with
> > some
> > > > > default values in dtcli?
> > > > > Then we can set such params in dt-env.sh.
> > > > >
> > > > >
> > > > > > 3. dt-sited.xml -> How is this file different from dt-site.xml?
> Not
> > > > sure
> > > > > if
> > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel are
> > > copied
> > > > to
> > > > > > home of new user when useradd program is called. But for existing
> > > users
> > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the
> fly
> > > if
> > > > > not
> > > > > > encountered for the first time. Again correct me if I'm wrong.
> > > > > >
> > > > >
> > > > > Yes, you are right. However this would probably be necessary step
> for
> > > > > rpm/deb, as they may also get installed during OS install and
> before
> > > user
> > > > > accounts were created.
> > > > >
> > > > >
> > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best
> > idea.
> > > > > We'll
> > > > > > anyway put dtcli in location which is by default present in path
> > i.e.
> > > > > > /usr/bin.
> > > > >
> > > > > For other env variables specific to apex, I'll prefer to use
> > dt-env.sh
> > > > and
> > > > > > source it in dtcli rather than changing bashrc etc...
> > > > >
> > > > >
> > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner
> way
> > > to
> > > > > achieve the same.
> > > > >
> > > > >
> > > > > >
> > > > > Thanks,
> > > > > > Chinmay.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > > aniruddha@datatorrent.com> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > @Chinmay,
> > > > > > >
> > > > > > > We need to consider following:
> > > > > > > System wide default config files can be located at following
> > > > locations:
> > > > > > >
> > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on
> > how
> > > it
> > > > > > finds
> > > > > > > those locations)
> > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on
> > how
> > > > it
> > > > > > > finds those locations)
> > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users that
> > will
> > > > be
> > > > > > > created in system in future.)
> > > > > > >
> > > > > > > We may also have to modify bashrc / bashprofile for population
> > any
> > > > > > > variables that are required.
> > > > > > >
> > > > > > > @dev, please put in your comments / suggestion.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > >
> > > > > > > Aniruddha
> > > > > > >
> > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > > thomas@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > -->
> > > > > > > >
> > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > > chinmay@datatorrent.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Everyone!!
> > > > > > > > >
> > > > > > > > > Continuing with packaging effort (rpm+deb) of apex, here
> are
> > > some
> > > > > > > > proposals
> > > > > > > > > about package structure etc..
> > > > > > > > > Before posting it on bbigtop mailing list, I have some
> > question
> > > > for
> > > > > > > apex
> > > > > > > > > community.
> > > > > > > > >
> > > > > > > > > Proposed Directory structure of apex package for both deb &
> > > rpm:
> > > > > > > > >
> > > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > > >
> > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > > <other dependent jars> = All the 3rd party jars which are
> > > > required
> > > > > > for
> > > > > > > > apex
> > > > > > > > > to run. Usually the dependencies are packaged as part of
> > > rpm/deb
> > > > by
> > > > > > any
> > > > > > > > > software in bigtop.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Can you specify what those jars are in Bigtop context? Same
> as
> > > > > shipped
> > > > > > > with
> > > > > > > > DT community addition under lib/ ?
> > > > > > > >
> > > > > > > >
> > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd party
> > jars
> > > > > which
> > > > > > > > needs
> > > > > > > > > to included while packaging.
> > > > > > > > >
> > > > > > > > > Questions related to this:
> > > > > > > > > 1. Should we call the cli of apex as "apex" instead of
> > "dtcli"
> > > in
> > > > > > > bigtop
> > > > > > > > > package?
> > > > > > > > >
> > > > > > > >
> > > > > > > > I think we should keep the name until we are able to change
> it
> > in
> > > > > Apex.
> > > > > > > > Otherwise this may get confusing.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 2. I see that all softwares in bigtop have man page for
> their
> > > > > > > > executables.
> > > > > > > > > I think we should have it too for dtcli. Is there any
> > > > documentation
> > > > > > > > which I
> > > > > > > > > can convert to man page? or can I use output of "dtcli
> > --help"?
> > > > > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > > > > > > 3.3.0-incubating?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Has to be -incubating
> > > > > > > >
> > > > > > > >
> > > > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1
> > > > version
> > > > > of
> > > > > > > > > bigtop hadoop? Any problems that we see with this
> dependency?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Should work without changed. I thought we certified against
> > > 2.7.0?
> > > > > > > >
> > > > > > > >
> > > > > > > > > 5. Following is the apache mirror from which bigtop will
> pick
> > > the
> > > > > > apex
> > > > > > > > > source for compilation and packaging. Please correct if
> > > > incorrect:
> > > > > > > > >
> > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by all the
> > > > > softwares
> > > > > > in
> > > > > > > > > bigtop.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Please share your opinion.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Chinmay.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > > > chinmay@datatorrent.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > > >
> > > > > > > > > > I've already started discussion on bigtop dev mailing
> list
> > > for
> > > > > > > > > > integration. Also created a Jira for the same. For this
> > > > > communities
> > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > > > thomas@datatorrent.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Chinmay,
> > > > > > > > > >>
> > > > > > > > > >> I don't see anything under prerequisites that would be a
> > > > > problem.
> > > > > > We
> > > > > > > > > >> looked
> > > > > > > > > >> at the ASF licencing compatibility as part of becoming
> an
> > > > > > incubator
> > > > > > > > > >> project.
> > > > > > > > > >>
> > > > > > > > > >> Please focus on the packaging during the next weeks.
> Since
> > > the
> > > > > > work
> > > > > > > > will
> > > > > > > > > >> be
> > > > > > > > > >> part of Bigtop, related discussions and JIRAs should
> also
> > be
> > > > > > there.
> > > > > > > > > >>
> > > > > > > > > >> Would be good to have the packaging in place by end Feb.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Thomas
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi All,
> > > > > > > > > >> >
> > > > > > > > > >> > We're planning a work on adding Apache Apex as a
> > component
> > > > to
> > > > > > > Apache
> > > > > > > > > >> > Bigtop.
> > > > > > > > > >> > Bigtop is the packaging system for the Apache big data
> > > > > > ecosystem.
> > > > > > > > > >> Several
> > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > > > >> >
> > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for the
> same:
> > > > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > > >> >
> > > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > > >> > Apache bigtop has some hard and soft expectation for
> new
> > > > > > > components
> > > > > > > > to
> > > > > > > > > >> get
> > > > > > > > > >> > integrated into Bigtop.
> > > > > > > > > >> > Here is the list of it:
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > > >> >
> > > > > > > > > >> > Most of them seems to be standard ASF based
> > requirements,
> > > > but
> > > > > > few
> > > > > > > > need
> > > > > > > > > >> to
> > > > > > > > > >> > be checked for:
> > > > > > > > > >> > 1. Software projects are expected to be Licensed under
> > > > Apache
> > > > > > > > License,
> > > > > > > > > >> > Version 2.0 (and their dependencies are expected to be
> > > > > > compatible
> > > > > > > > with
> > > > > > > > > >> this
> > > > > > > > > >> > license)
> > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > > > dependencies
> > > > > of
> > > > > > > > Apex
> > > > > > > > > >> are
> > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > > > verification
> > > > > > > check.
> > > > > > > > > >> > 2. Software projects are expected to be compatible
> with
> > > all
> > > > of
> > > > > > the
> > > > > > > > > >> > supported platforms that Bigtop distribution is
> > targeting
> > > > > > > > > >> >     - This needs verifying whether our software runs
> > fine
> > > in
> > > > > > > > centos-6
> > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> opensuse-13.2.
> > > > > > > > > >> > 3. What smoke tests that should be added for
> deployment.
> > > > > > > > > >> > 4. Identifying the test artifacts which goes beyond
> > smoke
> > > > test
> > > > > > > > > >> >     - These are basically the integration tests for
> > > > > verification
> > > > > > > > after
> > > > > > > > > >> the
> > > > > > > > > >> > deployment. This is a soft requirement, but aim is to
> > > > achieve
> > > > > > this
> > > > > > > > as
> > > > > > > > > >> well
> > > > > > > > > >> > or at least have explanation why not to include.
> > > > > > > > > >> >
> > > > > > > > > >> > If there are any from the link which explicitly needs
> to
> > > be
> > > > > > > checked
> > > > > > > > > >> other
> > > > > > > > > >> > than above 4, please let us know.
> > > > > > > > > >> >
> > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > > > >> > From one of the mail archive of Bigtop, it was learnt
> > that
> > > > the
> > > > > > > > bigtop
> > > > > > > > > >> > community want to see the addition of new components
> in
> > > > > phases.
> > > > > > > Here
> > > > > > > > > are
> > > > > > > > > >> > the phases:
> > > > > > > > > >> > 1. Packaging
> > > > > > > > > >> >     - This needs creating of package i.e. rpm & deb
> > files.
> > > > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER,
> NOTES
> > > etc
> > > > if
> > > > > > any
> > > > > > > > > >> needed.
> > > > > > > > > >> >     - Any documentation that need to be added to
> > > > distribution
> > > > > of
> > > > > > > our
> > > > > > > > > >> > software.
> > > > > > > > > >> >     - Any license information of dependencies required
> > to
> > > be
> > > > > > added
> > > > > > > > to
> > > > > > > > > >> > package
> > > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > > configuration
> > > > > > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > > > > > >> > 4. Integration tests
> > > > > > > > > >> >     - For verification of deployments.
> > > > > > > > > >> > 5. license clearance:
> > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff is
> > > compliant
> > > > > > with
> > > > > > > > ASF
> > > > > > > > > >> > license requirements. If you add code licenses under
> > > > different
> > > > > > > > > licenses,
> > > > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > > > >> >
> > > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > > >> > We'll start corresponding communication on bigtop
> > mailing
> > > > list
> > > > > > as
> > > > > > > > > well.
> > > > > > > > > >> >
> > > > > > > > > >> > We have some specific questions/suggestions related to
> > > what
> > > > > > should
> > > > > > > > be
> > > > > > > > > >> the
> > > > > > > > > >> > content of the package and what should be the smoke
> > tests,
> > > > but
> > > > > > in
> > > > > > > > the
> > > > > > > > > >> > interest of not having too much content here, we'll
> put
> > > the
> > > > > > > > questions
> > > > > > > > > >> as a
> > > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks,
> > > > > > > > > >> > Chinmay.
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@apache.org>]

Hi Thomas,

Thanks for the feedback.

First, we're not changing the name anywhere. We'll follow what currently is
in source tarball for 3.3.0-incubating version.

Secondly, I've mentioned a directory structure below which is inline with
other existing integrations with bigtop.

The need for each file is as follows:
1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of source
code apex 3.3.0-incubating.

2. conf/dt-env.sh -> This is source from dtcli. This searches for hadoop
binary path and exports an env variable for dtcli to use it.

3. lib/ -> Libraries for CLI. This list is extracted in a similar way to
how DT Community Edition finds dependency jars.
Please note that this is the first iteration list of jars. I'm trying to
narrow this down to only those which are really required.
The test that I'm running to check if dtcli runs fine with given dependency
is to launch+shutdown+kill+status for pi demo.

Above is the directory structure which is required for CLI to work.
What aniruddha and pradeep has mentioned are some additional files which
makes the like of administrator easier. For eg. /etc/skel

Please let me know if this is inline with what you're thinking.

Thanks,
Chinmay.


.
|-- bin
|   `-- dtcli
|-- conf
|   `-- dt-env.sh
`-- lib
    |-- ant-1.9.2.jar
    |-- ant-launcher-1.9.2.jar
    |-- apex-api-3.3.0-incubating.jar
    |-- apex-bufferserver-3.3.0-incubating.jar
    |-- apex-common-3.3.0-incubating.jar
    |-- apex-engine.jar
    |-- apex-engine-tests.jar
    |-- async-http-client-1.7.20.jar
    |-- bval-core-0.5.jar
    |-- bval-jsr303-0.5.jar
    |-- commons-beanutils-1.8.3.jar
    |-- commons-codec-1.10.jar
    |-- commons-lang3-3.1.jar
    |-- commons-logging-1.1.3.jar
    |-- grizzly-http-servlet-2.1.2.jar
    |-- hadoop-common-2.2.0-tests.jar
    |-- httpclient-4.3.5.jar
    |-- httpcore-4.3.2.jar
    |-- jackson-core-asl-1.9.2.jar
    |-- jackson-mapper-asl-1.9.2.jar
    |-- javax.servlet-3.1.jar
    |-- javax.servlet-api-3.0.1.jar
    |-- jersey-apache-client4-1.9.jar
    |-- jersey-client-1.9.jar
    |-- jetty-http-8.1.10.v20130312.jar
    |-- jetty-io-8.1.10.v20130312.jar
    |-- jetty-util-8.1.10.v20130312.jar
    |-- jetty-websocket-8.1.10.v20130312.jar
    |-- jline-2.11.jar
    |-- kryo-2.24.0.jar
    |-- mbassador-1.1.9.jar
    |-- minlog-1.2.jar
    |-- netlet-1.2.0.jar
    |-- netty-3.6.6.Final.jar
    |-- objenesis-2.1.jar
    |-- validation-api-1.1.0.Final.jar
    |-- xbean-asm5-shaded-4.3.jar
    `-- zip4j-1.3.2.jar

3 directories, 40 files


On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <th...@datatorrent.com>
wrote:

> Chinmay,
>
> Before discussing where to put the files, let's make sure they are really
> needed for the operation of the CLI. As for names, anything new needs to
> reflect Apex in the name and should follow common conventions, especially
> in Bigtop where there are many existing integrations to look at.
>
> Thanks,
> Thomas
>
>
> On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > Really good points Pradeep and Aniruddha.
> >
> > 1. I believe we won't need to change the dtcli considering it works with
> DT
> > Community edition. We can keep the directory structure similar to that.
> > dt-env.sh has variables which contains information required for dtcli to
> > launch.
> >
> > 2. Let me check with bigtop community that whether they facilitate the
> > installation of rpms/debs before user is created. In either case, current
> > dtcli creates a .dt folder in home directory. Also before putting
> anything
> > in /etc/skel we need to define what are the default contents that should
> go
> > to ~/.dt/ folder. If there is no defaults, probably we should not
> > explicitly add it in /etc/skel.
> >
> > 3.  /etc/profile.d approach looks nice. This way contents of dt-env.sh
> are
> > present as env variables. But I see a catch there. Adding dt-env.sh to
> > /etc/profile.d would make all the variables available at runtime all the
> > time. I feel a little skeptical about that. Maybe a possible collision
> can
> > occur with other application vars.
> > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > "~/.dt/dt-env.sh". So those variables are anyway available.
> >
> > @Aniruddha, the Jira and PR effort is happening at Bigtop
> Jira/repository.
> > Packaging related code usually goes there. (That's what all the
> components
> > in bigtop does).
> > Having said that, once a PR is created, I'll be sharing the link of PR
> > here, so that apex community as well can review it.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > aniruddha@datatorrent.com> wrote:
> >
> > > +1 on suggestions and approach.
> > > We may need to iron out details about exact paths etc.
> > > Which can be done on jira / PR comments.
> > > Is that right @dev?
> > >
> > >
> > >
> > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <ap...@pradeepdalvi.com>
> > > wrote:
> > >
> > > > Inline comments...
> > > >
> > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com>
> > > > wrote:
> > > >
> > > > > @Thomas, Not all the jar present in DT community edition will be
> > > included
> > > > > there.
> > > > >
> > > > > Here is the list of jars I found from common dependencies between
> of
> > > apex
> > > > > and DT Community edition:
> > > > > netlet-1.2.0.jar
> > > > > kryo-2.24.0.jar
> > > > > jackson-core-asl-1.9.2.jar
> > > > > jackson-mapper-asl-1.9.2.jar
> > > > > async-http-client-1.7.20.jar
> > > > > netty-3.6.6.Final.jar
> > > > > validation-api-1.1.0.Final.jar
> > > > > bval-jsr303-0.5.jar
> > > > > bval-core-0.5.jar
> > > > > commons-lang3-3.1.jar
> > > > > commons-beanutils-1.8.3.jar
> > > > > httpclient-4.3.5.jar
> > > > > commons-codec-1.10.jar
> > > > > zip4j-1.3.2.jar
> > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > xbean-asm5-shaded-4.3.jar
> > > > > jersey-apache-client4-1.9.jar
> > > > > jline-2.11.jar
> > > > > ant-1.9.2.jar
> > > > > ant-launcher-1.9.2.jar
> > > > > mbassador-1.1.9.jar
> > > > > jackson-jaxrs-1.9.2.jar
> > > > > jackson-xc-1.9.2.jar
> > > > > hadoop-common-2.2.0-tests.jar
> > > > >
> > > > > Ofcourse, I'll be running some tests do check that dtcli works
> > properly
> > > > for
> > > > > launch+shutdown+kill of apps with only these libraries present in
> > > > isolation
> > > > > without dependency on local m2.
> > > > > I'm believe that there are unwanted jars which are used for compile
> > > time
> > > > > dependency and not runtime in above list which I can drop to keep
> > > package
> > > > > size minimal.
> > > > >
> > > > >
> > > > > @Anniruddha,
> > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required.
> > Only
> > > > > required values I see in it are classpath. and dtcli is capable of
> > > > building
> > > > > it on the fly.
> > > > >
> > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar
> > > path,
> > > > > dt-env.sh and all such conf files. If we need a change in dtcli,
> then
> > > > > community's opinion is required for whether dtcli should change in
> > our
> > > > repo
> > > > > or a copy of that with required changes exist in bigtop repo until
> we
> > > > make
> > > > > the dtcli generic enough.
> > > > >
> > > >
> > > > Can we have these file paths set using environment variables with
> some
> > > > default values in dtcli?
> > > > Then we can set such params in dt-env.sh.
> > > >
> > > >
> > > > > 3. dt-sited.xml -> How is this file different from dt-site.xml? Not
> > > sure
> > > > if
> > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel are
> > copied
> > > to
> > > > > home of new user when useradd program is called. But for existing
> > users
> > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly
> > if
> > > > not
> > > > > encountered for the first time. Again correct me if I'm wrong.
> > > > >
> > > >
> > > > Yes, you are right. However this would probably be necessary step for
> > > > rpm/deb, as they may also get installed during OS install and before
> > user
> > > > accounts were created.
> > > >
> > > >
> > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best
> idea.
> > > > We'll
> > > > > anyway put dtcli in location which is by default present in path
> i.e.
> > > > > /usr/bin.
> > > >
> > > > For other env variables specific to apex, I'll prefer to use
> dt-env.sh
> > > and
> > > > > source it in dtcli rather than changing bashrc etc...
> > > >
> > > >
> > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way
> > to
> > > > achieve the same.
> > > >
> > > >
> > > > >
> > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > aniruddha@datatorrent.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > @Chinmay,
> > > > > >
> > > > > > We need to consider following:
> > > > > > System wide default config files can be located at following
> > > locations:
> > > > > >
> > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on
> how
> > it
> > > > > finds
> > > > > > those locations)
> > > > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on
> how
> > > it
> > > > > > finds those locations)
> > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users that
> will
> > > be
> > > > > > created in system in future.)
> > > > > >
> > > > > > We may also have to modify bashrc / bashprofile for population
> any
> > > > > > variables that are required.
> > > > > >
> > > > > > @dev, please put in your comments / suggestion.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > >
> > > > > > Aniruddha
> > > > > >
> > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > thomas@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > -->
> > > > > > >
> > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > chinmay@datatorrent.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Everyone!!
> > > > > > > >
> > > > > > > > Continuing with packaging effort (rpm+deb) of apex, here are
> > some
> > > > > > > proposals
> > > > > > > > about package structure etc..
> > > > > > > > Before posting it on bbigtop mailing list, I have some
> question
> > > for
> > > > > > apex
> > > > > > > > community.
> > > > > > > >
> > > > > > > > Proposed Directory structure of apex package for both deb &
> > rpm:
> > > > > > > >
> > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > >
> > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > >
> > > > > > > >
> > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > <other dependent jars> = All the 3rd party jars which are
> > > required
> > > > > for
> > > > > > > apex
> > > > > > > > to run. Usually the dependencies are packaged as part of
> > rpm/deb
> > > by
> > > > > any
> > > > > > > > software in bigtop.
> > > > > > > >
> > > > > > >
> > > > > > > Can you specify what those jars are in Bigtop context? Same as
> > > > shipped
> > > > > > with
> > > > > > > DT community addition under lib/ ?
> > > > > > >
> > > > > > >
> > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd party
> jars
> > > > which
> > > > > > > needs
> > > > > > > > to included while packaging.
> > > > > > > >
> > > > > > > > Questions related to this:
> > > > > > > > 1. Should we call the cli of apex as "apex" instead of
> "dtcli"
> > in
> > > > > > bigtop
> > > > > > > > package?
> > > > > > > >
> > > > > > >
> > > > > > > I think we should keep the name until we are able to change it
> in
> > > > Apex.
> > > > > > > Otherwise this may get confusing.
> > > > > > >
> > > > > > >
> > > > > > > > 2. I see that all softwares in bigtop have man page for their
> > > > > > > executables.
> > > > > > > > I think we should have it too for dtcli. Is there any
> > > documentation
> > > > > > > which I
> > > > > > > > can convert to man page? or can I use output of "dtcli
> --help"?
> > > > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > > > > > 3.3.0-incubating?
> > > > > > > >
> > > > > > >
> > > > > > > Has to be -incubating
> > > > > > >
> > > > > > >
> > > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1
> > > version
> > > > of
> > > > > > > > bigtop hadoop? Any problems that we see with this dependency?
> > > > > > > >
> > > > > > >
> > > > > > > Should work without changed. I thought we certified against
> > 2.7.0?
> > > > > > >
> > > > > > >
> > > > > > > > 5. Following is the apache mirror from which bigtop will pick
> > the
> > > > > apex
> > > > > > > > source for compilation and packaging. Please correct if
> > > incorrect:
> > > > > > > >
> http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > >     NOTE: apache.osuosl.org is the mirror used by all the
> > > > softwares
> > > > > in
> > > > > > > > bigtop.
> > > > > > > >
> > > > > > > >
> > > > > > > > Please share your opinion.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chinmay.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > > chinmay@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > >
> > > > > > > > > I've already started discussion on bigtop dev mailing list
> > for
> > > > > > > > > integration. Also created a Jira for the same. For this
> > > > communities
> > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > > thomas@datatorrent.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Chinmay,
> > > > > > > > >>
> > > > > > > > >> I don't see anything under prerequisites that would be a
> > > > problem.
> > > > > We
> > > > > > > > >> looked
> > > > > > > > >> at the ASF licencing compatibility as part of becoming an
> > > > > incubator
> > > > > > > > >> project.
> > > > > > > > >>
> > > > > > > > >> Please focus on the packaging during the next weeks. Since
> > the
> > > > > work
> > > > > > > will
> > > > > > > > >> be
> > > > > > > > >> part of Bigtop, related discussions and JIRAs should also
> be
> > > > > there.
> > > > > > > > >>
> > > > > > > > >> Would be good to have the packaging in place by end Feb.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Thomas
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi All,
> > > > > > > > >> >
> > > > > > > > >> > We're planning a work on adding Apache Apex as a
> component
> > > to
> > > > > > Apache
> > > > > > > > >> > Bigtop.
> > > > > > > > >> > Bigtop is the packaging system for the Apache big data
> > > > > ecosystem.
> > > > > > > > >> Several
> > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > > >> >
> > > > > > > > >> > Here is the tracking Jira task in APEXCORE for the same:
> > > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > >> >
> > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > >> > Apache bigtop has some hard and soft expectation for new
> > > > > > components
> > > > > > > to
> > > > > > > > >> get
> > > > > > > > >> > integrated into Bigtop.
> > > > > > > > >> > Here is the list of it:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > >> >
> > > > > > > > >> > Most of them seems to be standard ASF based
> requirements,
> > > but
> > > > > few
> > > > > > > need
> > > > > > > > >> to
> > > > > > > > >> > be checked for:
> > > > > > > > >> > 1. Software projects are expected to be Licensed under
> > > Apache
> > > > > > > License,
> > > > > > > > >> > Version 2.0 (and their dependencies are expected to be
> > > > > compatible
> > > > > > > with
> > > > > > > > >> this
> > > > > > > > >> > license)
> > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > > dependencies
> > > > of
> > > > > > > Apex
> > > > > > > > >> are
> > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > > verification
> > > > > > check.
> > > > > > > > >> > 2. Software projects are expected to be compatible with
> > all
> > > of
> > > > > the
> > > > > > > > >> > supported platforms that Bigtop distribution is
> targeting
> > > > > > > > >> >     - This needs verifying whether our software runs
> fine
> > in
> > > > > > > centos-6
> > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > > > > > > > >> > 3. What smoke tests that should be added for deployment.
> > > > > > > > >> > 4. Identifying the test artifacts which goes beyond
> smoke
> > > test
> > > > > > > > >> >     - These are basically the integration tests for
> > > > verification
> > > > > > > after
> > > > > > > > >> the
> > > > > > > > >> > deployment. This is a soft requirement, but aim is to
> > > achieve
> > > > > this
> > > > > > > as
> > > > > > > > >> well
> > > > > > > > >> > or at least have explanation why not to include.
> > > > > > > > >> >
> > > > > > > > >> > If there are any from the link which explicitly needs to
> > be
> > > > > > checked
> > > > > > > > >> other
> > > > > > > > >> > than above 4, please let us know.
> > > > > > > > >> >
> > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > > >> > From one of the mail archive of Bigtop, it was learnt
> that
> > > the
> > > > > > > bigtop
> > > > > > > > >> > community want to see the addition of new components in
> > > > phases.
> > > > > > Here
> > > > > > > > are
> > > > > > > > >> > the phases:
> > > > > > > > >> > 1. Packaging
> > > > > > > > >> >     - This needs creating of package i.e. rpm & deb
> files.
> > > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES
> > etc
> > > if
> > > > > any
> > > > > > > > >> needed.
> > > > > > > > >> >     - Any documentation that need to be added to
> > > distribution
> > > > of
> > > > > > our
> > > > > > > > >> > software.
> > > > > > > > >> >     - Any license information of dependencies required
> to
> > be
> > > > > added
> > > > > > > to
> > > > > > > > >> > package
> > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > configuration
> > > > > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > > > > >> > 4. Integration tests
> > > > > > > > >> >     - For verification of deployments.
> > > > > > > > >> > 5. license clearance:
> > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff is
> > compliant
> > > > > with
> > > > > > > ASF
> > > > > > > > >> > license requirements. If you add code licenses under
> > > different
> > > > > > > > licenses,
> > > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > > >> >
> > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > >> > We'll start corresponding communication on bigtop
> mailing
> > > list
> > > > > as
> > > > > > > > well.
> > > > > > > > >> >
> > > > > > > > >> > We have some specific questions/suggestions related to
> > what
> > > > > should
> > > > > > > be
> > > > > > > > >> the
> > > > > > > > >> > content of the package and what should be the smoke
> tests,
> > > but
> > > > > in
> > > > > > > the
> > > > > > > > >> > interest of not having too much content here, we'll put
> > the
> > > > > > > questions
> > > > > > > > >> as a
> > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Chinmay.
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

Chinmay,

Before discussing where to put the files, let's make sure they are really
needed for the operation of the CLI. As for names, anything new needs to
reflect Apex in the name and should follow common conventions, especially
in Bigtop where there are many existing integrations to look at.

Thanks,
Thomas


On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> Really good points Pradeep and Aniruddha.
>
> 1. I believe we won't need to change the dtcli considering it works with DT
> Community edition. We can keep the directory structure similar to that.
> dt-env.sh has variables which contains information required for dtcli to
> launch.
>
> 2. Let me check with bigtop community that whether they facilitate the
> installation of rpms/debs before user is created. In either case, current
> dtcli creates a .dt folder in home directory. Also before putting anything
> in /etc/skel we need to define what are the default contents that should go
> to ~/.dt/ folder. If there is no defaults, probably we should not
> explicitly add it in /etc/skel.
>
> 3.  /etc/profile.d approach looks nice. This way contents of dt-env.sh are
> present as env variables. But I see a catch there. Adding dt-env.sh to
> /etc/profile.d would make all the variables available at runtime all the
> time. I feel a little skeptical about that. Maybe a possible collision can
> occur with other application vars.
> Moreover, current dtcli does source "../conf/dt-env.sh" and
> "~/.dt/dt-env.sh". So those variables are anyway available.
>
> @Aniruddha, the Jira and PR effort is happening at Bigtop Jira/repository.
> Packaging related code usually goes there. (That's what all the components
> in bigtop does).
> Having said that, once a PR is created, I'll be sharing the link of PR
> here, so that apex community as well can review it.
>
> Thanks,
> Chinmay.
>
>
> On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> aniruddha@datatorrent.com> wrote:
>
> > +1 on suggestions and approach.
> > We may need to iron out details about exact paths etc.
> > Which can be done on jira / PR comments.
> > Is that right @dev?
> >
> >
> >
> > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <ap...@pradeepdalvi.com>
> > wrote:
> >
> > > Inline comments...
> > >
> > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com>
> > > wrote:
> > >
> > > > @Thomas, Not all the jar present in DT community edition will be
> > included
> > > > there.
> > > >
> > > > Here is the list of jars I found from common dependencies between of
> > apex
> > > > and DT Community edition:
> > > > netlet-1.2.0.jar
> > > > kryo-2.24.0.jar
> > > > jackson-core-asl-1.9.2.jar
> > > > jackson-mapper-asl-1.9.2.jar
> > > > async-http-client-1.7.20.jar
> > > > netty-3.6.6.Final.jar
> > > > validation-api-1.1.0.Final.jar
> > > > bval-jsr303-0.5.jar
> > > > bval-core-0.5.jar
> > > > commons-lang3-3.1.jar
> > > > commons-beanutils-1.8.3.jar
> > > > httpclient-4.3.5.jar
> > > > commons-codec-1.10.jar
> > > > zip4j-1.3.2.jar
> > > > jetty-websocket-8.1.10.v20130312.jar
> > > > xbean-asm5-shaded-4.3.jar
> > > > jersey-apache-client4-1.9.jar
> > > > jline-2.11.jar
> > > > ant-1.9.2.jar
> > > > ant-launcher-1.9.2.jar
> > > > mbassador-1.1.9.jar
> > > > jackson-jaxrs-1.9.2.jar
> > > > jackson-xc-1.9.2.jar
> > > > hadoop-common-2.2.0-tests.jar
> > > >
> > > > Ofcourse, I'll be running some tests do check that dtcli works
> properly
> > > for
> > > > launch+shutdown+kill of apps with only these libraries present in
> > > isolation
> > > > without dependency on local m2.
> > > > I'm believe that there are unwanted jars which are used for compile
> > time
> > > > dependency and not runtime in above list which I can drop to keep
> > package
> > > > size minimal.
> > > >
> > > >
> > > > @Anniruddha,
> > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required.
> Only
> > > > required values I see in it are classpath. and dtcli is capable of
> > > building
> > > > it on the fly.
> > > >
> > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar
> > path,
> > > > dt-env.sh and all such conf files. If we need a change in dtcli, then
> > > > community's opinion is required for whether dtcli should change in
> our
> > > repo
> > > > or a copy of that with required changes exist in bigtop repo until we
> > > make
> > > > the dtcli generic enough.
> > > >
> > >
> > > Can we have these file paths set using environment variables with some
> > > default values in dtcli?
> > > Then we can set such params in dt-env.sh.
> > >
> > >
> > > > 3. dt-sited.xml -> How is this file different from dt-site.xml? Not
> > sure
> > > if
> > > > adding in /etc/skel would help. Files & Dirs from /etc/skel are
> copied
> > to
> > > > home of new user when useradd program is called. But for existing
> users
> > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly
> if
> > > not
> > > > encountered for the first time. Again correct me if I'm wrong.
> > > >
> > >
> > > Yes, you are right. However this would probably be necessary step for
> > > rpm/deb, as they may also get installed during OS install and before
> user
> > > accounts were created.
> > >
> > >
> > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best idea.
> > > We'll
> > > > anyway put dtcli in location which is by default present in path i.e.
> > > > /usr/bin.
> > >
> > > For other env variables specific to apex, I'll prefer to use dt-env.sh
> > and
> > > > source it in dtcli rather than changing bashrc etc...
> > >
> > >
> > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way
> to
> > > achieve the same.
> > >
> > >
> > > >
> > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > >
> > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > aniruddha@datatorrent.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > @Chinmay,
> > > > >
> > > > > We need to consider following:
> > > > > System wide default config files can be located at following
> > locations:
> > > > >
> > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on how
> it
> > > > finds
> > > > > those locations)
> > > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on how
> > it
> > > > > finds those locations)
> > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users that will
> > be
> > > > > created in system in future.)
> > > > >
> > > > > We may also have to modify bashrc / bashprofile for population any
> > > > > variables that are required.
> > > > >
> > > > > @dev, please put in your comments / suggestion.
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > >
> > > > > Aniruddha
> > > > >
> > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > thomas@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > -->
> > > > > >
> > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > chinmay@datatorrent.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Everyone!!
> > > > > > >
> > > > > > > Continuing with packaging effort (rpm+deb) of apex, here are
> some
> > > > > > proposals
> > > > > > > about package structure etc..
> > > > > > > Before posting it on bbigtop mailing list, I have some question
> > for
> > > > > apex
> > > > > > > community.
> > > > > > >
> > > > > > > Proposed Directory structure of apex package for both deb &
> rpm:
> > > > > > >
> > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > >
> > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > >
> > > > > > >
> > > > > > > <version> = 3.3.0-incubating.
> > > > > > > <other dependent jars> = All the 3rd party jars which are
> > required
> > > > for
> > > > > > apex
> > > > > > > to run. Usually the dependencies are packaged as part of
> rpm/deb
> > by
> > > > any
> > > > > > > software in bigtop.
> > > > > > >
> > > > > >
> > > > > > Can you specify what those jars are in Bigtop context? Same as
> > > shipped
> > > > > with
> > > > > > DT community addition under lib/ ?
> > > > > >
> > > > > >
> > > > > > > <package LICENSE> = Licenses of corresponding of 3rd party jars
> > > which
> > > > > > needs
> > > > > > > to included while packaging.
> > > > > > >
> > > > > > > Questions related to this:
> > > > > > > 1. Should we call the cli of apex as "apex" instead of "dtcli"
> in
> > > > > bigtop
> > > > > > > package?
> > > > > > >
> > > > > >
> > > > > > I think we should keep the name until we are able to change it in
> > > Apex.
> > > > > > Otherwise this may get confusing.
> > > > > >
> > > > > >
> > > > > > > 2. I see that all softwares in bigtop have man page for their
> > > > > > executables.
> > > > > > > I think we should have it too for dtcli. Is there any
> > documentation
> > > > > > which I
> > > > > > > can convert to man page? or can I use output of "dtcli --help"?
> > > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > > > > 3.3.0-incubating?
> > > > > > >
> > > > > >
> > > > > > Has to be -incubating
> > > > > >
> > > > > >
> > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1
> > version
> > > of
> > > > > > > bigtop hadoop? Any problems that we see with this dependency?
> > > > > > >
> > > > > >
> > > > > > Should work without changed. I thought we certified against
> 2.7.0?
> > > > > >
> > > > > >
> > > > > > > 5. Following is the apache mirror from which bigtop will pick
> the
> > > > apex
> > > > > > > source for compilation and packaging. Please correct if
> > incorrect:
> > > > > > >     http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > >     NOTE: apache.osuosl.org is the mirror used by all the
> > > softwares
> > > > in
> > > > > > > bigtop.
> > > > > > >
> > > > > > >
> > > > > > > Please share your opinion.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Chinmay.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > chinmay@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > >
> > > > > > > > I've already started discussion on bigtop dev mailing list
> for
> > > > > > > > integration. Also created a Jira for the same. For this
> > > communities
> > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > thomas@datatorrent.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Chinmay,
> > > > > > > >>
> > > > > > > >> I don't see anything under prerequisites that would be a
> > > problem.
> > > > We
> > > > > > > >> looked
> > > > > > > >> at the ASF licencing compatibility as part of becoming an
> > > > incubator
> > > > > > > >> project.
> > > > > > > >>
> > > > > > > >> Please focus on the packaging during the next weeks. Since
> the
> > > > work
> > > > > > will
> > > > > > > >> be
> > > > > > > >> part of Bigtop, related discussions and JIRAs should also be
> > > > there.
> > > > > > > >>
> > > > > > > >> Would be good to have the packaging in place by end Feb.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Thomas
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > > >> chinmay@datatorrent.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Hi All,
> > > > > > > >> >
> > > > > > > >> > We're planning a work on adding Apache Apex as a component
> > to
> > > > > Apache
> > > > > > > >> > Bigtop.
> > > > > > > >> > Bigtop is the packaging system for the Apache big data
> > > > ecosystem.
> > > > > > > >> Several
> > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > >> >
> > > > > > > >> > Here is the tracking Jira task in APEXCORE for the same:
> > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > >> >
> > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > >> > Apache bigtop has some hard and soft expectation for new
> > > > > components
> > > > > > to
> > > > > > > >> get
> > > > > > > >> > integrated into Bigtop.
> > > > > > > >> > Here is the list of it:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > >> >
> > > > > > > >> > Most of them seems to be standard ASF based requirements,
> > but
> > > > few
> > > > > > need
> > > > > > > >> to
> > > > > > > >> > be checked for:
> > > > > > > >> > 1. Software projects are expected to be Licensed under
> > Apache
> > > > > > License,
> > > > > > > >> > Version 2.0 (and their dependencies are expected to be
> > > > compatible
> > > > > > with
> > > > > > > >> this
> > > > > > > >> > license)
> > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> > dependencies
> > > of
> > > > > > Apex
> > > > > > > >> are
> > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > verification
> > > > > check.
> > > > > > > >> > 2. Software projects are expected to be compatible with
> all
> > of
> > > > the
> > > > > > > >> > supported platforms that Bigtop distribution is targeting
> > > > > > > >> >     - This needs verifying whether our software runs fine
> in
> > > > > > centos-6
> > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > > > > > > >> > 3. What smoke tests that should be added for deployment.
> > > > > > > >> > 4. Identifying the test artifacts which goes beyond smoke
> > test
> > > > > > > >> >     - These are basically the integration tests for
> > > verification
> > > > > > after
> > > > > > > >> the
> > > > > > > >> > deployment. This is a soft requirement, but aim is to
> > achieve
> > > > this
> > > > > > as
> > > > > > > >> well
> > > > > > > >> > or at least have explanation why not to include.
> > > > > > > >> >
> > > > > > > >> > If there are any from the link which explicitly needs to
> be
> > > > > checked
> > > > > > > >> other
> > > > > > > >> > than above 4, please let us know.
> > > > > > > >> >
> > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > >> > From one of the mail archive of Bigtop, it was learnt that
> > the
> > > > > > bigtop
> > > > > > > >> > community want to see the addition of new components in
> > > phases.
> > > > > Here
> > > > > > > are
> > > > > > > >> > the phases:
> > > > > > > >> > 1. Packaging
> > > > > > > >> >     - This needs creating of package i.e. rpm & deb files.
> > > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES
> etc
> > if
> > > > any
> > > > > > > >> needed.
> > > > > > > >> >     - Any documentation that need to be added to
> > distribution
> > > of
> > > > > our
> > > > > > > >> > software.
> > > > > > > >> >     - Any license information of dependencies required to
> be
> > > > added
> > > > > > to
> > > > > > > >> > package
> > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > >> > 3. Puppet recipes for automatic deployment and
> configuration
> > > > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > > > >> > 4. Integration tests
> > > > > > > >> >     - For verification of deployments.
> > > > > > > >> > 5. license clearance:
> > > > > > > >> >     Run 'gradle rat' to make sure all new stuff is
> compliant
> > > > with
> > > > > > ASF
> > > > > > > >> > license requirements. If you add code licenses under
> > different
> > > > > > > licenses,
> > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > >> >
> > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > >> > We'll start corresponding communication on bigtop mailing
> > list
> > > > as
> > > > > > > well.
> > > > > > > >> >
> > > > > > > >> > We have some specific questions/suggestions related to
> what
> > > > should
> > > > > > be
> > > > > > > >> the
> > > > > > > >> > content of the package and what should be the smoke tests,
> > but
> > > > in
> > > > > > the
> > > > > > > >> > interest of not having too much content here, we'll put
> the
> > > > > > questions
> > > > > > > >> as a
> > > > > > > >> > separate mail in this mailthread.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Chinmay.
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@datatorrent.com>]

Really good points Pradeep and Aniruddha.

1. I believe we won't need to change the dtcli considering it works with DT
Community edition. We can keep the directory structure similar to that.
dt-env.sh has variables which contains information required for dtcli to
launch.

2. Let me check with bigtop community that whether they facilitate the
installation of rpms/debs before user is created. In either case, current
dtcli creates a .dt folder in home directory. Also before putting anything
in /etc/skel we need to define what are the default contents that should go
to ~/.dt/ folder. If there is no defaults, probably we should not
explicitly add it in /etc/skel.

3.  /etc/profile.d approach looks nice. This way contents of dt-env.sh are
present as env variables. But I see a catch there. Adding dt-env.sh to
/etc/profile.d would make all the variables available at runtime all the
time. I feel a little skeptical about that. Maybe a possible collision can
occur with other application vars.
Moreover, current dtcli does source "../conf/dt-env.sh" and
"~/.dt/dt-env.sh". So those variables are anyway available.

@Aniruddha, the Jira and PR effort is happening at Bigtop Jira/repository.
Packaging related code usually goes there. (That's what all the components
in bigtop does).
Having said that, once a PR is created, I'll be sharing the link of PR
here, so that apex community as well can review it.

Thanks,
Chinmay.


On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
aniruddha@datatorrent.com> wrote:

> +1 on suggestions and approach.
> We may need to iron out details about exact paths etc.
> Which can be done on jira / PR comments.
> Is that right @dev?
>
>
>
> On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <ap...@pradeepdalvi.com>
> wrote:
>
> > Inline comments...
> >
> > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com>
> > wrote:
> >
> > > @Thomas, Not all the jar present in DT community edition will be
> included
> > > there.
> > >
> > > Here is the list of jars I found from common dependencies between of
> apex
> > > and DT Community edition:
> > > netlet-1.2.0.jar
> > > kryo-2.24.0.jar
> > > jackson-core-asl-1.9.2.jar
> > > jackson-mapper-asl-1.9.2.jar
> > > async-http-client-1.7.20.jar
> > > netty-3.6.6.Final.jar
> > > validation-api-1.1.0.Final.jar
> > > bval-jsr303-0.5.jar
> > > bval-core-0.5.jar
> > > commons-lang3-3.1.jar
> > > commons-beanutils-1.8.3.jar
> > > httpclient-4.3.5.jar
> > > commons-codec-1.10.jar
> > > zip4j-1.3.2.jar
> > > jetty-websocket-8.1.10.v20130312.jar
> > > xbean-asm5-shaded-4.3.jar
> > > jersey-apache-client4-1.9.jar
> > > jline-2.11.jar
> > > ant-1.9.2.jar
> > > ant-launcher-1.9.2.jar
> > > mbassador-1.1.9.jar
> > > jackson-jaxrs-1.9.2.jar
> > > jackson-xc-1.9.2.jar
> > > hadoop-common-2.2.0-tests.jar
> > >
> > > Ofcourse, I'll be running some tests do check that dtcli works properly
> > for
> > > launch+shutdown+kill of apps with only these libraries present in
> > isolation
> > > without dependency on local m2.
> > > I'm believe that there are unwanted jars which are used for compile
> time
> > > dependency and not runtime in above list which I can drop to keep
> package
> > > size minimal.
> > >
> > >
> > > @Anniruddha,
> > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required. Only
> > > required values I see in it are classpath. and dtcli is capable of
> > building
> > > it on the fly.
> > >
> > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar
> path,
> > > dt-env.sh and all such conf files. If we need a change in dtcli, then
> > > community's opinion is required for whether dtcli should change in our
> > repo
> > > or a copy of that with required changes exist in bigtop repo until we
> > make
> > > the dtcli generic enough.
> > >
> >
> > Can we have these file paths set using environment variables with some
> > default values in dtcli?
> > Then we can set such params in dt-env.sh.
> >
> >
> > > 3. dt-sited.xml -> How is this file different from dt-site.xml? Not
> sure
> > if
> > > adding in /etc/skel would help. Files & Dirs from /etc/skel are copied
> to
> > > home of new user when useradd program is called. But for existing users
> > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly if
> > not
> > > encountered for the first time. Again correct me if I'm wrong.
> > >
> >
> > Yes, you are right. However this would probably be necessary step for
> > rpm/deb, as they may also get installed during OS install and before user
> > accounts were created.
> >
> >
> > > 4. Changing bashrc & bash_profile -> I'm not sure its the best idea.
> > We'll
> > > anyway put dtcli in location which is by default present in path i.e.
> > > /usr/bin.
> >
> > For other env variables specific to apex, I'll prefer to use dt-env.sh
> and
> > > source it in dtcli rather than changing bashrc etc...
> >
> >
> > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way to
> > achieve the same.
> >
> >
> > >
> > Thanks,
> > > Chinmay.
> > >
> > >
> > >
> > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > aniruddha@datatorrent.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > @Chinmay,
> > > >
> > > > We need to consider following:
> > > > System wide default config files can be located at following
> locations:
> > > >
> > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on how it
> > > finds
> > > > those locations)
> > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on how
> it
> > > > finds those locations)
> > > > /etc/skel/.dt/dt-sited.xml and other files (for new users that will
> be
> > > > created in system in future.)
> > > >
> > > > We may also have to modify bashrc / bashprofile for population any
> > > > variables that are required.
> > > >
> > > > @dev, please put in your comments / suggestion.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > Aniruddha
> > > >
> > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> thomas@datatorrent.com>
> > > > wrote:
> > > >
> > > > > -->
> > > > >
> > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello Everyone!!
> > > > > >
> > > > > > Continuing with packaging effort (rpm+deb) of apex, here are some
> > > > > proposals
> > > > > > about package structure etc..
> > > > > > Before posting it on bbigtop mailing list, I have some question
> for
> > > > apex
> > > > > > community.
> > > > > >
> > > > > > Proposed Directory structure of apex package for both deb & rpm:
> > > > > >
> > > > > > /usr/lib/apex/bin/dtcli
> > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > >
> > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > /usr/share/doc/apex/NOTICE
> > > > > >
> > > > > >
> > > > > > <version> = 3.3.0-incubating.
> > > > > > <other dependent jars> = All the 3rd party jars which are
> required
> > > for
> > > > > apex
> > > > > > to run. Usually the dependencies are packaged as part of rpm/deb
> by
> > > any
> > > > > > software in bigtop.
> > > > > >
> > > > >
> > > > > Can you specify what those jars are in Bigtop context? Same as
> > shipped
> > > > with
> > > > > DT community addition under lib/ ?
> > > > >
> > > > >
> > > > > > <package LICENSE> = Licenses of corresponding of 3rd party jars
> > which
> > > > > needs
> > > > > > to included while packaging.
> > > > > >
> > > > > > Questions related to this:
> > > > > > 1. Should we call the cli of apex as "apex" instead of "dtcli" in
> > > > bigtop
> > > > > > package?
> > > > > >
> > > > >
> > > > > I think we should keep the name until we are able to change it in
> > Apex.
> > > > > Otherwise this may get confusing.
> > > > >
> > > > >
> > > > > > 2. I see that all softwares in bigtop have man page for their
> > > > > executables.
> > > > > > I think we should have it too for dtcli. Is there any
> documentation
> > > > > which I
> > > > > > can convert to man page? or can I use output of "dtcli --help"?
> > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > > > 3.3.0-incubating?
> > > > > >
> > > > >
> > > > > Has to be -incubating
> > > > >
> > > > >
> > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1
> version
> > of
> > > > > > bigtop hadoop? Any problems that we see with this dependency?
> > > > > >
> > > > >
> > > > > Should work without changed. I thought we certified against 2.7.0?
> > > > >
> > > > >
> > > > > > 5. Following is the apache mirror from which bigtop will pick the
> > > apex
> > > > > > source for compilation and packaging. Please correct if
> incorrect:
> > > > > >     http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > >     NOTE: apache.osuosl.org is the mirror used by all the
> > softwares
> > > in
> > > > > > bigtop.
> > > > > >
> > > > > >
> > > > > > Please share your opinion.
> > > > > >
> > > > > > Thanks,
> > > > > > Chinmay.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > chinmay@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Yes.. Starting to work on the packaging.
> > > > > > >
> > > > > > > I've already started discussion on bigtop dev mailing list for
> > > > > > > integration. Also created a Jira for the same. For this
> > communities
> > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > thomas@datatorrent.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Chinmay,
> > > > > > >>
> > > > > > >> I don't see anything under prerequisites that would be a
> > problem.
> > > We
> > > > > > >> looked
> > > > > > >> at the ASF licencing compatibility as part of becoming an
> > > incubator
> > > > > > >> project.
> > > > > > >>
> > > > > > >> Please focus on the packaging during the next weeks. Since the
> > > work
> > > > > will
> > > > > > >> be
> > > > > > >> part of Bigtop, related discussions and JIRAs should also be
> > > there.
> > > > > > >>
> > > > > > >> Would be good to have the packaging in place by end Feb.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Thomas
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > >> chinmay@datatorrent.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi All,
> > > > > > >> >
> > > > > > >> > We're planning a work on adding Apache Apex as a component
> to
> > > > Apache
> > > > > > >> > Bigtop.
> > > > > > >> > Bigtop is the packaging system for the Apache big data
> > > ecosystem.
> > > > > > >> Several
> > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > >> >
> > > > > > >> > Here is the tracking Jira task in APEXCORE for the same:
> > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > >> >
> > > > > > >> > Proposed plan of execution is as follows:
> > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > >> > Apache bigtop has some hard and soft expectation for new
> > > > components
> > > > > to
> > > > > > >> get
> > > > > > >> > integrated into Bigtop.
> > > > > > >> > Here is the list of it:
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > >> >
> > > > > > >> > Most of them seems to be standard ASF based requirements,
> but
> > > few
> > > > > need
> > > > > > >> to
> > > > > > >> > be checked for:
> > > > > > >> > 1. Software projects are expected to be Licensed under
> Apache
> > > > > License,
> > > > > > >> > Version 2.0 (and their dependencies are expected to be
> > > compatible
> > > > > with
> > > > > > >> this
> > > > > > >> > license)
> > > > > > >> >     - Apex is under ASL 2.0 but need to check if
> dependencies
> > of
> > > > > Apex
> > > > > > >> are
> > > > > > >> > compatible with ASL 2.0. This I guess would be a
> verification
> > > > check.
> > > > > > >> > 2. Software projects are expected to be compatible with all
> of
> > > the
> > > > > > >> > supported platforms that Bigtop distribution is targeting
> > > > > > >> >     - This needs verifying whether our software runs fine in
> > > > > centos-6
> > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > > > > > >> > 3. What smoke tests that should be added for deployment.
> > > > > > >> > 4. Identifying the test artifacts which goes beyond smoke
> test
> > > > > > >> >     - These are basically the integration tests for
> > verification
> > > > > after
> > > > > > >> the
> > > > > > >> > deployment. This is a soft requirement, but aim is to
> achieve
> > > this
> > > > > as
> > > > > > >> well
> > > > > > >> > or at least have explanation why not to include.
> > > > > > >> >
> > > > > > >> > If there are any from the link which explicitly needs to be
> > > > checked
> > > > > > >> other
> > > > > > >> > than above 4, please let us know.
> > > > > > >> >
> > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > >> > From one of the mail archive of Bigtop, it was learnt that
> the
> > > > > bigtop
> > > > > > >> > community want to see the addition of new components in
> > phases.
> > > > Here
> > > > > > are
> > > > > > >> > the phases:
> > > > > > >> > 1. Packaging
> > > > > > >> >     - This needs creating of package i.e. rpm & deb files.
> > > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc
> if
> > > any
> > > > > > >> needed.
> > > > > > >> >     - Any documentation that need to be added to
> distribution
> > of
> > > > our
> > > > > > >> > software.
> > > > > > >> >     - Any license information of dependencies required to be
> > > added
> > > > > to
> > > > > > >> > package
> > > > > > >> > 2. Smoke tests (at very least)
> > > > > > >> >     - Adding smoke test for packaging.
> > > > > > >> > 3. Puppet recipes for automatic deployment and configuration
> > > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > > >> > 4. Integration tests
> > > > > > >> >     - For verification of deployments.
> > > > > > >> > 5. license clearance:
> > > > > > >> >     Run 'gradle rat' to make sure all new stuff is compliant
> > > with
> > > > > ASF
> > > > > > >> > license requirements. If you add code licenses under
> different
> > > > > > licenses,
> > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > >> >
> > > > > > >> > Please share your thoughts on the approach.
> > > > > > >> > We'll start corresponding communication on bigtop mailing
> list
> > > as
> > > > > > well.
> > > > > > >> >
> > > > > > >> > We have some specific questions/suggestions related to what
> > > should
> > > > > be
> > > > > > >> the
> > > > > > >> > content of the package and what should be the smoke tests,
> but
> > > in
> > > > > the
> > > > > > >> > interest of not having too much content here, we'll put the
> > > > > questions
> > > > > > >> as a
> > > > > > >> > separate mail in this mailthread.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Chinmay.
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Aniruddha Thombare <an...@datatorrent.com>]

+1 on suggestions and approach.
We may need to iron out details about exact paths etc.
Which can be done on jira / PR comments.
Is that right @dev?



On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <ap...@pradeepdalvi.com>
wrote:

> Inline comments...
>
> On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > @Thomas, Not all the jar present in DT community edition will be included
> > there.
> >
> > Here is the list of jars I found from common dependencies between of apex
> > and DT Community edition:
> > netlet-1.2.0.jar
> > kryo-2.24.0.jar
> > jackson-core-asl-1.9.2.jar
> > jackson-mapper-asl-1.9.2.jar
> > async-http-client-1.7.20.jar
> > netty-3.6.6.Final.jar
> > validation-api-1.1.0.Final.jar
> > bval-jsr303-0.5.jar
> > bval-core-0.5.jar
> > commons-lang3-3.1.jar
> > commons-beanutils-1.8.3.jar
> > httpclient-4.3.5.jar
> > commons-codec-1.10.jar
> > zip4j-1.3.2.jar
> > jetty-websocket-8.1.10.v20130312.jar
> > xbean-asm5-shaded-4.3.jar
> > jersey-apache-client4-1.9.jar
> > jline-2.11.jar
> > ant-1.9.2.jar
> > ant-launcher-1.9.2.jar
> > mbassador-1.1.9.jar
> > jackson-jaxrs-1.9.2.jar
> > jackson-xc-1.9.2.jar
> > hadoop-common-2.2.0-tests.jar
> >
> > Ofcourse, I'll be running some tests do check that dtcli works properly
> for
> > launch+shutdown+kill of apps with only these libraries present in
> isolation
> > without dependency on local m2.
> > I'm believe that there are unwanted jars which are used for compile time
> > dependency and not runtime in above list which I can drop to keep package
> > size minimal.
> >
> >
> > @Anniruddha,
> > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required. Only
> > required values I see in it are classpath. and dtcli is capable of
> building
> > it on the fly.
> >
> > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar path,
> > dt-env.sh and all such conf files. If we need a change in dtcli, then
> > community's opinion is required for whether dtcli should change in our
> repo
> > or a copy of that with required changes exist in bigtop repo until we
> make
> > the dtcli generic enough.
> >
>
> Can we have these file paths set using environment variables with some
> default values in dtcli?
> Then we can set such params in dt-env.sh.
>
>
> > 3. dt-sited.xml -> How is this file different from dt-site.xml? Not sure
> if
> > adding in /etc/skel would help. Files & Dirs from /etc/skel are copied to
> > home of new user when useradd program is called. But for existing users
> > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly if
> not
> > encountered for the first time. Again correct me if I'm wrong.
> >
>
> Yes, you are right. However this would probably be necessary step for
> rpm/deb, as they may also get installed during OS install and before user
> accounts were created.
>
>
> > 4. Changing bashrc & bash_profile -> I'm not sure its the best idea.
> We'll
> > anyway put dtcli in location which is by default present in path i.e.
> > /usr/bin.
>
> For other env variables specific to apex, I'll prefer to use dt-env.sh and
> > source it in dtcli rather than changing bashrc etc...
>
>
> Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way to
> achieve the same.
>
>
> >
> Thanks,
> > Chinmay.
> >
> >
> >
> > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > aniruddha@datatorrent.com> wrote:
> >
> > > Hi,
> > >
> > > @Chinmay,
> > >
> > > We need to consider following:
> > > System wide default config files can be located at following locations:
> > >
> > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on how it
> > finds
> > > those locations)
> > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on how it
> > > finds those locations)
> > > /etc/skel/.dt/dt-sited.xml and other files (for new users that will be
> > > created in system in future.)
> > >
> > > We may also have to modify bashrc / bashprofile for population any
> > > variables that are required.
> > >
> > > @dev, please put in your comments / suggestion.
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Aniruddha
> > >
> > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <th...@datatorrent.com>
> > > wrote:
> > >
> > > > -->
> > > >
> > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hello Everyone!!
> > > > >
> > > > > Continuing with packaging effort (rpm+deb) of apex, here are some
> > > > proposals
> > > > > about package structure etc..
> > > > > Before posting it on bbigtop mailing list, I have some question for
> > > apex
> > > > > community.
> > > > >
> > > > > Proposed Directory structure of apex package for both deb & rpm:
> > > > >
> > > > > /usr/lib/apex/bin/dtcli
> > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > /usr/lib/apex/lib/<other dependent jars>
> > > >
> > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > /usr/share/doc/apex/CHANGELOG
> > > > > /usr/share/doc/apex/NOTICE
> > > > >
> > > > >
> > > > > <version> = 3.3.0-incubating.
> > > > > <other dependent jars> = All the 3rd party jars which are required
> > for
> > > > apex
> > > > > to run. Usually the dependencies are packaged as part of rpm/deb by
> > any
> > > > > software in bigtop.
> > > > >
> > > >
> > > > Can you specify what those jars are in Bigtop context? Same as
> shipped
> > > with
> > > > DT community addition under lib/ ?
> > > >
> > > >
> > > > > <package LICENSE> = Licenses of corresponding of 3rd party jars
> which
> > > > needs
> > > > > to included while packaging.
> > > > >
> > > > > Questions related to this:
> > > > > 1. Should we call the cli of apex as "apex" instead of "dtcli" in
> > > bigtop
> > > > > package?
> > > > >
> > > >
> > > > I think we should keep the name until we are able to change it in
> Apex.
> > > > Otherwise this may get confusing.
> > > >
> > > >
> > > > > 2. I see that all softwares in bigtop have man page for their
> > > > executables.
> > > > > I think we should have it too for dtcli. Is there any documentation
> > > > which I
> > > > > can convert to man page? or can I use output of "dtcli --help"?
> > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > > 3.3.0-incubating?
> > > > >
> > > >
> > > > Has to be -incubating
> > > >
> > > >
> > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1  version
> of
> > > > > bigtop hadoop? Any problems that we see with this dependency?
> > > > >
> > > >
> > > > Should work without changed. I thought we certified against 2.7.0?
> > > >
> > > >
> > > > > 5. Following is the apache mirror from which bigtop will pick the
> > apex
> > > > > source for compilation and packaging. Please correct if incorrect:
> > > > >     http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > >     NOTE: apache.osuosl.org is the mirror used by all the
> softwares
> > in
> > > > > bigtop.
> > > > >
> > > > >
> > > > > Please share your opinion.
> > > > >
> > > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > chinmay@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Yes.. Starting to work on the packaging.
> > > > > >
> > > > > > I've already started discussion on bigtop dev mailing list for
> > > > > > integration. Also created a Jira for the same. For this
> communities
> > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > thomas@datatorrent.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Chinmay,
> > > > > >>
> > > > > >> I don't see anything under prerequisites that would be a
> problem.
> > We
> > > > > >> looked
> > > > > >> at the ASF licencing compatibility as part of becoming an
> > incubator
> > > > > >> project.
> > > > > >>
> > > > > >> Please focus on the packaging during the next weeks. Since the
> > work
> > > > will
> > > > > >> be
> > > > > >> part of Bigtop, related discussions and JIRAs should also be
> > there.
> > > > > >>
> > > > > >> Would be good to have the packaging in place by end Feb.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Thomas
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > >> chinmay@datatorrent.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi All,
> > > > > >> >
> > > > > >> > We're planning a work on adding Apache Apex as a component to
> > > Apache
> > > > > >> > Bigtop.
> > > > > >> > Bigtop is the packaging system for the Apache big data
> > ecosystem.
> > > > > >> Several
> > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > >> >
> > > > > >> > Here is the tracking Jira task in APEXCORE for the same:
> > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > >> >
> > > > > >> > Proposed plan of execution is as follows:
> > > > > >> > *Step 1) Handle prerequisites*
> > > > > >> > Apache bigtop has some hard and soft expectation for new
> > > components
> > > > to
> > > > > >> get
> > > > > >> > integrated into Bigtop.
> > > > > >> > Here is the list of it:
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > >> >
> > > > > >> > Most of them seems to be standard ASF based requirements, but
> > few
> > > > need
> > > > > >> to
> > > > > >> > be checked for:
> > > > > >> > 1. Software projects are expected to be Licensed under Apache
> > > > License,
> > > > > >> > Version 2.0 (and their dependencies are expected to be
> > compatible
> > > > with
> > > > > >> this
> > > > > >> > license)
> > > > > >> >     - Apex is under ASL 2.0 but need to check if dependencies
> of
> > > > Apex
> > > > > >> are
> > > > > >> > compatible with ASL 2.0. This I guess would be a verification
> > > check.
> > > > > >> > 2. Software projects are expected to be compatible with all of
> > the
> > > > > >> > supported platforms that Bigtop distribution is targeting
> > > > > >> >     - This needs verifying whether our software runs fine in
> > > > centos-6
> > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > > > > >> > 3. What smoke tests that should be added for deployment.
> > > > > >> > 4. Identifying the test artifacts which goes beyond smoke test
> > > > > >> >     - These are basically the integration tests for
> verification
> > > > after
> > > > > >> the
> > > > > >> > deployment. This is a soft requirement, but aim is to achieve
> > this
> > > > as
> > > > > >> well
> > > > > >> > or at least have explanation why not to include.
> > > > > >> >
> > > > > >> > If there are any from the link which explicitly needs to be
> > > checked
> > > > > >> other
> > > > > >> > than above 4, please let us know.
> > > > > >> >
> > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > >> > From one of the mail archive of Bigtop, it was learnt that the
> > > > bigtop
> > > > > >> > community want to see the addition of new components in
> phases.
> > > Here
> > > > > are
> > > > > >> > the phases:
> > > > > >> > 1. Packaging
> > > > > >> >     - This needs creating of package i.e. rpm & deb files.
> > > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if
> > any
> > > > > >> needed.
> > > > > >> >     - Any documentation that need to be added to distribution
> of
> > > our
> > > > > >> > software.
> > > > > >> >     - Any license information of dependencies required to be
> > added
> > > > to
> > > > > >> > package
> > > > > >> > 2. Smoke tests (at very least)
> > > > > >> >     - Adding smoke test for packaging.
> > > > > >> > 3. Puppet recipes for automatic deployment and configuration
> > > > > >> >     - Add puppet recipes for automatic deployment
> > > > > >> > 4. Integration tests
> > > > > >> >     - For verification of deployments.
> > > > > >> > 5. license clearance:
> > > > > >> >     Run 'gradle rat' to make sure all new stuff is compliant
> > with
> > > > ASF
> > > > > >> > license requirements. If you add code licenses under different
> > > > > licenses,
> > > > > >> > those would need to be listed in the NOTICE.
> > > > > >> >
> > > > > >> > Please share your thoughts on the approach.
> > > > > >> > We'll start corresponding communication on bigtop mailing list
> > as
> > > > > well.
> > > > > >> >
> > > > > >> > We have some specific questions/suggestions related to what
> > should
> > > > be
> > > > > >> the
> > > > > >> > content of the package and what should be the smoke tests, but
> > in
> > > > the
> > > > > >> > interest of not having too much content here, we'll put the
> > > > questions
> > > > > >> as a
> > > > > >> > separate mail in this mailthread.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Chinmay.
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

["Pradeep A. Dalvi" <ap...@pradeepdalvi.com>]

Inline comments...

On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> @Thomas, Not all the jar present in DT community edition will be included
> there.
>
> Here is the list of jars I found from common dependencies between of apex
> and DT Community edition:
> netlet-1.2.0.jar
> kryo-2.24.0.jar
> jackson-core-asl-1.9.2.jar
> jackson-mapper-asl-1.9.2.jar
> async-http-client-1.7.20.jar
> netty-3.6.6.Final.jar
> validation-api-1.1.0.Final.jar
> bval-jsr303-0.5.jar
> bval-core-0.5.jar
> commons-lang3-3.1.jar
> commons-beanutils-1.8.3.jar
> httpclient-4.3.5.jar
> commons-codec-1.10.jar
> zip4j-1.3.2.jar
> jetty-websocket-8.1.10.v20130312.jar
> xbean-asm5-shaded-4.3.jar
> jersey-apache-client4-1.9.jar
> jline-2.11.jar
> ant-1.9.2.jar
> ant-launcher-1.9.2.jar
> mbassador-1.1.9.jar
> jackson-jaxrs-1.9.2.jar
> jackson-xc-1.9.2.jar
> hadoop-common-2.2.0-tests.jar
>
> Ofcourse, I'll be running some tests do check that dtcli works properly for
> launch+shutdown+kill of apps with only these libraries present in isolation
> without dependency on local m2.
> I'm believe that there are unwanted jars which are used for compile time
> dependency and not runtime in above list which I can drop to keep package
> size minimal.
>
>
> @Anniruddha,
> 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required. Only
> required values I see in it are classpath. and dtcli is capable of building
> it on the fly.
>
> 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar path,
> dt-env.sh and all such conf files. If we need a change in dtcli, then
> community's opinion is required for whether dtcli should change in our repo
> or a copy of that with required changes exist in bigtop repo until we make
> the dtcli generic enough.
>

Can we have these file paths set using environment variables with some
default values in dtcli?
Then we can set such params in dt-env.sh.


> 3. dt-sited.xml -> How is this file different from dt-site.xml? Not sure if
> adding in /etc/skel would help. Files & Dirs from /etc/skel are copied to
> home of new user when useradd program is called. But for existing users
> this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly if not
> encountered for the first time. Again correct me if I'm wrong.
>

Yes, you are right. However this would probably be necessary step for
rpm/deb, as they may also get installed during OS install and before user
accounts were created.


> 4. Changing bashrc & bash_profile -> I'm not sure its the best idea. We'll
> anyway put dtcli in location which is by default present in path i.e.
> /usr/bin.

For other env variables specific to apex, I'll prefer to use dt-env.sh and
> source it in dtcli rather than changing bashrc etc...


Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way to
achieve the same.


>
Thanks,
> Chinmay.
>
>
>
> On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> aniruddha@datatorrent.com> wrote:
>
> > Hi,
> >
> > @Chinmay,
> >
> > We need to consider following:
> > System wide default config files can be located at following locations:
> >
> > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on how it
> finds
> > those locations)
> > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on how it
> > finds those locations)
> > /etc/skel/.dt/dt-sited.xml and other files (for new users that will be
> > created in system in future.)
> >
> > We may also have to modify bashrc / bashprofile for population any
> > variables that are required.
> >
> > @dev, please put in your comments / suggestion.
> >
> >
> >
> > Thanks,
> >
> >
> > Aniruddha
> >
> > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> > > -->
> > >
> > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > Hello Everyone!!
> > > >
> > > > Continuing with packaging effort (rpm+deb) of apex, here are some
> > > proposals
> > > > about package structure etc..
> > > > Before posting it on bbigtop mailing list, I have some question for
> > apex
> > > > community.
> > > >
> > > > Proposed Directory structure of apex package for both deb & rpm:
> > > >
> > > > /usr/lib/apex/bin/dtcli
> > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > /usr/lib/apex/lib/<other dependent jars>
> > >
> > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > /usr/share/doc/apex/CHANGELOG
> > > > /usr/share/doc/apex/NOTICE
> > > >
> > > >
> > > > <version> = 3.3.0-incubating.
> > > > <other dependent jars> = All the 3rd party jars which are required
> for
> > > apex
> > > > to run. Usually the dependencies are packaged as part of rpm/deb by
> any
> > > > software in bigtop.
> > > >
> > >
> > > Can you specify what those jars are in Bigtop context? Same as shipped
> > with
> > > DT community addition under lib/ ?
> > >
> > >
> > > > <package LICENSE> = Licenses of corresponding of 3rd party jars which
> > > needs
> > > > to included while packaging.
> > > >
> > > > Questions related to this:
> > > > 1. Should we call the cli of apex as "apex" instead of "dtcli" in
> > bigtop
> > > > package?
> > > >
> > >
> > > I think we should keep the name until we are able to change it in Apex.
> > > Otherwise this may get confusing.
> > >
> > >
> > > > 2. I see that all softwares in bigtop have man page for their
> > > executables.
> > > > I think we should have it too for dtcli. Is there any documentation
> > > which I
> > > > can convert to man page? or can I use output of "dtcli --help"?
> > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > 3.3.0-incubating?
> > > >
> > >
> > > Has to be -incubating
> > >
> > >
> > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1  version of
> > > > bigtop hadoop? Any problems that we see with this dependency?
> > > >
> > >
> > > Should work without changed. I thought we certified against 2.7.0?
> > >
> > >
> > > > 5. Following is the apache mirror from which bigtop will pick the
> apex
> > > > source for compilation and packaging. Please correct if incorrect:
> > > >     http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > >     NOTE: apache.osuosl.org is the mirror used by all the softwares
> in
> > > > bigtop.
> > > >
> > > >
> > > > Please share your opinion.
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Yes.. Starting to work on the packaging.
> > > > >
> > > > > I've already started discussion on bigtop dev mailing list for
> > > > > integration. Also created a Jira for the same. For this communities
> > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> thomas@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Chinmay,
> > > > >>
> > > > >> I don't see anything under prerequisites that would be a problem.
> We
> > > > >> looked
> > > > >> at the ASF licencing compatibility as part of becoming an
> incubator
> > > > >> project.
> > > > >>
> > > > >> Please focus on the packaging during the next weeks. Since the
> work
> > > will
> > > > >> be
> > > > >> part of Bigtop, related discussions and JIRAs should also be
> there.
> > > > >>
> > > > >> Would be good to have the packaging in place by end Feb.
> > > > >>
> > > > >> Thanks,
> > > > >> Thomas
> > > > >>
> > > > >>
> > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > >> chinmay@datatorrent.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi All,
> > > > >> >
> > > > >> > We're planning a work on adding Apache Apex as a component to
> > Apache
> > > > >> > Bigtop.
> > > > >> > Bigtop is the packaging system for the Apache big data
> ecosystem.
> > > > >> Several
> > > > >> > Hadoop distros use it, most recently EMR.
> > > > >> >
> > > > >> > Here is the tracking Jira task in APEXCORE for the same:
> > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > >> >
> > > > >> > Proposed plan of execution is as follows:
> > > > >> > *Step 1) Handle prerequisites*
> > > > >> > Apache bigtop has some hard and soft expectation for new
> > components
> > > to
> > > > >> get
> > > > >> > integrated into Bigtop.
> > > > >> > Here is the list of it:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > >> >
> > > > >> > Most of them seems to be standard ASF based requirements, but
> few
> > > need
> > > > >> to
> > > > >> > be checked for:
> > > > >> > 1. Software projects are expected to be Licensed under Apache
> > > License,
> > > > >> > Version 2.0 (and their dependencies are expected to be
> compatible
> > > with
> > > > >> this
> > > > >> > license)
> > > > >> >     - Apex is under ASL 2.0 but need to check if dependencies of
> > > Apex
> > > > >> are
> > > > >> > compatible with ASL 2.0. This I guess would be a verification
> > check.
> > > > >> > 2. Software projects are expected to be compatible with all of
> the
> > > > >> > supported platforms that Bigtop distribution is targeting
> > > > >> >     - This needs verifying whether our software runs fine in
> > > centos-6
> > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > > > >> > 3. What smoke tests that should be added for deployment.
> > > > >> > 4. Identifying the test artifacts which goes beyond smoke test
> > > > >> >     - These are basically the integration tests for verification
> > > after
> > > > >> the
> > > > >> > deployment. This is a soft requirement, but aim is to achieve
> this
> > > as
> > > > >> well
> > > > >> > or at least have explanation why not to include.
> > > > >> >
> > > > >> > If there are any from the link which explicitly needs to be
> > checked
> > > > >> other
> > > > >> > than above 4, please let us know.
> > > > >> >
> > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > >> > From one of the mail archive of Bigtop, it was learnt that the
> > > bigtop
> > > > >> > community want to see the addition of new components in phases.
> > Here
> > > > are
> > > > >> > the phases:
> > > > >> > 1. Packaging
> > > > >> >     - This needs creating of package i.e. rpm & deb files.
> > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if
> any
> > > > >> needed.
> > > > >> >     - Any documentation that need to be added to distribution of
> > our
> > > > >> > software.
> > > > >> >     - Any license information of dependencies required to be
> added
> > > to
> > > > >> > package
> > > > >> > 2. Smoke tests (at very least)
> > > > >> >     - Adding smoke test for packaging.
> > > > >> > 3. Puppet recipes for automatic deployment and configuration
> > > > >> >     - Add puppet recipes for automatic deployment
> > > > >> > 4. Integration tests
> > > > >> >     - For verification of deployments.
> > > > >> > 5. license clearance:
> > > > >> >     Run 'gradle rat' to make sure all new stuff is compliant
> with
> > > ASF
> > > > >> > license requirements. If you add code licenses under different
> > > > licenses,
> > > > >> > those would need to be listed in the NOTICE.
> > > > >> >
> > > > >> > Please share your thoughts on the approach.
> > > > >> > We'll start corresponding communication on bigtop mailing list
> as
> > > > well.
> > > > >> >
> > > > >> > We have some specific questions/suggestions related to what
> should
> > > be
> > > > >> the
> > > > >> > content of the package and what should be the smoke tests, but
> in
> > > the
> > > > >> > interest of not having too much content here, we'll put the
> > > questions
> > > > >> as a
> > > > >> > separate mail in this mailthread.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Chinmay.
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@datatorrent.com>]

@Thomas, Not all the jar present in DT community edition will be included
there.

Here is the list of jars I found from common dependencies between of apex
and DT Community edition:
netlet-1.2.0.jar
kryo-2.24.0.jar
jackson-core-asl-1.9.2.jar
jackson-mapper-asl-1.9.2.jar
async-http-client-1.7.20.jar
netty-3.6.6.Final.jar
validation-api-1.1.0.Final.jar
bval-jsr303-0.5.jar
bval-core-0.5.jar
commons-lang3-3.1.jar
commons-beanutils-1.8.3.jar
httpclient-4.3.5.jar
commons-codec-1.10.jar
zip4j-1.3.2.jar
jetty-websocket-8.1.10.v20130312.jar
xbean-asm5-shaded-4.3.jar
jersey-apache-client4-1.9.jar
jline-2.11.jar
ant-1.9.2.jar
ant-launcher-1.9.2.jar
mbassador-1.1.9.jar
jackson-jaxrs-1.9.2.jar
jackson-xc-1.9.2.jar
hadoop-common-2.2.0-tests.jar

Ofcourse, I'll be running some tests do check that dtcli works properly for
launch+shutdown+kill of apps with only these libraries present in isolation
without dependency on local m2.
I'm believe that there are unwanted jars which are used for compile time
dependency and not runtime in above list which I can drop to keep package
size minimal.


@Anniruddha,
1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required. Only
required values I see in it are classpath. and dtcli is capable of building
it on the fly.

2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar path,
dt-env.sh and all such conf files. If we need a change in dtcli, then
community's opinion is required for whether dtcli should change in our repo
or a copy of that with required changes exist in bigtop repo until we make
the dtcli generic enough.

3. dt-sited.xml -> How is this file different from dt-site.xml? Not sure if
adding in /etc/skel would help. Files & Dirs from /etc/skel are copied to
home of new user when useradd program is called. But for existing users
this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly if not
encountered for the first time. Again correct me if I'm wrong.

4. Changing bashrc & bash_profile -> I'm not sure its the best idea. We'll
anyway put dtcli in location which is by default present in path i.e.
/usr/bin.
For other env variables specific to apex, I'll prefer to use dt-env.sh and
source it in dtcli rather than changing bashrc etc...

Thanks,
Chinmay.



On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
aniruddha@datatorrent.com> wrote:

> Hi,
>
> @Chinmay,
>
> We need to consider following:
> System wide default config files can be located at following locations:
>
> /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on how it finds
> those locations)
> /etc/apex/dt-site.xml (we may have to change dtcli behaviour on how it
> finds those locations)
> /etc/skel/.dt/dt-sited.xml and other files (for new users that will be
> created in system in future.)
>
> We may also have to modify bashrc / bashprofile for population any
> variables that are required.
>
> @dev, please put in your comments / suggestion.
>
>
>
> Thanks,
>
>
> Aniruddha
>
> On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
> > -->
> >
> > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> chinmay@datatorrent.com
> > >
> > wrote:
> >
> > > Hello Everyone!!
> > >
> > > Continuing with packaging effort (rpm+deb) of apex, here are some
> > proposals
> > > about package structure etc..
> > > Before posting it on bbigtop mailing list, I have some question for
> apex
> > > community.
> > >
> > > Proposed Directory structure of apex package for both deb & rpm:
> > >
> > > /usr/lib/apex/bin/dtcli
> > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > /usr/lib/apex/lib/<other dependent jars>
> >
> > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > /usr/share/doc/man/man1/dtcli.1.gz
> > > /usr/share/doc/apex/license/LICENSE.txt
> > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > /usr/share/doc/apex/CHANGELOG
> > > /usr/share/doc/apex/NOTICE
> > >
> > >
> > > <version> = 3.3.0-incubating.
> > > <other dependent jars> = All the 3rd party jars which are required for
> > apex
> > > to run. Usually the dependencies are packaged as part of rpm/deb by any
> > > software in bigtop.
> > >
> >
> > Can you specify what those jars are in Bigtop context? Same as shipped
> with
> > DT community addition under lib/ ?
> >
> >
> > > <package LICENSE> = Licenses of corresponding of 3rd party jars which
> > needs
> > > to included while packaging.
> > >
> > > Questions related to this:
> > > 1. Should we call the cli of apex as "apex" instead of "dtcli" in
> bigtop
> > > package?
> > >
> >
> > I think we should keep the name until we are able to change it in Apex.
> > Otherwise this may get confusing.
> >
> >
> > > 2. I see that all softwares in bigtop have man page for their
> > executables.
> > > I think we should have it too for dtcli. Is there any documentation
> > which I
> > > can convert to man page? or can I use output of "dtcli --help"?
> > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > 3.3.0-incubating?
> > >
> >
> > Has to be -incubating
> >
> >
> > > 4. Is it ok for apex package of bigtop to depend on 2.7.1  version of
> > > bigtop hadoop? Any problems that we see with this dependency?
> > >
> >
> > Should work without changed. I thought we certified against 2.7.0?
> >
> >
> > > 5. Following is the apache mirror from which bigtop will pick the apex
> > > source for compilation and packaging. Please correct if incorrect:
> > >     http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > >     NOTE: apache.osuosl.org is the mirror used by all the softwares in
> > > bigtop.
> > >
> > >
> > > Please share your opinion.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com>
> > > wrote:
> > >
> > > > Yes.. Starting to work on the packaging.
> > > >
> > > > I've already started discussion on bigtop dev mailing list for
> > > > integration. Also created a Jira for the same. For this communities
> > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > >
> > > >
> > > >
> > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <thomas@datatorrent.com
> >
> > > > wrote:
> > > >
> > > >> Chinmay,
> > > >>
> > > >> I don't see anything under prerequisites that would be a problem. We
> > > >> looked
> > > >> at the ASF licencing compatibility as part of becoming an incubator
> > > >> project.
> > > >>
> > > >> Please focus on the packaging during the next weeks. Since the work
> > will
> > > >> be
> > > >> part of Bigtop, related discussions and JIRAs should also be there.
> > > >>
> > > >> Would be good to have the packaging in place by end Feb.
> > > >>
> > > >> Thanks,
> > > >> Thomas
> > > >>
> > > >>
> > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > >> chinmay@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >> > Hi All,
> > > >> >
> > > >> > We're planning a work on adding Apache Apex as a component to
> Apache
> > > >> > Bigtop.
> > > >> > Bigtop is the packaging system for the Apache big data ecosystem.
> > > >> Several
> > > >> > Hadoop distros use it, most recently EMR.
> > > >> >
> > > >> > Here is the tracking Jira task in APEXCORE for the same:
> > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > >> >
> > > >> > Proposed plan of execution is as follows:
> > > >> > *Step 1) Handle prerequisites*
> > > >> > Apache bigtop has some hard and soft expectation for new
> components
> > to
> > > >> get
> > > >> > integrated into Bigtop.
> > > >> > Here is the list of it:
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > >> >
> > > >> > Most of them seems to be standard ASF based requirements, but few
> > need
> > > >> to
> > > >> > be checked for:
> > > >> > 1. Software projects are expected to be Licensed under Apache
> > License,
> > > >> > Version 2.0 (and their dependencies are expected to be compatible
> > with
> > > >> this
> > > >> > license)
> > > >> >     - Apex is under ASL 2.0 but need to check if dependencies of
> > Apex
> > > >> are
> > > >> > compatible with ASL 2.0. This I guess would be a verification
> check.
> > > >> > 2. Software projects are expected to be compatible with all of the
> > > >> > supported platforms that Bigtop distribution is targeting
> > > >> >     - This needs verifying whether our software runs fine in
> > centos-6
> > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > > >> > 3. What smoke tests that should be added for deployment.
> > > >> > 4. Identifying the test artifacts which goes beyond smoke test
> > > >> >     - These are basically the integration tests for verification
> > after
> > > >> the
> > > >> > deployment. This is a soft requirement, but aim is to achieve this
> > as
> > > >> well
> > > >> > or at least have explanation why not to include.
> > > >> >
> > > >> > If there are any from the link which explicitly needs to be
> checked
> > > >> other
> > > >> > than above 4, please let us know.
> > > >> >
> > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > >> > From one of the mail archive of Bigtop, it was learnt that the
> > bigtop
> > > >> > community want to see the addition of new components in phases.
> Here
> > > are
> > > >> > the phases:
> > > >> > 1. Packaging
> > > >> >     - This needs creating of package i.e. rpm & deb files.
> > > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if any
> > > >> needed.
> > > >> >     - Any documentation that need to be added to distribution of
> our
> > > >> > software.
> > > >> >     - Any license information of dependencies required to be added
> > to
> > > >> > package
> > > >> > 2. Smoke tests (at very least)
> > > >> >     - Adding smoke test for packaging.
> > > >> > 3. Puppet recipes for automatic deployment and configuration
> > > >> >     - Add puppet recipes for automatic deployment
> > > >> > 4. Integration tests
> > > >> >     - For verification of deployments.
> > > >> > 5. license clearance:
> > > >> >     Run 'gradle rat' to make sure all new stuff is compliant with
> > ASF
> > > >> > license requirements. If you add code licenses under different
> > > licenses,
> > > >> > those would need to be listed in the NOTICE.
> > > >> >
> > > >> > Please share your thoughts on the approach.
> > > >> > We'll start corresponding communication on bigtop mailing list as
> > > well.
> > > >> >
> > > >> > We have some specific questions/suggestions related to what should
> > be
> > > >> the
> > > >> > content of the package and what should be the smoke tests, but in
> > the
> > > >> > interest of not having too much content here, we'll put the
> > questions
> > > >> as a
> > > >> > separate mail in this mailthread.
> > > >> >
> > > >> > Thanks,
> > > >> > Chinmay.
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Aniruddha Thombare <an...@datatorrent.com>]

Hi,

@Chinmay,

We need to consider following:
System wide default config files can be located at following locations:

/etc/apex/dt-env.sh  (we may have to change dtcli behaviour on how it finds
those locations)
/etc/apex/dt-site.xml (we may have to change dtcli behaviour on how it
finds those locations)
/etc/skel/.dt/dt-sited.xml and other files (for new users that will be
created in system in future.)

We may also have to modify bashrc / bashprofile for population any
variables that are required.

@dev, please put in your comments / suggestion.



Thanks,


Aniruddha

On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <th...@datatorrent.com>
wrote:

> -->
>
> On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <chinmay@datatorrent.com
> >
> wrote:
>
> > Hello Everyone!!
> >
> > Continuing with packaging effort (rpm+deb) of apex, here are some
> proposals
> > about package structure etc..
> > Before posting it on bbigtop mailing list, I have some question for apex
> > community.
> >
> > Proposed Directory structure of apex package for both deb & rpm:
> >
> > /usr/lib/apex/bin/dtcli
> > /usr/lib/apex/lib/apex-api-<version>.jar
> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > /usr/lib/apex/lib/apex-common-<version>.jar
> > /usr/lib/apex/lib/<other dependent jars>
>
> /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > /usr/share/doc/man/man1/dtcli.1.gz
> > /usr/share/doc/apex/license/LICENSE.txt
> > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > /usr/share/doc/apex/CHANGELOG
> > /usr/share/doc/apex/NOTICE
> >
> >
> > <version> = 3.3.0-incubating.
> > <other dependent jars> = All the 3rd party jars which are required for
> apex
> > to run. Usually the dependencies are packaged as part of rpm/deb by any
> > software in bigtop.
> >
>
> Can you specify what those jars are in Bigtop context? Same as shipped with
> DT community addition under lib/ ?
>
>
> > <package LICENSE> = Licenses of corresponding of 3rd party jars which
> needs
> > to included while packaging.
> >
> > Questions related to this:
> > 1. Should we call the cli of apex as "apex" instead of "dtcli" in bigtop
> > package?
> >
>
> I think we should keep the name until we are able to change it in Apex.
> Otherwise this may get confusing.
>
>
> > 2. I see that all softwares in bigtop have man page for their
> executables.
> > I think we should have it too for dtcli. Is there any documentation
> which I
> > can convert to man page? or can I use output of "dtcli --help"?
> > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > 3.3.0-incubating?
> >
>
> Has to be -incubating
>
>
> > 4. Is it ok for apex package of bigtop to depend on 2.7.1  version of
> > bigtop hadoop? Any problems that we see with this dependency?
> >
>
> Should work without changed. I thought we certified against 2.7.0?
>
>
> > 5. Following is the apache mirror from which bigtop will pick the apex
> > source for compilation and packaging. Please correct if incorrect:
> >     http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >     NOTE: apache.osuosl.org is the mirror used by all the softwares in
> > bigtop.
> >
> >
> > Please share your opinion.
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> >
> >
> > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com>
> > wrote:
> >
> > > Yes.. Starting to work on the packaging.
> > >
> > > I've already started discussion on bigtop dev mailing list for
> > > integration. Also created a Jira for the same. For this communities
> > > reference, here is the bigtop Jira: BIGTOP-2313.
> > >
> > >
> > >
> > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <th...@datatorrent.com>
> > > wrote:
> > >
> > >> Chinmay,
> > >>
> > >> I don't see anything under prerequisites that would be a problem. We
> > >> looked
> > >> at the ASF licencing compatibility as part of becoming an incubator
> > >> project.
> > >>
> > >> Please focus on the packaging during the next weeks. Since the work
> will
> > >> be
> > >> part of Bigtop, related discussions and JIRAs should also be there.
> > >>
> > >> Would be good to have the packaging in place by end Feb.
> > >>
> > >> Thanks,
> > >> Thomas
> > >>
> > >>
> > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > >> chinmay@datatorrent.com>
> > >> wrote:
> > >>
> > >> > Hi All,
> > >> >
> > >> > We're planning a work on adding Apache Apex as a component to Apache
> > >> > Bigtop.
> > >> > Bigtop is the packaging system for the Apache big data ecosystem.
> > >> Several
> > >> > Hadoop distros use it, most recently EMR.
> > >> >
> > >> > Here is the tracking Jira task in APEXCORE for the same:
> > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > >> >
> > >> > Proposed plan of execution is as follows:
> > >> > *Step 1) Handle prerequisites*
> > >> > Apache bigtop has some hard and soft expectation for new components
> to
> > >> get
> > >> > integrated into Bigtop.
> > >> > Here is the list of it:
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > >> >
> > >> > Most of them seems to be standard ASF based requirements, but few
> need
> > >> to
> > >> > be checked for:
> > >> > 1. Software projects are expected to be Licensed under Apache
> License,
> > >> > Version 2.0 (and their dependencies are expected to be compatible
> with
> > >> this
> > >> > license)
> > >> >     - Apex is under ASL 2.0 but need to check if dependencies of
> Apex
> > >> are
> > >> > compatible with ASL 2.0. This I guess would be a verification check.
> > >> > 2. Software projects are expected to be compatible with all of the
> > >> > supported platforms that Bigtop distribution is targeting
> > >> >     - This needs verifying whether our software runs fine in
> centos-6
> > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > >> > 3. What smoke tests that should be added for deployment.
> > >> > 4. Identifying the test artifacts which goes beyond smoke test
> > >> >     - These are basically the integration tests for verification
> after
> > >> the
> > >> > deployment. This is a soft requirement, but aim is to achieve this
> as
> > >> well
> > >> > or at least have explanation why not to include.
> > >> >
> > >> > If there are any from the link which explicitly needs to be checked
> > >> other
> > >> > than above 4, please let us know.
> > >> >
> > >> > *Step 2) Adding Apex as component to Bigtop*
> > >> > From one of the mail archive of Bigtop, it was learnt that the
> bigtop
> > >> > community want to see the addition of new components in phases. Here
> > are
> > >> > the phases:
> > >> > 1. Packaging
> > >> >     - This needs creating of package i.e. rpm & deb files.
> > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if any
> > >> needed.
> > >> >     - Any documentation that need to be added to distribution of our
> > >> > software.
> > >> >     - Any license information of dependencies required to be added
> to
> > >> > package
> > >> > 2. Smoke tests (at very least)
> > >> >     - Adding smoke test for packaging.
> > >> > 3. Puppet recipes for automatic deployment and configuration
> > >> >     - Add puppet recipes for automatic deployment
> > >> > 4. Integration tests
> > >> >     - For verification of deployments.
> > >> > 5. license clearance:
> > >> >     Run 'gradle rat' to make sure all new stuff is compliant with
> ASF
> > >> > license requirements. If you add code licenses under different
> > licenses,
> > >> > those would need to be listed in the NOTICE.
> > >> >
> > >> > Please share your thoughts on the approach.
> > >> > We'll start corresponding communication on bigtop mailing list as
> > well.
> > >> >
> > >> > We have some specific questions/suggestions related to what should
> be
> > >> the
> > >> > content of the package and what should be the smoke tests, but in
> the
> > >> > interest of not having too much content here, we'll put the
> questions
> > >> as a
> > >> > separate mail in this mailthread.
> > >> >
> > >> > Thanks,
> > >> > Chinmay.
> > >> >
> > >>
> > >
> > >
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

-->

On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> Hello Everyone!!
>
> Continuing with packaging effort (rpm+deb) of apex, here are some proposals
> about package structure etc..
> Before posting it on bbigtop mailing list, I have some question for apex
> community.
>
> Proposed Directory structure of apex package for both deb & rpm:
>
> /usr/lib/apex/bin/dtcli
> /usr/lib/apex/lib/apex-api-<version>.jar
> /usr/lib/apex/lib/apex-engine-<version>.jar
> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> /usr/lib/apex/lib/apex-common-<version>.jar
> /usr/lib/apex/lib/<other dependent jars>

/usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> /usr/share/doc/man/man1/dtcli.1.gz
> /usr/share/doc/apex/license/LICENSE.txt
> /usr/share/doc/apex/license/<package>-LICENSE.txt
> /usr/share/doc/apex/CHANGELOG
> /usr/share/doc/apex/NOTICE
>
>
> <version> = 3.3.0-incubating.
> <other dependent jars> = All the 3rd party jars which are required for apex
> to run. Usually the dependencies are packaged as part of rpm/deb by any
> software in bigtop.
>

Can you specify what those jars are in Bigtop context? Same as shipped with
DT community addition under lib/ ?


> <package LICENSE> = Licenses of corresponding of 3rd party jars which needs
> to included while packaging.
>
> Questions related to this:
> 1. Should we call the cli of apex as "apex" instead of "dtcli" in bigtop
> package?
>

I think we should keep the name until we are able to change it in Apex.
Otherwise this may get confusing.


> 2. I see that all softwares in bigtop have man page for their executables.
> I think we should have it too for dtcli. Is there any documentation which I
> can convert to man page? or can I use output of "dtcli --help"?
> 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> 3.3.0-incubating?
>

Has to be -incubating


> 4. Is it ok for apex package of bigtop to depend on 2.7.1  version of
> bigtop hadoop? Any problems that we see with this dependency?
>

Should work without changed. I thought we certified against 2.7.0?


> 5. Following is the apache mirror from which bigtop will pick the apex
> source for compilation and packaging. Please correct if incorrect:
>     http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>     NOTE: apache.osuosl.org is the mirror used by all the softwares in
> bigtop.
>
>
> Please share your opinion.
>
> Thanks,
> Chinmay.
>
>
>
>
>
> On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > Yes.. Starting to work on the packaging.
> >
> > I've already started discussion on bigtop dev mailing list for
> > integration. Also created a Jira for the same. For this communities
> > reference, here is the bigtop Jira: BIGTOP-2313.
> >
> >
> >
> > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> >> Chinmay,
> >>
> >> I don't see anything under prerequisites that would be a problem. We
> >> looked
> >> at the ASF licencing compatibility as part of becoming an incubator
> >> project.
> >>
> >> Please focus on the packaging during the next weeks. Since the work will
> >> be
> >> part of Bigtop, related discussions and JIRAs should also be there.
> >>
> >> Would be good to have the packaging in place by end Feb.
> >>
> >> Thanks,
> >> Thomas
> >>
> >>
> >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> >> chinmay@datatorrent.com>
> >> wrote:
> >>
> >> > Hi All,
> >> >
> >> > We're planning a work on adding Apache Apex as a component to Apache
> >> > Bigtop.
> >> > Bigtop is the packaging system for the Apache big data ecosystem.
> >> Several
> >> > Hadoop distros use it, most recently EMR.
> >> >
> >> > Here is the tracking Jira task in APEXCORE for the same:
> >> > https://issues.apache.org/jira/browse/APEXCORE-331
> >> >
> >> > Proposed plan of execution is as follows:
> >> > *Step 1) Handle prerequisites*
> >> > Apache bigtop has some hard and soft expectation for new components to
> >> get
> >> > integrated into Bigtop.
> >> > Here is the list of it:
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> >
> >> > Most of them seems to be standard ASF based requirements, but few need
> >> to
> >> > be checked for:
> >> > 1. Software projects are expected to be Licensed under Apache License,
> >> > Version 2.0 (and their dependencies are expected to be compatible with
> >> this
> >> > license)
> >> >     - Apex is under ASL 2.0 but need to check if dependencies of Apex
> >> are
> >> > compatible with ASL 2.0. This I guess would be a verification check.
> >> > 2. Software projects are expected to be compatible with all of the
> >> > supported platforms that Bigtop distribution is targeting
> >> >     - This needs verifying whether our software runs fine in centos-6
> >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> >> > 3. What smoke tests that should be added for deployment.
> >> > 4. Identifying the test artifacts which goes beyond smoke test
> >> >     - These are basically the integration tests for verification after
> >> the
> >> > deployment. This is a soft requirement, but aim is to achieve this as
> >> well
> >> > or at least have explanation why not to include.
> >> >
> >> > If there are any from the link which explicitly needs to be checked
> >> other
> >> > than above 4, please let us know.
> >> >
> >> > *Step 2) Adding Apex as component to Bigtop*
> >> > From one of the mail archive of Bigtop, it was learnt that the bigtop
> >> > community want to see the addition of new components in phases. Here
> are
> >> > the phases:
> >> > 1. Packaging
> >> >     - This needs creating of package i.e. rpm & deb files.
> >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if any
> >> needed.
> >> >     - Any documentation that need to be added to distribution of our
> >> > software.
> >> >     - Any license information of dependencies required to be added to
> >> > package
> >> > 2. Smoke tests (at very least)
> >> >     - Adding smoke test for packaging.
> >> > 3. Puppet recipes for automatic deployment and configuration
> >> >     - Add puppet recipes for automatic deployment
> >> > 4. Integration tests
> >> >     - For verification of deployments.
> >> > 5. license clearance:
> >> >     Run 'gradle rat' to make sure all new stuff is compliant with ASF
> >> > license requirements. If you add code licenses under different
> licenses,
> >> > those would need to be listed in the NOTICE.
> >> >
> >> > Please share your thoughts on the approach.
> >> > We'll start corresponding communication on bigtop mailing list as
> well.
> >> >
> >> > We have some specific questions/suggestions related to what should be
> >> the
> >> > content of the package and what should be the smoke tests, but in the
> >> > interest of not having too much content here, we'll put the questions
> >> as a
> >> > separate mail in this mailthread.
> >> >
> >> > Thanks,
> >> > Chinmay.
> >> >
> >>
> >
> >
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@datatorrent.com>]

Hello Everyone!!

Continuing with packaging effort (rpm+deb) of apex, here are some proposals
about package structure etc..
Before posting it on bbigtop mailing list, I have some question for apex
community.

Proposed Directory structure of apex package for both deb & rpm:

/usr/lib/apex/bin/dtcli
/usr/lib/apex/lib/apex-api-<version>.jar
/usr/lib/apex/lib/apex-engine-<version>.jar
/usr/lib/apex/lib/apex-bufferserver-<version>.jar
/usr/lib/apex/lib/apex-common-<version>.jar
/usr/lib/apex/lib/<other dependent jars>
/usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
/usr/share/doc/man/man1/dtcli.1.gz
/usr/share/doc/apex/license/LICENSE.txt
/usr/share/doc/apex/license/<package>-LICENSE.txt
/usr/share/doc/apex/CHANGELOG
/usr/share/doc/apex/NOTICE


<version> = 3.3.0-incubating.
<other dependent jars> = All the 3rd party jars which are required for apex
to run. Usually the dependencies are packaged as part of rpm/deb by any
software in bigtop.
<package LICENSE> = Licenses of corresponding of 3rd party jars which needs
to included while packaging.

Questions related to this:
1. Should we call the cli of apex as "apex" instead of "dtcli" in bigtop
package?
2. I see that all softwares in bigtop have man page for their executables.
I think we should have it too for dtcli. Is there any documentation which I
can convert to man page? or can I use output of "dtcli --help"?
3. Do we want to call version of apex in Bigtop as 3.3.0 OR
3.3.0-incubating?
4. Is it ok for apex package of bigtop to depend on 2.7.1  version of
bigtop hadoop? Any problems that we see with this dependency?
5. Following is the apache mirror from which bigtop will pick the apex
source for compilation and packaging. Please correct if incorrect:
    http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
    NOTE: apache.osuosl.org is the mirror used by all the softwares in
bigtop.


Please share your opinion.

Thanks,
Chinmay.





On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> Yes.. Starting to work on the packaging.
>
> I've already started discussion on bigtop dev mailing list for
> integration. Also created a Jira for the same. For this communities
> reference, here is the bigtop Jira: BIGTOP-2313.
>
>
>
> On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
>> Chinmay,
>>
>> I don't see anything under prerequisites that would be a problem. We
>> looked
>> at the ASF licencing compatibility as part of becoming an incubator
>> project.
>>
>> Please focus on the packaging during the next weeks. Since the work will
>> be
>> part of Bigtop, related discussions and JIRAs should also be there.
>>
>> Would be good to have the packaging in place by end Feb.
>>
>> Thanks,
>> Thomas
>>
>>
>> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
>> chinmay@datatorrent.com>
>> wrote:
>>
>> > Hi All,
>> >
>> > We're planning a work on adding Apache Apex as a component to Apache
>> > Bigtop.
>> > Bigtop is the packaging system for the Apache big data ecosystem.
>> Several
>> > Hadoop distros use it, most recently EMR.
>> >
>> > Here is the tracking Jira task in APEXCORE for the same:
>> > https://issues.apache.org/jira/browse/APEXCORE-331
>> >
>> > Proposed plan of execution is as follows:
>> > *Step 1) Handle prerequisites*
>> > Apache bigtop has some hard and soft expectation for new components to
>> get
>> > integrated into Bigtop.
>> > Here is the list of it:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> >
>> > Most of them seems to be standard ASF based requirements, but few need
>> to
>> > be checked for:
>> > 1. Software projects are expected to be Licensed under Apache License,
>> > Version 2.0 (and their dependencies are expected to be compatible with
>> this
>> > license)
>> >     - Apex is under ASL 2.0 but need to check if dependencies of Apex
>> are
>> > compatible with ASL 2.0. This I guess would be a verification check.
>> > 2. Software projects are expected to be compatible with all of the
>> > supported platforms that Bigtop distribution is targeting
>> >     - This needs verifying whether our software runs fine in centos-6
>> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
>> > 3. What smoke tests that should be added for deployment.
>> > 4. Identifying the test artifacts which goes beyond smoke test
>> >     - These are basically the integration tests for verification after
>> the
>> > deployment. This is a soft requirement, but aim is to achieve this as
>> well
>> > or at least have explanation why not to include.
>> >
>> > If there are any from the link which explicitly needs to be checked
>> other
>> > than above 4, please let us know.
>> >
>> > *Step 2) Adding Apex as component to Bigtop*
>> > From one of the mail archive of Bigtop, it was learnt that the bigtop
>> > community want to see the addition of new components in phases. Here are
>> > the phases:
>> > 1. Packaging
>> >     - This needs creating of package i.e. rpm & deb files.
>> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if any
>> needed.
>> >     - Any documentation that need to be added to distribution of our
>> > software.
>> >     - Any license information of dependencies required to be added to
>> > package
>> > 2. Smoke tests (at very least)
>> >     - Adding smoke test for packaging.
>> > 3. Puppet recipes for automatic deployment and configuration
>> >     - Add puppet recipes for automatic deployment
>> > 4. Integration tests
>> >     - For verification of deployments.
>> > 5. license clearance:
>> >     Run 'gradle rat' to make sure all new stuff is compliant with ASF
>> > license requirements. If you add code licenses under different licenses,
>> > those would need to be listed in the NOTICE.
>> >
>> > Please share your thoughts on the approach.
>> > We'll start corresponding communication on bigtop mailing list as well.
>> >
>> > We have some specific questions/suggestions related to what should be
>> the
>> > content of the package and what should be the smoke tests, but in the
>> > interest of not having too much content here, we'll put the questions
>> as a
>> > separate mail in this mailthread.
>> >
>> > Thanks,
>> > Chinmay.
>> >
>>
>
>

Re: Apache Apex as component in Apache bigtop

[Chinmay Kolhatkar <ch...@datatorrent.com>]

Yes.. Starting to work on the packaging.

I've already started discussion on bigtop dev mailing list for integration.
Also created a Jira for the same. For this communities reference, here is
the bigtop Jira: BIGTOP-2313.



On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <th...@datatorrent.com> wrote:

> Chinmay,
>
> I don't see anything under prerequisites that would be a problem. We looked
> at the ASF licencing compatibility as part of becoming an incubator
> project.
>
> Please focus on the packaging during the next weeks. Since the work will be
> part of Bigtop, related discussions and JIRAs should also be there.
>
> Would be good to have the packaging in place by end Feb.
>
> Thanks,
> Thomas
>
>
> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > Hi All,
> >
> > We're planning a work on adding Apache Apex as a component to Apache
> > Bigtop.
> > Bigtop is the packaging system for the Apache big data ecosystem. Several
> > Hadoop distros use it, most recently EMR.
> >
> > Here is the tracking Jira task in APEXCORE for the same:
> > https://issues.apache.org/jira/browse/APEXCORE-331
> >
> > Proposed plan of execution is as follows:
> > *Step 1) Handle prerequisites*
> > Apache bigtop has some hard and soft expectation for new components to
> get
> > integrated into Bigtop.
> > Here is the list of it:
> >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >
> > Most of them seems to be standard ASF based requirements, but few need to
> > be checked for:
> > 1. Software projects are expected to be Licensed under Apache License,
> > Version 2.0 (and their dependencies are expected to be compatible with
> this
> > license)
> >     - Apex is under ASL 2.0 but need to check if dependencies of Apex are
> > compatible with ASL 2.0. This I guess would be a verification check.
> > 2. Software projects are expected to be compatible with all of the
> > supported platforms that Bigtop distribution is targeting
> >     - This needs verifying whether our software runs fine in centos-6
> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > 3. What smoke tests that should be added for deployment.
> > 4. Identifying the test artifacts which goes beyond smoke test
> >     - These are basically the integration tests for verification after
> the
> > deployment. This is a soft requirement, but aim is to achieve this as
> well
> > or at least have explanation why not to include.
> >
> > If there are any from the link which explicitly needs to be checked other
> > than above 4, please let us know.
> >
> > *Step 2) Adding Apex as component to Bigtop*
> > From one of the mail archive of Bigtop, it was learnt that the bigtop
> > community want to see the addition of new components in phases. Here are
> > the phases:
> > 1. Packaging
> >     - This needs creating of package i.e. rpm & deb files.
> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if any
> needed.
> >     - Any documentation that need to be added to distribution of our
> > software.
> >     - Any license information of dependencies required to be added to
> > package
> > 2. Smoke tests (at very least)
> >     - Adding smoke test for packaging.
> > 3. Puppet recipes for automatic deployment and configuration
> >     - Add puppet recipes for automatic deployment
> > 4. Integration tests
> >     - For verification of deployments.
> > 5. license clearance:
> >     Run 'gradle rat' to make sure all new stuff is compliant with ASF
> > license requirements. If you add code licenses under different licenses,
> > those would need to be listed in the NOTICE.
> >
> > Please share your thoughts on the approach.
> > We'll start corresponding communication on bigtop mailing list as well.
> >
> > We have some specific questions/suggestions related to what should be the
> > content of the package and what should be the smoke tests, but in the
> > interest of not having too much content here, we'll put the questions as
> a
> > separate mail in this mailthread.
> >
> > Thanks,
> > Chinmay.
> >
>

Re: Apache Apex as component in Apache bigtop

[Thomas Weise <th...@datatorrent.com>]

Chinmay,

I don't see anything under prerequisites that would be a problem. We looked
at the ASF licencing compatibility as part of becoming an incubator project.

Please focus on the packaging during the next weeks. Since the work will be
part of Bigtop, related discussions and JIRAs should also be there.

Would be good to have the packaging in place by end Feb.

Thanks,
Thomas


On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> Hi All,
>
> We're planning a work on adding Apache Apex as a component to Apache
> Bigtop.
> Bigtop is the packaging system for the Apache big data ecosystem. Several
> Hadoop distros use it, most recently EMR.
>
> Here is the tracking Jira task in APEXCORE for the same:
> https://issues.apache.org/jira/browse/APEXCORE-331
>
> Proposed plan of execution is as follows:
> *Step 1) Handle prerequisites*
> Apache bigtop has some hard and soft expectation for new components to get
> integrated into Bigtop.
> Here is the list of it:
>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>
> Most of them seems to be standard ASF based requirements, but few need to
> be checked for:
> 1. Software projects are expected to be Licensed under Apache License,
> Version 2.0 (and their dependencies are expected to be compatible with this
> license)
>     - Apex is under ASL 2.0 but need to check if dependencies of Apex are
> compatible with ASL 2.0. This I guess would be a verification check.
> 2. Software projects are expected to be compatible with all of the
> supported platforms that Bigtop distribution is targeting
>     - This needs verifying whether our software runs fine in centos-6
> centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> 3. What smoke tests that should be added for deployment.
> 4. Identifying the test artifacts which goes beyond smoke test
>     - These are basically the integration tests for verification after the
> deployment. This is a soft requirement, but aim is to achieve this as well
> or at least have explanation why not to include.
>
> If there are any from the link which explicitly needs to be checked other
> than above 4, please let us know.
>
> *Step 2) Adding Apex as component to Bigtop*
> From one of the mail archive of Bigtop, it was learnt that the bigtop
> community want to see the addition of new components in phases. Here are
> the phases:
> 1. Packaging
>     - This needs creating of package i.e. rpm & deb files.
>     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if any needed.
>     - Any documentation that need to be added to distribution of our
> software.
>     - Any license information of dependencies required to be added to
> package
> 2. Smoke tests (at very least)
>     - Adding smoke test for packaging.
> 3. Puppet recipes for automatic deployment and configuration
>     - Add puppet recipes for automatic deployment
> 4. Integration tests
>     - For verification of deployments.
> 5. license clearance:
>     Run 'gradle rat' to make sure all new stuff is compliant with ASF
> license requirements. If you add code licenses under different licenses,
> those would need to be listed in the NOTICE.
>
> Please share your thoughts on the approach.
> We'll start corresponding communication on bigtop mailing list as well.
>
> We have some specific questions/suggestions related to what should be the
> content of the package and what should be the smoke tests, but in the
> interest of not having too much content here, we'll put the questions as a
> separate mail in this mailthread.
>
> Thanks,
> Chinmay.
>