You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@amaterasu.apache.org by Yaniv Rodenski <ya...@shinto.io> on 2018/06/01 07:01:08 UTC
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Thanks Davor,
We will fix those for the next release.
Cheers,
Yaniv
On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci <db...@gmail.com> wrote:
> Great improvement!
>
> license/notice/disclaimer: ok
> headers: ~fine (missing in some non-code files, it would be great to add
> those -- non-blocking)
> binaries: there's a gradle binary, but that may be ruled as acceptable
> (non-blocking from me)
>
> Not sure why we need both .tar and .tar.gz. I'd get rid of one of them.
> Also, shipping private keys in the distribution looks weird as well.
>
> For the first release, this is good. So, +1 to send onwards for an
> Incubator review, and polish a few of these things for a subsequent
> release.
>
> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski <ya...@shinto.io> wrote:
>
> > Hi All,
> >
> > Just a kind reminder for our mentors, we need your vote on this one :)
> >
> > Cheers,
> > Yaniv
> >
> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <ar...@arunma.com>
> wrote:
> >
> > > Thanks a lot, Nadav. Will get home and spend some more time on this. I
> > was
> > > in a rush and did this poor workaround. My VM is just 8 GB.
> > >
> > > Cheers
> > > Arun
> > >
> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com>
> > wrote:
> > >
> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP
> docker
> > on
> > > > AWS). Both Spark-Scala and PySpark.
> > > > It worked on Yaniv's HDP cluster as well.
> > > > Worth noting:
> > > > 1. HDP 2.6.4
> > > > 2. Cluster has total of 32GB memory available
> > > > 3. Each container is allocated 1G memory.
> > > > 4. Amaterasu.properties:
> > > >
> > > > zk=sandbox-hdp.hortonworks.com
> > > > version=0.2.0-incubating-rc3
> > > > master=192.168.33.11
> > > > user=root
> > > > mode=yarn
> > > > webserver.port=8000
> > > > webserver.root=dist
> > > > spark.version=2.6.4.0-91
> > > > yarn.queue=default
> > > > yarn.jarspath=hdfs:///apps/amaterasu
> > > > spark.home=/usr/hdp/current/spark2-client
> > > >
> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.
> > cloudera2-1.cdh5.7.0.p0.
> > > 171658/lib/spark2
> > > > yarn.hadoop.home.dir=/etc/hadoop
> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > > >
> > > >
> > > > Arun, please share:
> > > > 1. YARN memory configurations
> > > > 2. amaterasu.properties content
> > > > 3. HDP version.
> > > >
> > > > Cheers,
> > > > Nadav
> > > >
> > > >
> > > > On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
> > > >
> > > > > The pmem disabling is just temporary. I'll do a detailed analysis
> and
> > > get
> > > > > back with a proper solution.
> > > > >
> > > > > Any hints on this front is highly appreciated.
> > > > >
> > > > > Cheers
> > > > > Arun
> > > > >
> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <nadavhartzvi@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Yaniv, Eyal, this might be related to the same issue you faced
> with
> > > > HDP.
> > > > > > Can you confirm?
> > > > > >
> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
> > wrote:
> > > > > >
> > > > > > > +1 from me
> > > > > > >
> > > > > > > Unit Tests and Build ran fine.
> > > > > > >
> > > > > > > Tested on HDP (VM) but had trouble allocating containers
> (didn't
> > > have
> > > > > > that
> > > > > > > before). Apparently Centos VMs are known to have this problem.
> > > > > Disabled
> > > > > > > physical memory check (yarn.nodemanager.pmem-check-enabled)
> and
> > > ran
> > > > > jobs
> > > > > > > successfully.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > > > > > kirupagaran.d@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Unit tests passing and build was successful on the branch
> > > > > > > > "version-0.2.0-incubating-rc3"
> > > > > > > >
> > > > > > > > +1 from me
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Kirupa
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
> whisrael@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 looks good to me
> > > > > > > > >
> > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> > > > nadavhartzvi@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 approve. Tested multiple times and after a long round
> of
> > > > > fixing
> > > > > > > and
> > > > > > > > > > testing over and over.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Nadav
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <yaniv@shinto.io
> >
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > We have fixed the legal issues, as well as a bug found
> by
> > > > > @Nadav
> > > > > > > > please
> > > > > > > > > > > review and vote on the release candidate #3 for the
> > version
> > > > > > > > > > > 0.2.0-incubating, as follows
> > > > > > > > > > >
> > > > > > > > > > > [ ] +1, Approve the release
> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
> > specific
> > > > > > > comments)
> > > > > > > > > > >
> > > > > > > > > > > The complete staging area is available for your review,
> > > which
> > > > > > > > includes:
> > > > > > > > > > >
> > > > > > > > > > > * JIRA release notes [1],
> > > > > > > > > > > * the official Apache source release to be deployed to
> > > > > > > > dist.apache.org
> > > > > > > > > > > [2],
> > > > > > > > > > > which is signed with the key with fingerprint [3],
> > > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > > > > > > > > > * Java artifacts were built with Gradle 3.1 and
> > > > OpenJDK/Oracle
> > > > > > JDK
> > > > > > > > > > > 1.8.0_151
> > > > > > > > > > >
> > > > > > > > > > > The vote will be open for at least 72 hours. It is
> > adopted
> > > by
> > > > > > > > majority
> > > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Yaniv
> > > > > > > > > > >
> > > > > > > > > > > [1] https://issues.apache.org/
> > jira/secure/ReleaseNote.jspa
> > > ?
> > > > > > > > > > > projectId=12321521&version=12342793
> > > > > > > > > > > [2] https://dist.apache.org/repos/
> > > > > dist/dev/incubator/amaterasu/
> > > > > > > > > 0.2.0rc3/
> > > > > > > > > > > [3]
> > > > > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/
> KEYS
> > > > > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Yaniv Rodenski
> >
> > +61 477 778 405
> > yaniv@shinto.io
> >
>
--
Yaniv Rodenski
+61 477 778 405
yaniv@shinto.io
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Posted by Davor Bonaci <da...@apache.org>.
>
> Based on what I've seen in other projects, it seems common to release bot
> the binaries (in Amaterasu those are released in the .tar file) as well as
> a source (which are in the -source.tar.gz file)
> The main question is if we do or do not need to release both.
>
Sorry, my mistake. It is perfectly fine to release convenience binaries in
addition to source code. You may want to name them better and refer to both
from the vote email, however ;-)
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Posted by Yaniv Rodenski <ya...@shinto.io>.
Hi Davor,
Sorry for keeping my emails short and undescriptive, I have only one hand
for the next month due to a skateboarding accident.
Based on what I've seen in other projects, it seems common to release bot
the binaries (in Amaterasu those are released in the .tar file) as well as
a source (which are in the -source.tar.gz file)
The main question is if we do or do not need to release both.
Cheers,
Yaniv
On Sat, Jun 2, 2018 at 1:13 AM, Davor Bonaci <db...@gmail.com> wrote:
> No problem with .tar.gz. But, you don't need .tar then.
>
> On Fri, Jun 1, 2018 at 12:04 AM, Yaniv Rodenski <ya...@shinto.io> wrote:
>
> > One thing to clarify, the tar.gz is the source code as I've seen this is
> > released in other projects. if that is not needed I'm happy to drop it
> from
> > future releases.
> >
> > On Fri, Jun 1, 2018 at 5:01 PM, Yaniv Rodenski <ya...@shinto.io> wrote:
> >
> > > Thanks Davor,
> > >
> > > We will fix those for the next release.
> > >
> > > Cheers,
> > > Yaniv
> > >
> > > On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci <db...@gmail.com>
> wrote:
> > >
> > >> Great improvement!
> > >>
> > >> license/notice/disclaimer: ok
> > >> headers: ~fine (missing in some non-code files, it would be great to
> add
> > >> those -- non-blocking)
> > >> binaries: there's a gradle binary, but that may be ruled as acceptable
> > >> (non-blocking from me)
> > >>
> > >> Not sure why we need both .tar and .tar.gz. I'd get rid of one of
> them.
> > >> Also, shipping private keys in the distribution looks weird as well.
> > >>
> > >> For the first release, this is good. So, +1 to send onwards for an
> > >> Incubator review, and polish a few of these things for a subsequent
> > >> release.
> > >>
> > >> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski <ya...@shinto.io>
> > wrote:
> > >>
> > >> > Hi All,
> > >> >
> > >> > Just a kind reminder for our mentors, we need your vote on this one
> :)
> > >> >
> > >> > Cheers,
> > >> > Yaniv
> > >> >
> > >> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <ar...@arunma.com>
> > >> wrote:
> > >> >
> > >> > > Thanks a lot, Nadav. Will get home and spend some more time on
> > this. I
> > >> > was
> > >> > > in a rush and did this poor workaround. My VM is just 8 GB.
> > >> > >
> > >> > > Cheers
> > >> > > Arun
> > >> > >
> > >> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <
> nadavhartzvi@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP
> > >> docker
> > >> > on
> > >> > > > AWS). Both Spark-Scala and PySpark.
> > >> > > > It worked on Yaniv's HDP cluster as well.
> > >> > > > Worth noting:
> > >> > > > 1. HDP 2.6.4
> > >> > > > 2. Cluster has total of 32GB memory available
> > >> > > > 3. Each container is allocated 1G memory.
> > >> > > > 4. Amaterasu.properties:
> > >> > > >
> > >> > > > zk=sandbox-hdp.hortonworks.com
> > >> > > > version=0.2.0-incubating-rc3
> > >> > > > master=192.168.33.11
> > >> > > > user=root
> > >> > > > mode=yarn
> > >> > > > webserver.port=8000
> > >> > > > webserver.root=dist
> > >> > > > spark.version=2.6.4.0-91
> > >> > > > yarn.queue=default
> > >> > > > yarn.jarspath=hdfs:///apps/amaterasu
> > >> > > > spark.home=/usr/hdp/current/spark2-client
> > >> > > >
> > >> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.
> > >> > cloudera2-1.cdh5.7.0.p0.
> > >> > > 171658/lib/spark2
> > >> > > > yarn.hadoop.home.dir=/etc/hadoop
> > >> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.
> > >> 6.4.0-91"
> > >> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.
> > version=2.6.4.0-91"
> > >> > > >
> > >> > > >
> > >> > > > Arun, please share:
> > >> > > > 1. YARN memory configurations
> > >> > > > 2. amaterasu.properties content
> > >> > > > 3. HDP version.
> > >> > > >
> > >> > > > Cheers,
> > >> > > > Nadav
> > >> > > >
> > >> > > >
> > >> > > > On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com>
> wrote:
> > >> > > >
> > >> > > > > The pmem disabling is just temporary. I'll do a detailed
> > analysis
> > >> and
> > >> > > get
> > >> > > > > back with a proper solution.
> > >> > > > >
> > >> > > > > Any hints on this front is highly appreciated.
> > >> > > > >
> > >> > > > > Cheers
> > >> > > > > Arun
> > >> > > > >
> > >> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <
> > >> nadavhartzvi@gmail.com>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > Yaniv, Eyal, this might be related to the same issue you
> faced
> > >> with
> > >> > > > HDP.
> > >> > > > > > Can you confirm?
> > >> > > > > >
> > >> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <
> arun@arunma.com>
> > >> > wrote:
> > >> > > > > >
> > >> > > > > > > +1 from me
> > >> > > > > > >
> > >> > > > > > > Unit Tests and Build ran fine.
> > >> > > > > > >
> > >> > > > > > > Tested on HDP (VM) but had trouble allocating containers
> > >> (didn't
> > >> > > have
> > >> > > > > > that
> > >> > > > > > > before). Apparently Centos VMs are known to have this
> > >> problem.
> > >> > > > > Disabled
> > >> > > > > > > physical memory check (yarn.nodemanager.pmem-check-
> > enabled)
> > >> and
> > >> > > ran
> > >> > > > > jobs
> > >> > > > > > > successfully.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > >> > > > > > kirupagaran.d@gmail.com
> > >> > > > > > > >
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Unit tests passing and build was successful on the
> branch
> > >> > > > > > > > "version-0.2.0-incubating-rc3"
> > >> > > > > > > >
> > >> > > > > > > > +1 from me
> > >> > > > > > > >
> > >> > > > > > > > Cheers,
> > >> > > > > > > > Kirupa
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
> > >> whisrael@gmail.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > +1 looks good to me
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> > >> > > > nadavhartzvi@gmail.com
> > >> > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > +1 approve. Tested multiple times and after a long
> > >> round of
> > >> > > > > fixing
> > >> > > > > > > and
> > >> > > > > > > > > > testing over and over.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Cheers,
> > >> > > > > > > > > > Nadav
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <
> > >> yaniv@shinto.io>
> > >> > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi everyone,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > We have fixed the legal issues, as well as a bug
> > >> found by
> > >> > > > > @Nadav
> > >> > > > > > > > please
> > >> > > > > > > > > > > review and vote on the release candidate #3 for
> the
> > >> > version
> > >> > > > > > > > > > > 0.2.0-incubating, as follows
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > [ ] +1, Approve the release
> > >> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
> > >> > specific
> > >> > > > > > > comments)
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > The complete staging area is available for your
> > >> review,
> > >> > > which
> > >> > > > > > > > includes:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > * JIRA release notes [1],
> > >> > > > > > > > > > > * the official Apache source release to be
> deployed
> > to
> > >> > > > > > > > dist.apache.org
> > >> > > > > > > > > > > [2],
> > >> > > > > > > > > > > which is signed with the key with fingerprint [3],
> > >> > > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3"
> > [4],
> > >> > > > > > > > > > > * Java artifacts were built with Gradle 3.1 and
> > >> > > > OpenJDK/Oracle
> > >> > > > > > JDK
> > >> > > > > > > > > > > 1.8.0_151
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > The vote will be open for at least 72 hours. It is
> > >> > adopted
> > >> > > by
> > >> > > > > > > > majority
> > >> > > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > Yaniv
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > [1] https://issues.apache.org/
> > >> > jira/secure/ReleaseNote.jspa
> > >> > > ?
> > >> > > > > > > > > > > projectId=12321521&version=12342793
> > >> > > > > > > > > > > [2] https://dist.apache.org/repos/
> > >> > > > > dist/dev/incubator/amaterasu/
> > >> > > > > > > > > 0.2.0rc3/
> > >> > > > > > > > > > > [3]
> > >> > > > > > > https://dist.apache.org/repos/
> > dist/dev/incubator/amaterasu/K
> > >> EYS
> > >> > > > > > > > > > > [4] https://github.com/apache/incu
> > >> bator-amaterasu/tags
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Yaniv Rodenski
> > >> >
> > >> > +61 477 778 405
> > >> > yaniv@shinto.io
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Yaniv Rodenski
> > >
> > > +61 477 778 405
> > > yaniv@shinto.io
> > >
> >
> >
> >
> > --
> > Yaniv Rodenski
> >
> > +61 477 778 405
> > yaniv@shinto.io
> >
>
--
Yaniv Rodenski
+61 477 778 405
yaniv@shinto.io
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Posted by Davor Bonaci <db...@gmail.com>.
No problem with .tar.gz. But, you don't need .tar then.
On Fri, Jun 1, 2018 at 12:04 AM, Yaniv Rodenski <ya...@shinto.io> wrote:
> One thing to clarify, the tar.gz is the source code as I've seen this is
> released in other projects. if that is not needed I'm happy to drop it from
> future releases.
>
> On Fri, Jun 1, 2018 at 5:01 PM, Yaniv Rodenski <ya...@shinto.io> wrote:
>
> > Thanks Davor,
> >
> > We will fix those for the next release.
> >
> > Cheers,
> > Yaniv
> >
> > On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci <db...@gmail.com> wrote:
> >
> >> Great improvement!
> >>
> >> license/notice/disclaimer: ok
> >> headers: ~fine (missing in some non-code files, it would be great to add
> >> those -- non-blocking)
> >> binaries: there's a gradle binary, but that may be ruled as acceptable
> >> (non-blocking from me)
> >>
> >> Not sure why we need both .tar and .tar.gz. I'd get rid of one of them.
> >> Also, shipping private keys in the distribution looks weird as well.
> >>
> >> For the first release, this is good. So, +1 to send onwards for an
> >> Incubator review, and polish a few of these things for a subsequent
> >> release.
> >>
> >> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski <ya...@shinto.io>
> wrote:
> >>
> >> > Hi All,
> >> >
> >> > Just a kind reminder for our mentors, we need your vote on this one :)
> >> >
> >> > Cheers,
> >> > Yaniv
> >> >
> >> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <ar...@arunma.com>
> >> wrote:
> >> >
> >> > > Thanks a lot, Nadav. Will get home and spend some more time on
> this. I
> >> > was
> >> > > in a rush and did this poor workaround. My VM is just 8 GB.
> >> > >
> >> > > Cheers
> >> > > Arun
> >> > >
> >> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP
> >> docker
> >> > on
> >> > > > AWS). Both Spark-Scala and PySpark.
> >> > > > It worked on Yaniv's HDP cluster as well.
> >> > > > Worth noting:
> >> > > > 1. HDP 2.6.4
> >> > > > 2. Cluster has total of 32GB memory available
> >> > > > 3. Each container is allocated 1G memory.
> >> > > > 4. Amaterasu.properties:
> >> > > >
> >> > > > zk=sandbox-hdp.hortonworks.com
> >> > > > version=0.2.0-incubating-rc3
> >> > > > master=192.168.33.11
> >> > > > user=root
> >> > > > mode=yarn
> >> > > > webserver.port=8000
> >> > > > webserver.root=dist
> >> > > > spark.version=2.6.4.0-91
> >> > > > yarn.queue=default
> >> > > > yarn.jarspath=hdfs:///apps/amaterasu
> >> > > > spark.home=/usr/hdp/current/spark2-client
> >> > > >
> >> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.
> >> > cloudera2-1.cdh5.7.0.p0.
> >> > > 171658/lib/spark2
> >> > > > yarn.hadoop.home.dir=/etc/hadoop
> >> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.
> >> 6.4.0-91"
> >> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.
> version=2.6.4.0-91"
> >> > > >
> >> > > >
> >> > > > Arun, please share:
> >> > > > 1. YARN memory configurations
> >> > > > 2. amaterasu.properties content
> >> > > > 3. HDP version.
> >> > > >
> >> > > > Cheers,
> >> > > > Nadav
> >> > > >
> >> > > >
> >> > > > On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
> >> > > >
> >> > > > > The pmem disabling is just temporary. I'll do a detailed
> analysis
> >> and
> >> > > get
> >> > > > > back with a proper solution.
> >> > > > >
> >> > > > > Any hints on this front is highly appreciated.
> >> > > > >
> >> > > > > Cheers
> >> > > > > Arun
> >> > > > >
> >> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <
> >> nadavhartzvi@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > > > Yaniv, Eyal, this might be related to the same issue you faced
> >> with
> >> > > > HDP.
> >> > > > > > Can you confirm?
> >> > > > > >
> >> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
> >> > wrote:
> >> > > > > >
> >> > > > > > > +1 from me
> >> > > > > > >
> >> > > > > > > Unit Tests and Build ran fine.
> >> > > > > > >
> >> > > > > > > Tested on HDP (VM) but had trouble allocating containers
> >> (didn't
> >> > > have
> >> > > > > > that
> >> > > > > > > before). Apparently Centos VMs are known to have this
> >> problem.
> >> > > > > Disabled
> >> > > > > > > physical memory check (yarn.nodemanager.pmem-check-
> enabled)
> >> and
> >> > > ran
> >> > > > > jobs
> >> > > > > > > successfully.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> >> > > > > > kirupagaran.d@gmail.com
> >> > > > > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Unit tests passing and build was successful on the branch
> >> > > > > > > > "version-0.2.0-incubating-rc3"
> >> > > > > > > >
> >> > > > > > > > +1 from me
> >> > > > > > > >
> >> > > > > > > > Cheers,
> >> > > > > > > > Kirupa
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
> >> whisrael@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > +1 looks good to me
> >> > > > > > > > >
> >> > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> >> > > > nadavhartzvi@gmail.com
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > +1 approve. Tested multiple times and after a long
> >> round of
> >> > > > > fixing
> >> > > > > > > and
> >> > > > > > > > > > testing over and over.
> >> > > > > > > > > >
> >> > > > > > > > > > Cheers,
> >> > > > > > > > > > Nadav
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <
> >> yaniv@shinto.io>
> >> > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi everyone,
> >> > > > > > > > > > >
> >> > > > > > > > > > > We have fixed the legal issues, as well as a bug
> >> found by
> >> > > > > @Nadav
> >> > > > > > > > please
> >> > > > > > > > > > > review and vote on the release candidate #3 for the
> >> > version
> >> > > > > > > > > > > 0.2.0-incubating, as follows
> >> > > > > > > > > > >
> >> > > > > > > > > > > [ ] +1, Approve the release
> >> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
> >> > specific
> >> > > > > > > comments)
> >> > > > > > > > > > >
> >> > > > > > > > > > > The complete staging area is available for your
> >> review,
> >> > > which
> >> > > > > > > > includes:
> >> > > > > > > > > > >
> >> > > > > > > > > > > * JIRA release notes [1],
> >> > > > > > > > > > > * the official Apache source release to be deployed
> to
> >> > > > > > > > dist.apache.org
> >> > > > > > > > > > > [2],
> >> > > > > > > > > > > which is signed with the key with fingerprint [3],
> >> > > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3"
> [4],
> >> > > > > > > > > > > * Java artifacts were built with Gradle 3.1 and
> >> > > > OpenJDK/Oracle
> >> > > > > > JDK
> >> > > > > > > > > > > 1.8.0_151
> >> > > > > > > > > > >
> >> > > > > > > > > > > The vote will be open for at least 72 hours. It is
> >> > adopted
> >> > > by
> >> > > > > > > > majority
> >> > > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks,
> >> > > > > > > > > > > Yaniv
> >> > > > > > > > > > >
> >> > > > > > > > > > > [1] https://issues.apache.org/
> >> > jira/secure/ReleaseNote.jspa
> >> > > ?
> >> > > > > > > > > > > projectId=12321521&version=12342793
> >> > > > > > > > > > > [2] https://dist.apache.org/repos/
> >> > > > > dist/dev/incubator/amaterasu/
> >> > > > > > > > > 0.2.0rc3/
> >> > > > > > > > > > > [3]
> >> > > > > > > https://dist.apache.org/repos/
> dist/dev/incubator/amaterasu/K
> >> EYS
> >> > > > > > > > > > > [4] https://github.com/apache/incu
> >> bator-amaterasu/tags
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Yaniv Rodenski
> >> >
> >> > +61 477 778 405
> >> > yaniv@shinto.io
> >> >
> >>
> >
> >
> >
> > --
> > Yaniv Rodenski
> >
> > +61 477 778 405
> > yaniv@shinto.io
> >
>
>
>
> --
> Yaniv Rodenski
>
> +61 477 778 405
> yaniv@shinto.io
>
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Posted by Yaniv Rodenski <ya...@shinto.io>.
One thing to clarify, the tar.gz is the source code as I've seen this is
released in other projects. if that is not needed I'm happy to drop it from
future releases.
On Fri, Jun 1, 2018 at 5:01 PM, Yaniv Rodenski <ya...@shinto.io> wrote:
> Thanks Davor,
>
> We will fix those for the next release.
>
> Cheers,
> Yaniv
>
> On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci <db...@gmail.com> wrote:
>
>> Great improvement!
>>
>> license/notice/disclaimer: ok
>> headers: ~fine (missing in some non-code files, it would be great to add
>> those -- non-blocking)
>> binaries: there's a gradle binary, but that may be ruled as acceptable
>> (non-blocking from me)
>>
>> Not sure why we need both .tar and .tar.gz. I'd get rid of one of them.
>> Also, shipping private keys in the distribution looks weird as well.
>>
>> For the first release, this is good. So, +1 to send onwards for an
>> Incubator review, and polish a few of these things for a subsequent
>> release.
>>
>> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski <ya...@shinto.io> wrote:
>>
>> > Hi All,
>> >
>> > Just a kind reminder for our mentors, we need your vote on this one :)
>> >
>> > Cheers,
>> > Yaniv
>> >
>> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <ar...@arunma.com>
>> wrote:
>> >
>> > > Thanks a lot, Nadav. Will get home and spend some more time on this. I
>> > was
>> > > in a rush and did this poor workaround. My VM is just 8 GB.
>> > >
>> > > Cheers
>> > > Arun
>> > >
>> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com>
>> > wrote:
>> > >
>> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP
>> docker
>> > on
>> > > > AWS). Both Spark-Scala and PySpark.
>> > > > It worked on Yaniv's HDP cluster as well.
>> > > > Worth noting:
>> > > > 1. HDP 2.6.4
>> > > > 2. Cluster has total of 32GB memory available
>> > > > 3. Each container is allocated 1G memory.
>> > > > 4. Amaterasu.properties:
>> > > >
>> > > > zk=sandbox-hdp.hortonworks.com
>> > > > version=0.2.0-incubating-rc3
>> > > > master=192.168.33.11
>> > > > user=root
>> > > > mode=yarn
>> > > > webserver.port=8000
>> > > > webserver.root=dist
>> > > > spark.version=2.6.4.0-91
>> > > > yarn.queue=default
>> > > > yarn.jarspath=hdfs:///apps/amaterasu
>> > > > spark.home=/usr/hdp/current/spark2-client
>> > > >
>> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.
>> > cloudera2-1.cdh5.7.0.p0.
>> > > 171658/lib/spark2
>> > > > yarn.hadoop.home.dir=/etc/hadoop
>> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.
>> 6.4.0-91"
>> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
>> > > >
>> > > >
>> > > > Arun, please share:
>> > > > 1. YARN memory configurations
>> > > > 2. amaterasu.properties content
>> > > > 3. HDP version.
>> > > >
>> > > > Cheers,
>> > > > Nadav
>> > > >
>> > > >
>> > > > On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
>> > > >
>> > > > > The pmem disabling is just temporary. I'll do a detailed analysis
>> and
>> > > get
>> > > > > back with a proper solution.
>> > > > >
>> > > > > Any hints on this front is highly appreciated.
>> > > > >
>> > > > > Cheers
>> > > > > Arun
>> > > > >
>> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <
>> nadavhartzvi@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > > Yaniv, Eyal, this might be related to the same issue you faced
>> with
>> > > > HDP.
>> > > > > > Can you confirm?
>> > > > > >
>> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
>> > wrote:
>> > > > > >
>> > > > > > > +1 from me
>> > > > > > >
>> > > > > > > Unit Tests and Build ran fine.
>> > > > > > >
>> > > > > > > Tested on HDP (VM) but had trouble allocating containers
>> (didn't
>> > > have
>> > > > > > that
>> > > > > > > before). Apparently Centos VMs are known to have this
>> problem.
>> > > > > Disabled
>> > > > > > > physical memory check (yarn.nodemanager.pmem-check-enabled)
>> and
>> > > ran
>> > > > > jobs
>> > > > > > > successfully.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
>> > > > > > kirupagaran.d@gmail.com
>> > > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Unit tests passing and build was successful on the branch
>> > > > > > > > "version-0.2.0-incubating-rc3"
>> > > > > > > >
>> > > > > > > > +1 from me
>> > > > > > > >
>> > > > > > > > Cheers,
>> > > > > > > > Kirupa
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
>> whisrael@gmail.com
>> > >
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > +1 looks good to me
>> > > > > > > > >
>> > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
>> > > > nadavhartzvi@gmail.com
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > +1 approve. Tested multiple times and after a long
>> round of
>> > > > > fixing
>> > > > > > > and
>> > > > > > > > > > testing over and over.
>> > > > > > > > > >
>> > > > > > > > > > Cheers,
>> > > > > > > > > > Nadav
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <
>> yaniv@shinto.io>
>> > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi everyone,
>> > > > > > > > > > >
>> > > > > > > > > > > We have fixed the legal issues, as well as a bug
>> found by
>> > > > > @Nadav
>> > > > > > > > please
>> > > > > > > > > > > review and vote on the release candidate #3 for the
>> > version
>> > > > > > > > > > > 0.2.0-incubating, as follows
>> > > > > > > > > > >
>> > > > > > > > > > > [ ] +1, Approve the release
>> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
>> > specific
>> > > > > > > comments)
>> > > > > > > > > > >
>> > > > > > > > > > > The complete staging area is available for your
>> review,
>> > > which
>> > > > > > > > includes:
>> > > > > > > > > > >
>> > > > > > > > > > > * JIRA release notes [1],
>> > > > > > > > > > > * the official Apache source release to be deployed to
>> > > > > > > > dist.apache.org
>> > > > > > > > > > > [2],
>> > > > > > > > > > > which is signed with the key with fingerprint [3],
>> > > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
>> > > > > > > > > > > * Java artifacts were built with Gradle 3.1 and
>> > > > OpenJDK/Oracle
>> > > > > > JDK
>> > > > > > > > > > > 1.8.0_151
>> > > > > > > > > > >
>> > > > > > > > > > > The vote will be open for at least 72 hours. It is
>> > adopted
>> > > by
>> > > > > > > > majority
>> > > > > > > > > > > approval, with at least 3 PMC affirmative votes.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > > Yaniv
>> > > > > > > > > > >
>> > > > > > > > > > > [1] https://issues.apache.org/
>> > jira/secure/ReleaseNote.jspa
>> > > ?
>> > > > > > > > > > > projectId=12321521&version=12342793
>> > > > > > > > > > > [2] https://dist.apache.org/repos/
>> > > > > dist/dev/incubator/amaterasu/
>> > > > > > > > > 0.2.0rc3/
>> > > > > > > > > > > [3]
>> > > > > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/K
>> EYS
>> > > > > > > > > > > [4] https://github.com/apache/incu
>> bator-amaterasu/tags
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Yaniv Rodenski
>> >
>> > +61 477 778 405
>> > yaniv@shinto.io
>> >
>>
>
>
>
> --
> Yaniv Rodenski
>
> +61 477 778 405
> yaniv@shinto.io
>
--
Yaniv Rodenski
+61 477 778 405
yaniv@shinto.io