You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@amaterasu.apache.org by Yaniv Rodenski <ya...@shinto.io> on 2018/05/29 04:38:21 UTC

[VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Hi everyone,

We have fixed the legal issues, as well as a bug found by @Nadav please
review and vote on the release candidate #3 for the version
0.2.0-incubating, as follows

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:

* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint [3],
* source code tag "version-0.2.0-incubating-rc3" [4],
* Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK 1.8.0_151

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Yaniv

[1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
projectId=12321521&version=12342793
[2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/0.2.0rc3/
[3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
[4] https://github.com/apache/incubator-amaterasu/tags

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1

I checked the signature, the headers: it looks good.

Thanks !
Regards
JB

On 02/06/2018 17:32, Arun Manivannan wrote:
> Gentlemen,
> 
> Apologies for coming back late.  The issue was just the minimum container
> size that was configured in my cluster
> (yarn.scheduler.maximum-allocation-mb).  It was set at 1 GB.
> 
> I didn't specify any spark specific memory parameters during my run (the
> memory defaults that the SparkSetupProvider was looking at) and to top it
> the code was setting the Xmx at 1 GB causing the overallocation and
> failure.
> 
> I have one minor proposal.  If this is agreeable, I can raise a quick PR.
> 
> Can we pull out the executor java options as a property in the
> amaterasu.properties?
> 
> amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true
> -Dhdp.version=2.6.5.0-292"
> 
> 
> As a side effect, we must provide the flexibility to allow quotes around
> the parameter but passing the quotes to the java command would fail.  I
> have stripped off the extra quotes in a dirty way at the moment. Should we
> consider proper command parsing (and possibly convert them to be bash
> compatible strings)?
> 
> s"java -cp spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/
> " +
>   s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+
> 
> 
> Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the latest
> from the branch.
> 
> Cheers,
> Arun
> 
> On Wed, May 30, 2018 at 1:25 PM Arun Manivannan <ar...@arunma.com> wrote:
> 
>> Thanks a lot, Nadav. Will get home and spend some more time on this. I was
>> in a rush and did this poor workaround. My VM is just 8 GB.
>>
>> Cheers
>> Arun
>>
>>
>> On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com> wrote:
>>
>>> Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on
>>> AWS). Both Spark-Scala and PySpark.
>>> It worked on Yaniv's HDP cluster as well.
>>> Worth noting:
>>> 1. HDP 2.6.4
>>> 2. Cluster has total of 32GB memory available
>>> 3. Each container is allocated 1G memory.
>>> 4. Amaterasu.properties:
>>>
>>> zk=sandbox-hdp.hortonworks.com
>>> version=0.2.0-incubating-rc3
>>> master=192.168.33.11
>>> user=root
>>> mode=yarn
>>> webserver.port=8000
>>> webserver.root=dist
>>> spark.version=2.6.4.0-91
>>> yarn.queue=default
>>> yarn.jarspath=hdfs:///apps/amaterasu
>>> spark.home=/usr/hdp/current/spark2-client
>>>
>>> #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2
>>> yarn.hadoop.home.dir=/etc/hadoop
>>> spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
>>> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
>>>
>>>
>>> Arun, please share:
>>> 1. YARN memory configurations
>>> 2. amaterasu.properties content
>>> 3. HDP version.
>>>
>>> Cheers,
>>> Nadav
>>>
>>>
>>> On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
>>>
>>>> The pmem disabling is just temporary. I'll do a detailed analysis and
>>> get
>>>> back with a proper solution.
>>>>
>>>> Any hints on this front is highly appreciated.
>>>>
>>>> Cheers
>>>> Arun
>>>>
>>>> On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <na...@gmail.com>
>>> wrote:
>>>>
>>>>> Yaniv, Eyal, this might be related to the same issue you faced with
>>> HDP.
>>>>> Can you confirm?
>>>>>
>>>>> On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com> wrote:
>>>>>
>>>>>> +1 from me
>>>>>>
>>>>>> Unit Tests and Build ran fine.
>>>>>>
>>>>>> Tested on HDP (VM) but had trouble allocating containers (didn't
>>> have
>>>>> that
>>>>>> before).  Apparently Centos VMs are known to have this problem.
>>>> Disabled
>>>>>> physical memory check  (yarn.nodemanager.pmem-check-enabled) and ran
>>>> jobs
>>>>>> successfully.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
>>>>> kirupagaran.d@gmail.com
>>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Unit tests passing and build was successful on the branch
>>>>>>> "version-0.2.0-incubating-rc3"
>>>>>>>
>>>>>>> +1 from me
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Kirupa
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 29, 2018 at 3:06 PM, guy peleg <wh...@gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>>> +1 looks good to me
>>>>>>>>
>>>>>>>> On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
>>> nadavhartzvi@gmail.com
>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1 approve. Tested multiple times and after a long round of
>>>> fixing
>>>>>> and
>>>>>>>>> testing over and over.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Nadav
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io>
>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> We have fixed the legal issues, as well as a bug found by
>>>> @Nadav
>>>>>>> please
>>>>>>>>>> review and vote on the release candidate #3 for the version
>>>>>>>>>> 0.2.0-incubating, as follows
>>>>>>>>>>
>>>>>>>>>> [ ] +1, Approve the release
>>>>>>>>>> [ ] -1, Do not approve the release (please provide specific
>>>>>> comments)
>>>>>>>>>>
>>>>>>>>>> The complete staging area is available for your review,
>>> which
>>>>>>> includes:
>>>>>>>>>>
>>>>>>>>>> * JIRA release notes [1],
>>>>>>>>>> * the official Apache source release to be deployed to
>>>>>>> dist.apache.org
>>>>>>>>>> [2],
>>>>>>>>>> which is signed with the key with fingerprint [3],
>>>>>>>>>> * source code tag "version-0.2.0-incubating-rc3" [4],
>>>>>>>>>> * Java artifacts were built with Gradle 3.1 and
>>> OpenJDK/Oracle
>>>>> JDK
>>>>>>>>>> 1.8.0_151
>>>>>>>>>>
>>>>>>>>>> The vote will be open for at least 72 hours. It is adopted
>>> by
>>>>>>> majority
>>>>>>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Yaniv
>>>>>>>>>>
>>>>>>>>>> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>>>>>>>>> projectId=12321521&version=12342793
>>>>>>>>>> [2] https://dist.apache.org/repos/
>>>> dist/dev/incubator/amaterasu/
>>>>>>>> 0.2.0rc3/
>>>>>>>>>> [3]
>>>>>> https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
>>>>>>>>>> [4] https://github.com/apache/incubator-amaterasu/tags
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Olivier Lamy <ol...@apache.org>.
+1


On Mon, 4 Jun 2018 at 17:02, Yaniv Rodenski <ya...@shinto.io> wrote:

> Hi All,
>
> Just following up with @JB and @Olivier, any chance you guys can have a
> look and vote?
>
> Cheers,
> Yaniv
>
> On Mon, Jun 4, 2018 at 12:08 AM, Arun Manivannan <ar...@arunma.com> wrote:
>
> > Ah. Missed the part where we could configure parameters at a job level.
> > Thanks a lot for clarifying, Yaniv.
> >
> > Cheers,
> > Arun
> >
> > On Sun, Jun 3, 2018 at 6:37 PM Yaniv Rodenski <ya...@shinto.io> wrote:
> >
> > > Hi Arun,
> > >
> > > Fair point, but I think it can be configured via
> env/[env_name]/spark.yml
> > > am I wrong?
> > > Anyway, ideally I think we should try and have per job configurations
> in
> > > the environment rather than in the amaterasu.properties
> > >
> > > Cheers,
> > > Yaniv
> > >
> > > On Sun, 3 Jun 2018 at 1:32 am, Arun Manivannan <ar...@arunma.com>
> wrote:
> > >
> > > > Gentlemen,
> > > >
> > > > Apologies for coming back late.  The issue was just the minimum
> > container
> > > > size that was configured in my cluster
> > > > (yarn.scheduler.maximum-allocation-mb).  It was set at 1 GB.
> > > >
> > > > I didn't specify any spark specific memory parameters during my run
> > (the
> > > > memory defaults that the SparkSetupProvider was looking at) and to
> top
> > it
> > > > the code was setting the Xmx at 1 GB causing the overallocation and
> > > > failure.
> > > >
> > > > I have one minor proposal.  If this is agreeable, I can raise a quick
> > PR.
> > > >
> > > > Can we pull out the executor java options as a property in the
> > > > amaterasu.properties?
> > > >
> > > > amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true
> > > > -Dhdp.version=2.6.5.0-292"
> > > >
> > > >
> > > > As a side effect, we must provide the flexibility to allow quotes
> > around
> > > > the parameter but passing the quotes to the java command would
> fail.  I
> > > > have stripped off the extra quotes in a dirty way at the moment.
> Should
> > > we
> > > > consider proper command parsing (and possibly convert them to be bash
> > > > compatible strings)?
> > > >
> > > > s"java -cp
> > > >
> > spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/
> > > > " +
> > > >   s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+
> > > >
> > > >
> > > > Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the
> > > latest
> > > > from the branch.
> > > >
> > > > Cheers,
> > > > Arun
> > > >
> > > > On Wed, May 30, 2018 at 1:25 PM Arun Manivannan <ar...@arunma.com>
> > wrote:
> > > >
> > > > > Thanks a lot, Nadav. Will get home and spend some more time on
> this.
> > I
> > > > was
> > > > > in a rush and did this poor workaround. My VM is just 8 GB.
> > > > >
> > > > > Cheers
> > > > > Arun
> > > > >
> > > > >
> > > > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <nadavhartzvi@gmail.com
> >
> > > > wrote:
> > > > >
> > > > >> Yaniv and I just tested it. It worked flawlessly on my end (HDP
> > docker
> > > > on
> > > > >> AWS). Both Spark-Scala and PySpark.
> > > > >> It worked on Yaniv's HDP cluster as well.
> > > > >> Worth noting:
> > > > >> 1. HDP 2.6.4
> > > > >> 2. Cluster has total of 32GB memory available
> > > > >> 3. Each container is allocated 1G memory.
> > > > >> 4. Amaterasu.properties:
> > > > >>
> > > > >> zk=sandbox-hdp.hortonworks.com
> > > > >> version=0.2.0-incubating-rc3
> > > > >> master=192.168.33.11
> > > > >> user=root
> > > > >> mode=yarn
> > > > >> webserver.port=8000
> > > > >> webserver.root=dist
> > > > >> spark.version=2.6.4.0-91
> > > > >> yarn.queue=default
> > > > >> yarn.jarspath=hdfs:///apps/amaterasu
> > > > >> spark.home=/usr/hdp/current/spark2-client
> > > > >>
> > > > >>
> > > >
> > >
> >
> #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2
> > > > >> yarn.hadoop.home.dir=/etc/hadoop
> > > > >> spark.opts.spark.yarn.am
> > .extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > > > >>
> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > > > >>
> > > > >>
> > > > >> Arun, please share:
> > > > >> 1. YARN memory configurations
> > > > >> 2. amaterasu.properties content
> > > > >> 3. HDP version.
> > > > >>
> > > > >> Cheers,
> > > > >> Nadav
> > > > >>
> > > > >>
> > > > >> On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
> > > > >>
> > > > >> > The pmem disabling is just temporary. I'll do a detailed
> analysis
> > > and
> > > > >> get
> > > > >> > back with a proper solution.
> > > > >> >
> > > > >> > Any hints on this front is highly appreciated.
> > > > >> >
> > > > >> > Cheers
> > > > >> > Arun
> > > > >> >
> > > > >> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <
> > nadavhartzvi@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Yaniv, Eyal, this might be related to the same issue you faced
> > > with
> > > > >> HDP.
> > > > >> > > Can you confirm?
> > > > >> > >
> > > > >> > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
> > > > wrote:
> > > > >> > >
> > > > >> > > > +1 from me
> > > > >> > > >
> > > > >> > > > Unit Tests and Build ran fine.
> > > > >> > > >
> > > > >> > > > Tested on HDP (VM) but had trouble allocating containers
> > (didn't
> > > > >> have
> > > > >> > > that
> > > > >> > > > before).  Apparently Centos VMs are known to have this
> > problem.
> > > > >> > Disabled
> > > > >> > > > physical memory check  (yarn.nodemanager.pmem-check-enabled)
> > and
> > > > ran
> > > > >> > jobs
> > > > >> > > > successfully.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > > > >> > > kirupagaran.d@gmail.com
> > > > >> > > > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Unit tests passing and build was successful on the branch
> > > > >> > > > > "version-0.2.0-incubating-rc3"
> > > > >> > > > >
> > > > >> > > > > +1 from me
> > > > >> > > > >
> > > > >> > > > > Cheers,
> > > > >> > > > > Kirupa
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
> > > whisrael@gmail.com>
> > > > >> > wrote:
> > > > >> > > > >
> > > > >> > > > > > +1 looks good to me
> > > > >> > > > > >
> > > > >> > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> > > > >> nadavhartzvi@gmail.com
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > +1 approve. Tested multiple times and after a long
> round
> > > of
> > > > >> > fixing
> > > > >> > > > and
> > > > >> > > > > > > testing over and over.
> > > > >> > > > > > >
> > > > >> > > > > > > Cheers,
> > > > >> > > > > > > Nadav
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <
> > yaniv@shinto.io>
> > > > >> wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi everyone,
> > > > >> > > > > > > >
> > > > >> > > > > > > > We have fixed the legal issues, as well as a bug
> found
> > > by
> > > > >> > @Nadav
> > > > >> > > > > please
> > > > >> > > > > > > > review and vote on the release candidate #3 for the
> > > > version
> > > > >> > > > > > > > 0.2.0-incubating, as follows
> > > > >> > > > > > > >
> > > > >> > > > > > > > [ ] +1, Approve the release
> > > > >> > > > > > > > [ ] -1, Do not approve the release (please provide
> > > > specific
> > > > >> > > > comments)
> > > > >> > > > > > > >
> > > > >> > > > > > > > The complete staging area is available for your
> > review,
> > > > >> which
> > > > >> > > > > includes:
> > > > >> > > > > > > >
> > > > >> > > > > > > > * JIRA release notes [1],
> > > > >> > > > > > > > * the official Apache source release to be deployed
> to
> > > > >> > > > > dist.apache.org
> > > > >> > > > > > > > [2],
> > > > >> > > > > > > > which is signed with the key with fingerprint [3],
> > > > >> > > > > > > > * source code tag "version-0.2.0-incubating-rc3"
> [4],
> > > > >> > > > > > > > * Java artifacts were built with Gradle 3.1 and
> > > > >> OpenJDK/Oracle
> > > > >> > > JDK
> > > > >> > > > > > > > 1.8.0_151
> > > > >> > > > > > > >
> > > > >> > > > > > > > The vote will be open for at least 72 hours. It is
> > > adopted
> > > > >> by
> > > > >> > > > > majority
> > > > >> > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thanks,
> > > > >> > > > > > > > Yaniv
> > > > >> > > > > > > >
> > > > >> > > > > > > > [1]
> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > >> > > > > > > > projectId=12321521&version=12342793
> > > > >> > > > > > > > [2] https://dist.apache.org/repos/
> > > > >> > dist/dev/incubator/amaterasu/
> > > > >> > > > > > 0.2.0rc3/
> > > > >> > > > > > > > [3]
> > > > >> > > >
> > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > > > >> > > > > > > > [4]
> > https://github.com/apache/incubator-amaterasu/tags
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > > --
> > > Yaniv Rodenski
> > >
> > > +61 477 778 405 <+61%20477%20778%20405>
> > > yaniv@shinto.io
> > >
> >
>
>
>
> --
> Yaniv Rodenski
>
> +61 477 778 405
> yaniv@shinto.io
> --
> Yaniv Rodenski
>
> +61 477 778 405
> yaniv@shinto.io
>


-- 
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Yaniv Rodenski <ya...@shinto.io>.
Hi All,

Just following up with @JB and @Olivier, any chance you guys can have a
look and vote?

Cheers,
Yaniv

On Mon, Jun 4, 2018 at 12:08 AM, Arun Manivannan <ar...@arunma.com> wrote:

> Ah. Missed the part where we could configure parameters at a job level.
> Thanks a lot for clarifying, Yaniv.
>
> Cheers,
> Arun
>
> On Sun, Jun 3, 2018 at 6:37 PM Yaniv Rodenski <ya...@shinto.io> wrote:
>
> > Hi Arun,
> >
> > Fair point, but I think it can be configured via env/[env_name]/spark.yml
> > am I wrong?
> > Anyway, ideally I think we should try and have per job configurations in
> > the environment rather than in the amaterasu.properties
> >
> > Cheers,
> > Yaniv
> >
> > On Sun, 3 Jun 2018 at 1:32 am, Arun Manivannan <ar...@arunma.com> wrote:
> >
> > > Gentlemen,
> > >
> > > Apologies for coming back late.  The issue was just the minimum
> container
> > > size that was configured in my cluster
> > > (yarn.scheduler.maximum-allocation-mb).  It was set at 1 GB.
> > >
> > > I didn't specify any spark specific memory parameters during my run
> (the
> > > memory defaults that the SparkSetupProvider was looking at) and to top
> it
> > > the code was setting the Xmx at 1 GB causing the overallocation and
> > > failure.
> > >
> > > I have one minor proposal.  If this is agreeable, I can raise a quick
> PR.
> > >
> > > Can we pull out the executor java options as a property in the
> > > amaterasu.properties?
> > >
> > > amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true
> > > -Dhdp.version=2.6.5.0-292"
> > >
> > >
> > > As a side effect, we must provide the flexibility to allow quotes
> around
> > > the parameter but passing the quotes to the java command would fail.  I
> > > have stripped off the extra quotes in a dirty way at the moment. Should
> > we
> > > consider proper command parsing (and possibly convert them to be bash
> > > compatible strings)?
> > >
> > > s"java -cp
> > >
> spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/
> > > " +
> > >   s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+
> > >
> > >
> > > Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the
> > latest
> > > from the branch.
> > >
> > > Cheers,
> > > Arun
> > >
> > > On Wed, May 30, 2018 at 1:25 PM Arun Manivannan <ar...@arunma.com>
> wrote:
> > >
> > > > Thanks a lot, Nadav. Will get home and spend some more time on this.
> I
> > > was
> > > > in a rush and did this poor workaround. My VM is just 8 GB.
> > > >
> > > > Cheers
> > > > Arun
> > > >
> > > >
> > > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com>
> > > wrote:
> > > >
> > > >> Yaniv and I just tested it. It worked flawlessly on my end (HDP
> docker
> > > on
> > > >> AWS). Both Spark-Scala and PySpark.
> > > >> It worked on Yaniv's HDP cluster as well.
> > > >> Worth noting:
> > > >> 1. HDP 2.6.4
> > > >> 2. Cluster has total of 32GB memory available
> > > >> 3. Each container is allocated 1G memory.
> > > >> 4. Amaterasu.properties:
> > > >>
> > > >> zk=sandbox-hdp.hortonworks.com
> > > >> version=0.2.0-incubating-rc3
> > > >> master=192.168.33.11
> > > >> user=root
> > > >> mode=yarn
> > > >> webserver.port=8000
> > > >> webserver.root=dist
> > > >> spark.version=2.6.4.0-91
> > > >> yarn.queue=default
> > > >> yarn.jarspath=hdfs:///apps/amaterasu
> > > >> spark.home=/usr/hdp/current/spark2-client
> > > >>
> > > >>
> > >
> >
> #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2
> > > >> yarn.hadoop.home.dir=/etc/hadoop
> > > >> spark.opts.spark.yarn.am
> .extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > > >> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > > >>
> > > >>
> > > >> Arun, please share:
> > > >> 1. YARN memory configurations
> > > >> 2. amaterasu.properties content
> > > >> 3. HDP version.
> > > >>
> > > >> Cheers,
> > > >> Nadav
> > > >>
> > > >>
> > > >> On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
> > > >>
> > > >> > The pmem disabling is just temporary. I'll do a detailed analysis
> > and
> > > >> get
> > > >> > back with a proper solution.
> > > >> >
> > > >> > Any hints on this front is highly appreciated.
> > > >> >
> > > >> > Cheers
> > > >> > Arun
> > > >> >
> > > >> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <
> nadavhartzvi@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > Yaniv, Eyal, this might be related to the same issue you faced
> > with
> > > >> HDP.
> > > >> > > Can you confirm?
> > > >> > >
> > > >> > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
> > > wrote:
> > > >> > >
> > > >> > > > +1 from me
> > > >> > > >
> > > >> > > > Unit Tests and Build ran fine.
> > > >> > > >
> > > >> > > > Tested on HDP (VM) but had trouble allocating containers
> (didn't
> > > >> have
> > > >> > > that
> > > >> > > > before).  Apparently Centos VMs are known to have this
> problem.
> > > >> > Disabled
> > > >> > > > physical memory check  (yarn.nodemanager.pmem-check-enabled)
> and
> > > ran
> > > >> > jobs
> > > >> > > > successfully.
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > > >> > > kirupagaran.d@gmail.com
> > > >> > > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Unit tests passing and build was successful on the branch
> > > >> > > > > "version-0.2.0-incubating-rc3"
> > > >> > > > >
> > > >> > > > > +1 from me
> > > >> > > > >
> > > >> > > > > Cheers,
> > > >> > > > > Kirupa
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
> > whisrael@gmail.com>
> > > >> > wrote:
> > > >> > > > >
> > > >> > > > > > +1 looks good to me
> > > >> > > > > >
> > > >> > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> > > >> nadavhartzvi@gmail.com
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > +1 approve. Tested multiple times and after a long round
> > of
> > > >> > fixing
> > > >> > > > and
> > > >> > > > > > > testing over and over.
> > > >> > > > > > >
> > > >> > > > > > > Cheers,
> > > >> > > > > > > Nadav
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <
> yaniv@shinto.io>
> > > >> wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi everyone,
> > > >> > > > > > > >
> > > >> > > > > > > > We have fixed the legal issues, as well as a bug found
> > by
> > > >> > @Nadav
> > > >> > > > > please
> > > >> > > > > > > > review and vote on the release candidate #3 for the
> > > version
> > > >> > > > > > > > 0.2.0-incubating, as follows
> > > >> > > > > > > >
> > > >> > > > > > > > [ ] +1, Approve the release
> > > >> > > > > > > > [ ] -1, Do not approve the release (please provide
> > > specific
> > > >> > > > comments)
> > > >> > > > > > > >
> > > >> > > > > > > > The complete staging area is available for your
> review,
> > > >> which
> > > >> > > > > includes:
> > > >> > > > > > > >
> > > >> > > > > > > > * JIRA release notes [1],
> > > >> > > > > > > > * the official Apache source release to be deployed to
> > > >> > > > > dist.apache.org
> > > >> > > > > > > > [2],
> > > >> > > > > > > > which is signed with the key with fingerprint [3],
> > > >> > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > >> > > > > > > > * Java artifacts were built with Gradle 3.1 and
> > > >> OpenJDK/Oracle
> > > >> > > JDK
> > > >> > > > > > > > 1.8.0_151
> > > >> > > > > > > >
> > > >> > > > > > > > The vote will be open for at least 72 hours. It is
> > adopted
> > > >> by
> > > >> > > > > majority
> > > >> > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks,
> > > >> > > > > > > > Yaniv
> > > >> > > > > > > >
> > > >> > > > > > > > [1]
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > >> > > > > > > > projectId=12321521&version=12342793
> > > >> > > > > > > > [2] https://dist.apache.org/repos/
> > > >> > dist/dev/incubator/amaterasu/
> > > >> > > > > > 0.2.0rc3/
> > > >> > > > > > > > [3]
> > > >> > > >
> https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > > >> > > > > > > > [4]
> https://github.com/apache/incubator-amaterasu/tags
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> > --
> > Yaniv Rodenski
> >
> > +61 477 778 405 <+61%20477%20778%20405>
> > yaniv@shinto.io
> >
>



-- 
Yaniv Rodenski

+61 477 778 405
yaniv@shinto.io
-- 
Yaniv Rodenski

+61 477 778 405
yaniv@shinto.io

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Arun Manivannan <ar...@arunma.com>.
Ah. Missed the part where we could configure parameters at a job level.
Thanks a lot for clarifying, Yaniv.

Cheers,
Arun

On Sun, Jun 3, 2018 at 6:37 PM Yaniv Rodenski <ya...@shinto.io> wrote:

> Hi Arun,
>
> Fair point, but I think it can be configured via env/[env_name]/spark.yml
> am I wrong?
> Anyway, ideally I think we should try and have per job configurations in
> the environment rather than in the amaterasu.properties
>
> Cheers,
> Yaniv
>
> On Sun, 3 Jun 2018 at 1:32 am, Arun Manivannan <ar...@arunma.com> wrote:
>
> > Gentlemen,
> >
> > Apologies for coming back late.  The issue was just the minimum container
> > size that was configured in my cluster
> > (yarn.scheduler.maximum-allocation-mb).  It was set at 1 GB.
> >
> > I didn't specify any spark specific memory parameters during my run (the
> > memory defaults that the SparkSetupProvider was looking at) and to top it
> > the code was setting the Xmx at 1 GB causing the overallocation and
> > failure.
> >
> > I have one minor proposal.  If this is agreeable, I can raise a quick PR.
> >
> > Can we pull out the executor java options as a property in the
> > amaterasu.properties?
> >
> > amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true
> > -Dhdp.version=2.6.5.0-292"
> >
> >
> > As a side effect, we must provide the flexibility to allow quotes around
> > the parameter but passing the quotes to the java command would fail.  I
> > have stripped off the extra quotes in a dirty way at the moment. Should
> we
> > consider proper command parsing (and possibly convert them to be bash
> > compatible strings)?
> >
> > s"java -cp
> > spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/
> > " +
> >   s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+
> >
> >
> > Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the
> latest
> > from the branch.
> >
> > Cheers,
> > Arun
> >
> > On Wed, May 30, 2018 at 1:25 PM Arun Manivannan <ar...@arunma.com> wrote:
> >
> > > Thanks a lot, Nadav. Will get home and spend some more time on this. I
> > was
> > > in a rush and did this poor workaround. My VM is just 8 GB.
> > >
> > > Cheers
> > > Arun
> > >
> > >
> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com>
> > wrote:
> > >
> > >> Yaniv and I just tested it. It worked flawlessly on my end (HDP docker
> > on
> > >> AWS). Both Spark-Scala and PySpark.
> > >> It worked on Yaniv's HDP cluster as well.
> > >> Worth noting:
> > >> 1. HDP 2.6.4
> > >> 2. Cluster has total of 32GB memory available
> > >> 3. Each container is allocated 1G memory.
> > >> 4. Amaterasu.properties:
> > >>
> > >> zk=sandbox-hdp.hortonworks.com
> > >> version=0.2.0-incubating-rc3
> > >> master=192.168.33.11
> > >> user=root
> > >> mode=yarn
> > >> webserver.port=8000
> > >> webserver.root=dist
> > >> spark.version=2.6.4.0-91
> > >> yarn.queue=default
> > >> yarn.jarspath=hdfs:///apps/amaterasu
> > >> spark.home=/usr/hdp/current/spark2-client
> > >>
> > >>
> >
> #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2
> > >> yarn.hadoop.home.dir=/etc/hadoop
> > >> spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > >> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > >>
> > >>
> > >> Arun, please share:
> > >> 1. YARN memory configurations
> > >> 2. amaterasu.properties content
> > >> 3. HDP version.
> > >>
> > >> Cheers,
> > >> Nadav
> > >>
> > >>
> > >> On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
> > >>
> > >> > The pmem disabling is just temporary. I'll do a detailed analysis
> and
> > >> get
> > >> > back with a proper solution.
> > >> >
> > >> > Any hints on this front is highly appreciated.
> > >> >
> > >> > Cheers
> > >> > Arun
> > >> >
> > >> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <na...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Yaniv, Eyal, this might be related to the same issue you faced
> with
> > >> HDP.
> > >> > > Can you confirm?
> > >> > >
> > >> > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
> > wrote:
> > >> > >
> > >> > > > +1 from me
> > >> > > >
> > >> > > > Unit Tests and Build ran fine.
> > >> > > >
> > >> > > > Tested on HDP (VM) but had trouble allocating containers (didn't
> > >> have
> > >> > > that
> > >> > > > before).  Apparently Centos VMs are known to have this problem.
> > >> > Disabled
> > >> > > > physical memory check  (yarn.nodemanager.pmem-check-enabled) and
> > ran
> > >> > jobs
> > >> > > > successfully.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > >> > > kirupagaran.d@gmail.com
> > >> > > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Unit tests passing and build was successful on the branch
> > >> > > > > "version-0.2.0-incubating-rc3"
> > >> > > > >
> > >> > > > > +1 from me
> > >> > > > >
> > >> > > > > Cheers,
> > >> > > > > Kirupa
> > >> > > > >
> > >> > > > >
> > >> > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
> whisrael@gmail.com>
> > >> > wrote:
> > >> > > > >
> > >> > > > > > +1 looks good to me
> > >> > > > > >
> > >> > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> > >> nadavhartzvi@gmail.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > >
> > >> > > > > > > +1 approve. Tested multiple times and after a long round
> of
> > >> > fixing
> > >> > > > and
> > >> > > > > > > testing over and over.
> > >> > > > > > >
> > >> > > > > > > Cheers,
> > >> > > > > > > Nadav
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io>
> > >> wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi everyone,
> > >> > > > > > > >
> > >> > > > > > > > We have fixed the legal issues, as well as a bug found
> by
> > >> > @Nadav
> > >> > > > > please
> > >> > > > > > > > review and vote on the release candidate #3 for the
> > version
> > >> > > > > > > > 0.2.0-incubating, as follows
> > >> > > > > > > >
> > >> > > > > > > > [ ] +1, Approve the release
> > >> > > > > > > > [ ] -1, Do not approve the release (please provide
> > specific
> > >> > > > comments)
> > >> > > > > > > >
> > >> > > > > > > > The complete staging area is available for your review,
> > >> which
> > >> > > > > includes:
> > >> > > > > > > >
> > >> > > > > > > > * JIRA release notes [1],
> > >> > > > > > > > * the official Apache source release to be deployed to
> > >> > > > > dist.apache.org
> > >> > > > > > > > [2],
> > >> > > > > > > > which is signed with the key with fingerprint [3],
> > >> > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > >> > > > > > > > * Java artifacts were built with Gradle 3.1 and
> > >> OpenJDK/Oracle
> > >> > > JDK
> > >> > > > > > > > 1.8.0_151
> > >> > > > > > > >
> > >> > > > > > > > The vote will be open for at least 72 hours. It is
> adopted
> > >> by
> > >> > > > > majority
> > >> > > > > > > > approval, with at least 3 PMC affirmative votes.
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > > Yaniv
> > >> > > > > > > >
> > >> > > > > > > > [1]
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > >> > > > > > > > projectId=12321521&version=12342793
> > >> > > > > > > > [2] https://dist.apache.org/repos/
> > >> > dist/dev/incubator/amaterasu/
> > >> > > > > > 0.2.0rc3/
> > >> > > > > > > > [3]
> > >> > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > >> > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
> --
> Yaniv Rodenski
>
> +61 477 778 405 <+61%20477%20778%20405>
> yaniv@shinto.io
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Yaniv Rodenski <ya...@shinto.io>.
Hi Arun,

Fair point, but I think it can be configured via env/[env_name]/spark.yml
am I wrong?
Anyway, ideally I think we should try and have per job configurations in
the environment rather than in the amaterasu.properties

Cheers,
Yaniv

On Sun, 3 Jun 2018 at 1:32 am, Arun Manivannan <ar...@arunma.com> wrote:

> Gentlemen,
>
> Apologies for coming back late.  The issue was just the minimum container
> size that was configured in my cluster
> (yarn.scheduler.maximum-allocation-mb).  It was set at 1 GB.
>
> I didn't specify any spark specific memory parameters during my run (the
> memory defaults that the SparkSetupProvider was looking at) and to top it
> the code was setting the Xmx at 1 GB causing the overallocation and
> failure.
>
> I have one minor proposal.  If this is agreeable, I can raise a quick PR.
>
> Can we pull out the executor java options as a property in the
> amaterasu.properties?
>
> amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true
> -Dhdp.version=2.6.5.0-292"
>
>
> As a side effect, we must provide the flexibility to allow quotes around
> the parameter but passing the quotes to the java command would fail.  I
> have stripped off the extra quotes in a dirty way at the moment. Should we
> consider proper command parsing (and possibly convert them to be bash
> compatible strings)?
>
> s"java -cp
> spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/
> " +
>   s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+
>
>
> Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the latest
> from the branch.
>
> Cheers,
> Arun
>
> On Wed, May 30, 2018 at 1:25 PM Arun Manivannan <ar...@arunma.com> wrote:
>
> > Thanks a lot, Nadav. Will get home and spend some more time on this. I
> was
> > in a rush and did this poor workaround. My VM is just 8 GB.
> >
> > Cheers
> > Arun
> >
> >
> > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com>
> wrote:
> >
> >> Yaniv and I just tested it. It worked flawlessly on my end (HDP docker
> on
> >> AWS). Both Spark-Scala and PySpark.
> >> It worked on Yaniv's HDP cluster as well.
> >> Worth noting:
> >> 1. HDP 2.6.4
> >> 2. Cluster has total of 32GB memory available
> >> 3. Each container is allocated 1G memory.
> >> 4. Amaterasu.properties:
> >>
> >> zk=sandbox-hdp.hortonworks.com
> >> version=0.2.0-incubating-rc3
> >> master=192.168.33.11
> >> user=root
> >> mode=yarn
> >> webserver.port=8000
> >> webserver.root=dist
> >> spark.version=2.6.4.0-91
> >> yarn.queue=default
> >> yarn.jarspath=hdfs:///apps/amaterasu
> >> spark.home=/usr/hdp/current/spark2-client
> >>
> >>
> #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2
> >> yarn.hadoop.home.dir=/etc/hadoop
> >> spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> >> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> >>
> >>
> >> Arun, please share:
> >> 1. YARN memory configurations
> >> 2. amaterasu.properties content
> >> 3. HDP version.
> >>
> >> Cheers,
> >> Nadav
> >>
> >>
> >> On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
> >>
> >> > The pmem disabling is just temporary. I'll do a detailed analysis and
> >> get
> >> > back with a proper solution.
> >> >
> >> > Any hints on this front is highly appreciated.
> >> >
> >> > Cheers
> >> > Arun
> >> >
> >> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <na...@gmail.com>
> >> wrote:
> >> >
> >> > > Yaniv, Eyal, this might be related to the same issue you faced with
> >> HDP.
> >> > > Can you confirm?
> >> > >
> >> > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
> wrote:
> >> > >
> >> > > > +1 from me
> >> > > >
> >> > > > Unit Tests and Build ran fine.
> >> > > >
> >> > > > Tested on HDP (VM) but had trouble allocating containers (didn't
> >> have
> >> > > that
> >> > > > before).  Apparently Centos VMs are known to have this problem.
> >> > Disabled
> >> > > > physical memory check  (yarn.nodemanager.pmem-check-enabled) and
> ran
> >> > jobs
> >> > > > successfully.
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> >> > > kirupagaran.d@gmail.com
> >> > > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Unit tests passing and build was successful on the branch
> >> > > > > "version-0.2.0-incubating-rc3"
> >> > > > >
> >> > > > > +1 from me
> >> > > > >
> >> > > > > Cheers,
> >> > > > > Kirupa
> >> > > > >
> >> > > > >
> >> > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <wh...@gmail.com>
> >> > wrote:
> >> > > > >
> >> > > > > > +1 looks good to me
> >> > > > > >
> >> > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> >> nadavhartzvi@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > +1 approve. Tested multiple times and after a long round of
> >> > fixing
> >> > > > and
> >> > > > > > > testing over and over.
> >> > > > > > >
> >> > > > > > > Cheers,
> >> > > > > > > Nadav
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io>
> >> wrote:
> >> > > > > > >
> >> > > > > > > > Hi everyone,
> >> > > > > > > >
> >> > > > > > > > We have fixed the legal issues, as well as a bug found by
> >> > @Nadav
> >> > > > > please
> >> > > > > > > > review and vote on the release candidate #3 for the
> version
> >> > > > > > > > 0.2.0-incubating, as follows
> >> > > > > > > >
> >> > > > > > > > [ ] +1, Approve the release
> >> > > > > > > > [ ] -1, Do not approve the release (please provide
> specific
> >> > > > comments)
> >> > > > > > > >
> >> > > > > > > > The complete staging area is available for your review,
> >> which
> >> > > > > includes:
> >> > > > > > > >
> >> > > > > > > > * JIRA release notes [1],
> >> > > > > > > > * the official Apache source release to be deployed to
> >> > > > > dist.apache.org
> >> > > > > > > > [2],
> >> > > > > > > > which is signed with the key with fingerprint [3],
> >> > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> >> > > > > > > > * Java artifacts were built with Gradle 3.1 and
> >> OpenJDK/Oracle
> >> > > JDK
> >> > > > > > > > 1.8.0_151
> >> > > > > > > >
> >> > > > > > > > The vote will be open for at least 72 hours. It is adopted
> >> by
> >> > > > > majority
> >> > > > > > > > approval, with at least 3 PMC affirmative votes.
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > Yaniv
> >> > > > > > > >
> >> > > > > > > > [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >> > > > > > > > projectId=12321521&version=12342793
> >> > > > > > > > [2] https://dist.apache.org/repos/
> >> > dist/dev/incubator/amaterasu/
> >> > > > > > 0.2.0rc3/
> >> > > > > > > > [3]
> >> > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> >> > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
-- 
Yaniv Rodenski

+61 477 778 405
yaniv@shinto.io

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Arun Manivannan <ar...@arunma.com>.
Gentlemen,

Apologies for coming back late.  The issue was just the minimum container
size that was configured in my cluster
(yarn.scheduler.maximum-allocation-mb).  It was set at 1 GB.

I didn't specify any spark specific memory parameters during my run (the
memory defaults that the SparkSetupProvider was looking at) and to top it
the code was setting the Xmx at 1 GB causing the overallocation and
failure.

I have one minor proposal.  If this is agreeable, I can raise a quick PR.

Can we pull out the executor java options as a property in the
amaterasu.properties?

amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true
-Dhdp.version=2.6.5.0-292"


As a side effect, we must provide the flexibility to allow quotes around
the parameter but passing the quotes to the java command would fail.  I
have stripped off the extra quotes in a dirty way at the moment. Should we
consider proper command parsing (and possibly convert them to be bash
compatible strings)?

s"java -cp spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/
" +
  s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+


Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the latest
from the branch.

Cheers,
Arun

On Wed, May 30, 2018 at 1:25 PM Arun Manivannan <ar...@arunma.com> wrote:

> Thanks a lot, Nadav. Will get home and spend some more time on this. I was
> in a rush and did this poor workaround. My VM is just 8 GB.
>
> Cheers
> Arun
>
>
> On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com> wrote:
>
>> Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on
>> AWS). Both Spark-Scala and PySpark.
>> It worked on Yaniv's HDP cluster as well.
>> Worth noting:
>> 1. HDP 2.6.4
>> 2. Cluster has total of 32GB memory available
>> 3. Each container is allocated 1G memory.
>> 4. Amaterasu.properties:
>>
>> zk=sandbox-hdp.hortonworks.com
>> version=0.2.0-incubating-rc3
>> master=192.168.33.11
>> user=root
>> mode=yarn
>> webserver.port=8000
>> webserver.root=dist
>> spark.version=2.6.4.0-91
>> yarn.queue=default
>> yarn.jarspath=hdfs:///apps/amaterasu
>> spark.home=/usr/hdp/current/spark2-client
>>
>> #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2
>> yarn.hadoop.home.dir=/etc/hadoop
>> spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
>> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
>>
>>
>> Arun, please share:
>> 1. YARN memory configurations
>> 2. amaterasu.properties content
>> 3. HDP version.
>>
>> Cheers,
>> Nadav
>>
>>
>> On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
>>
>> > The pmem disabling is just temporary. I'll do a detailed analysis and
>> get
>> > back with a proper solution.
>> >
>> > Any hints on this front is highly appreciated.
>> >
>> > Cheers
>> > Arun
>> >
>> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <na...@gmail.com>
>> wrote:
>> >
>> > > Yaniv, Eyal, this might be related to the same issue you faced with
>> HDP.
>> > > Can you confirm?
>> > >
>> > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com> wrote:
>> > >
>> > > > +1 from me
>> > > >
>> > > > Unit Tests and Build ran fine.
>> > > >
>> > > > Tested on HDP (VM) but had trouble allocating containers (didn't
>> have
>> > > that
>> > > > before).  Apparently Centos VMs are known to have this problem.
>> > Disabled
>> > > > physical memory check  (yarn.nodemanager.pmem-check-enabled) and ran
>> > jobs
>> > > > successfully.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
>> > > kirupagaran.d@gmail.com
>> > > > >
>> > > > wrote:
>> > > >
>> > > > > Unit tests passing and build was successful on the branch
>> > > > > "version-0.2.0-incubating-rc3"
>> > > > >
>> > > > > +1 from me
>> > > > >
>> > > > > Cheers,
>> > > > > Kirupa
>> > > > >
>> > > > >
>> > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <wh...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > > +1 looks good to me
>> > > > > >
>> > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
>> nadavhartzvi@gmail.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > +1 approve. Tested multiple times and after a long round of
>> > fixing
>> > > > and
>> > > > > > > testing over and over.
>> > > > > > >
>> > > > > > > Cheers,
>> > > > > > > Nadav
>> > > > > > >
>> > > > > > >
>> > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io>
>> wrote:
>> > > > > > >
>> > > > > > > > Hi everyone,
>> > > > > > > >
>> > > > > > > > We have fixed the legal issues, as well as a bug found by
>> > @Nadav
>> > > > > please
>> > > > > > > > review and vote on the release candidate #3 for the version
>> > > > > > > > 0.2.0-incubating, as follows
>> > > > > > > >
>> > > > > > > > [ ] +1, Approve the release
>> > > > > > > > [ ] -1, Do not approve the release (please provide specific
>> > > > comments)
>> > > > > > > >
>> > > > > > > > The complete staging area is available for your review,
>> which
>> > > > > includes:
>> > > > > > > >
>> > > > > > > > * JIRA release notes [1],
>> > > > > > > > * the official Apache source release to be deployed to
>> > > > > dist.apache.org
>> > > > > > > > [2],
>> > > > > > > > which is signed with the key with fingerprint [3],
>> > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
>> > > > > > > > * Java artifacts were built with Gradle 3.1 and
>> OpenJDK/Oracle
>> > > JDK
>> > > > > > > > 1.8.0_151
>> > > > > > > >
>> > > > > > > > The vote will be open for at least 72 hours. It is adopted
>> by
>> > > > > majority
>> > > > > > > > approval, with at least 3 PMC affirmative votes.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Yaniv
>> > > > > > > >
>> > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> > > > > > > > projectId=12321521&version=12342793
>> > > > > > > > [2] https://dist.apache.org/repos/
>> > dist/dev/incubator/amaterasu/
>> > > > > > 0.2.0rc3/
>> > > > > > > > [3]
>> > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
>> > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Davor Bonaci <da...@apache.org>.
>
> Based on what I've seen in other projects, it seems common to release bot
> the binaries (in Amaterasu those are released in the .tar file) as well as
> a source (which are in the -source.tar.gz file)
> The main question is if we do or do not need to release both.
>

Sorry, my mistake. It is perfectly fine to release convenience binaries in
addition to source code. You may want to name them better and refer to both
from the vote email, however ;-)

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Yaniv Rodenski <ya...@shinto.io>.
Hi Davor,

Sorry for keeping my emails short and undescriptive, I have only one hand
for the next month due to a skateboarding accident.
Based on what I've seen in other projects, it seems common to release bot
the binaries (in Amaterasu those are released in the .tar file) as well as
a source (which are in the -source.tar.gz file)
The main question is if we do or do not need to release both.

Cheers,
Yaniv

On Sat, Jun 2, 2018 at 1:13 AM, Davor Bonaci <db...@gmail.com> wrote:

> No problem with .tar.gz. But, you don't need .tar then.
>
> On Fri, Jun 1, 2018 at 12:04 AM, Yaniv Rodenski <ya...@shinto.io> wrote:
>
> > One thing to clarify, the tar.gz is the source code as I've seen this is
> > released in other projects. if that is not needed I'm happy to drop it
> from
> > future releases.
> >
> > On Fri, Jun 1, 2018 at 5:01 PM, Yaniv Rodenski <ya...@shinto.io> wrote:
> >
> > > Thanks Davor,
> > >
> > > We will fix those for the next release.
> > >
> > > Cheers,
> > > Yaniv
> > >
> > > On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci <db...@gmail.com>
> wrote:
> > >
> > >> Great improvement!
> > >>
> > >> license/notice/disclaimer: ok
> > >> headers: ~fine (missing in some non-code files, it would be great to
> add
> > >> those -- non-blocking)
> > >> binaries: there's a gradle binary, but that may be ruled as acceptable
> > >> (non-blocking from me)
> > >>
> > >> Not sure why we need both .tar and .tar.gz. I'd get rid of one of
> them.
> > >> Also, shipping private keys in the distribution looks weird as well.
> > >>
> > >> For the first release, this is good. So, +1 to send onwards for an
> > >> Incubator review, and polish a few of these things for a subsequent
> > >> release.
> > >>
> > >> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski <ya...@shinto.io>
> > wrote:
> > >>
> > >> > Hi All,
> > >> >
> > >> > Just a kind reminder for our mentors, we need your vote on this one
> :)
> > >> >
> > >> > Cheers,
> > >> > Yaniv
> > >> >
> > >> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <ar...@arunma.com>
> > >> wrote:
> > >> >
> > >> > > Thanks a lot, Nadav. Will get home and spend some more time on
> > this. I
> > >> > was
> > >> > > in a rush and did this poor workaround. My VM is just 8 GB.
> > >> > >
> > >> > > Cheers
> > >> > > Arun
> > >> > >
> > >> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <
> nadavhartzvi@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP
> > >> docker
> > >> > on
> > >> > > > AWS). Both Spark-Scala and PySpark.
> > >> > > > It worked on Yaniv's HDP cluster as well.
> > >> > > > Worth noting:
> > >> > > > 1. HDP 2.6.4
> > >> > > > 2. Cluster has total of 32GB memory available
> > >> > > > 3. Each container is allocated 1G memory.
> > >> > > > 4. Amaterasu.properties:
> > >> > > >
> > >> > > > zk=sandbox-hdp.hortonworks.com
> > >> > > > version=0.2.0-incubating-rc3
> > >> > > > master=192.168.33.11
> > >> > > > user=root
> > >> > > > mode=yarn
> > >> > > > webserver.port=8000
> > >> > > > webserver.root=dist
> > >> > > > spark.version=2.6.4.0-91
> > >> > > > yarn.queue=default
> > >> > > > yarn.jarspath=hdfs:///apps/amaterasu
> > >> > > > spark.home=/usr/hdp/current/spark2-client
> > >> > > >
> > >> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.
> > >> > cloudera2-1.cdh5.7.0.p0.
> > >> > > 171658/lib/spark2
> > >> > > > yarn.hadoop.home.dir=/etc/hadoop
> > >> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.
> > >> 6.4.0-91"
> > >> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.
> > version=2.6.4.0-91"
> > >> > > >
> > >> > > >
> > >> > > > Arun, please share:
> > >> > > > 1. YARN memory configurations
> > >> > > > 2. amaterasu.properties content
> > >> > > > 3. HDP version.
> > >> > > >
> > >> > > > Cheers,
> > >> > > > Nadav
> > >> > > >
> > >> > > >
> > >> > > > On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com>
> wrote:
> > >> > > >
> > >> > > > > The pmem disabling is just temporary. I'll do a detailed
> > analysis
> > >> and
> > >> > > get
> > >> > > > > back with a proper solution.
> > >> > > > >
> > >> > > > > Any hints on this front is highly appreciated.
> > >> > > > >
> > >> > > > > Cheers
> > >> > > > > Arun
> > >> > > > >
> > >> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <
> > >> nadavhartzvi@gmail.com>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > Yaniv, Eyal, this might be related to the same issue you
> faced
> > >> with
> > >> > > > HDP.
> > >> > > > > > Can you confirm?
> > >> > > > > >
> > >> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <
> arun@arunma.com>
> > >> > wrote:
> > >> > > > > >
> > >> > > > > > > +1 from me
> > >> > > > > > >
> > >> > > > > > > Unit Tests and Build ran fine.
> > >> > > > > > >
> > >> > > > > > > Tested on HDP (VM) but had trouble allocating containers
> > >> (didn't
> > >> > > have
> > >> > > > > > that
> > >> > > > > > > before).  Apparently Centos VMs are known to have this
> > >> problem.
> > >> > > > > Disabled
> > >> > > > > > > physical memory check  (yarn.nodemanager.pmem-check-
> > enabled)
> > >> and
> > >> > > ran
> > >> > > > > jobs
> > >> > > > > > > successfully.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > >> > > > > > kirupagaran.d@gmail.com
> > >> > > > > > > >
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Unit tests passing and build was successful on the
> branch
> > >> > > > > > > > "version-0.2.0-incubating-rc3"
> > >> > > > > > > >
> > >> > > > > > > > +1 from me
> > >> > > > > > > >
> > >> > > > > > > > Cheers,
> > >> > > > > > > > Kirupa
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
> > >> whisrael@gmail.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > +1 looks good to me
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> > >> > > > nadavhartzvi@gmail.com
> > >> > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > +1 approve. Tested multiple times and after a long
> > >> round of
> > >> > > > > fixing
> > >> > > > > > > and
> > >> > > > > > > > > > testing over and over.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Cheers,
> > >> > > > > > > > > > Nadav
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <
> > >> yaniv@shinto.io>
> > >> > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi everyone,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > We have fixed the legal issues, as well as a bug
> > >> found by
> > >> > > > > @Nadav
> > >> > > > > > > > please
> > >> > > > > > > > > > > review and vote on the release candidate #3 for
> the
> > >> > version
> > >> > > > > > > > > > > 0.2.0-incubating, as follows
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > [ ] +1, Approve the release
> > >> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
> > >> > specific
> > >> > > > > > > comments)
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > The complete staging area is available for your
> > >> review,
> > >> > > which
> > >> > > > > > > > includes:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > * JIRA release notes [1],
> > >> > > > > > > > > > > * the official Apache source release to be
> deployed
> > to
> > >> > > > > > > > dist.apache.org
> > >> > > > > > > > > > > [2],
> > >> > > > > > > > > > > which is signed with the key with fingerprint [3],
> > >> > > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3"
> > [4],
> > >> > > > > > > > > > > * Java artifacts were built with Gradle 3.1 and
> > >> > > > OpenJDK/Oracle
> > >> > > > > > JDK
> > >> > > > > > > > > > > 1.8.0_151
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > The vote will be open for at least 72 hours. It is
> > >> > adopted
> > >> > > by
> > >> > > > > > > > majority
> > >> > > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > Yaniv
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > [1] https://issues.apache.org/
> > >> > jira/secure/ReleaseNote.jspa
> > >> > > ?
> > >> > > > > > > > > > > projectId=12321521&version=12342793
> > >> > > > > > > > > > > [2] https://dist.apache.org/repos/
> > >> > > > > dist/dev/incubator/amaterasu/
> > >> > > > > > > > > 0.2.0rc3/
> > >> > > > > > > > > > > [3]
> > >> > > > > > > https://dist.apache.org/repos/
> > dist/dev/incubator/amaterasu/K
> > >> EYS
> > >> > > > > > > > > > > [4] https://github.com/apache/incu
> > >> bator-amaterasu/tags
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Yaniv Rodenski
> > >> >
> > >> > +61 477 778 405
> > >> > yaniv@shinto.io
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Yaniv Rodenski
> > >
> > > +61 477 778 405
> > > yaniv@shinto.io
> > >
> >
> >
> >
> > --
> > Yaniv Rodenski
> >
> > +61 477 778 405
> > yaniv@shinto.io
> >
>



-- 
Yaniv Rodenski

+61 477 778 405
yaniv@shinto.io

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Davor Bonaci <db...@gmail.com>.
No problem with .tar.gz. But, you don't need .tar then.

On Fri, Jun 1, 2018 at 12:04 AM, Yaniv Rodenski <ya...@shinto.io> wrote:

> One thing to clarify, the tar.gz is the source code as I've seen this is
> released in other projects. if that is not needed I'm happy to drop it from
> future releases.
>
> On Fri, Jun 1, 2018 at 5:01 PM, Yaniv Rodenski <ya...@shinto.io> wrote:
>
> > Thanks Davor,
> >
> > We will fix those for the next release.
> >
> > Cheers,
> > Yaniv
> >
> > On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci <db...@gmail.com> wrote:
> >
> >> Great improvement!
> >>
> >> license/notice/disclaimer: ok
> >> headers: ~fine (missing in some non-code files, it would be great to add
> >> those -- non-blocking)
> >> binaries: there's a gradle binary, but that may be ruled as acceptable
> >> (non-blocking from me)
> >>
> >> Not sure why we need both .tar and .tar.gz. I'd get rid of one of them.
> >> Also, shipping private keys in the distribution looks weird as well.
> >>
> >> For the first release, this is good. So, +1 to send onwards for an
> >> Incubator review, and polish a few of these things for a subsequent
> >> release.
> >>
> >> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski <ya...@shinto.io>
> wrote:
> >>
> >> > Hi All,
> >> >
> >> > Just a kind reminder for our mentors, we need your vote on this one :)
> >> >
> >> > Cheers,
> >> > Yaniv
> >> >
> >> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <ar...@arunma.com>
> >> wrote:
> >> >
> >> > > Thanks a lot, Nadav. Will get home and spend some more time on
> this. I
> >> > was
> >> > > in a rush and did this poor workaround. My VM is just 8 GB.
> >> > >
> >> > > Cheers
> >> > > Arun
> >> > >
> >> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP
> >> docker
> >> > on
> >> > > > AWS). Both Spark-Scala and PySpark.
> >> > > > It worked on Yaniv's HDP cluster as well.
> >> > > > Worth noting:
> >> > > > 1. HDP 2.6.4
> >> > > > 2. Cluster has total of 32GB memory available
> >> > > > 3. Each container is allocated 1G memory.
> >> > > > 4. Amaterasu.properties:
> >> > > >
> >> > > > zk=sandbox-hdp.hortonworks.com
> >> > > > version=0.2.0-incubating-rc3
> >> > > > master=192.168.33.11
> >> > > > user=root
> >> > > > mode=yarn
> >> > > > webserver.port=8000
> >> > > > webserver.root=dist
> >> > > > spark.version=2.6.4.0-91
> >> > > > yarn.queue=default
> >> > > > yarn.jarspath=hdfs:///apps/amaterasu
> >> > > > spark.home=/usr/hdp/current/spark2-client
> >> > > >
> >> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.
> >> > cloudera2-1.cdh5.7.0.p0.
> >> > > 171658/lib/spark2
> >> > > > yarn.hadoop.home.dir=/etc/hadoop
> >> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.
> >> 6.4.0-91"
> >> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.
> version=2.6.4.0-91"
> >> > > >
> >> > > >
> >> > > > Arun, please share:
> >> > > > 1. YARN memory configurations
> >> > > > 2. amaterasu.properties content
> >> > > > 3. HDP version.
> >> > > >
> >> > > > Cheers,
> >> > > > Nadav
> >> > > >
> >> > > >
> >> > > > On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
> >> > > >
> >> > > > > The pmem disabling is just temporary. I'll do a detailed
> analysis
> >> and
> >> > > get
> >> > > > > back with a proper solution.
> >> > > > >
> >> > > > > Any hints on this front is highly appreciated.
> >> > > > >
> >> > > > > Cheers
> >> > > > > Arun
> >> > > > >
> >> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <
> >> nadavhartzvi@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > > > Yaniv, Eyal, this might be related to the same issue you faced
> >> with
> >> > > > HDP.
> >> > > > > > Can you confirm?
> >> > > > > >
> >> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
> >> > wrote:
> >> > > > > >
> >> > > > > > > +1 from me
> >> > > > > > >
> >> > > > > > > Unit Tests and Build ran fine.
> >> > > > > > >
> >> > > > > > > Tested on HDP (VM) but had trouble allocating containers
> >> (didn't
> >> > > have
> >> > > > > > that
> >> > > > > > > before).  Apparently Centos VMs are known to have this
> >> problem.
> >> > > > > Disabled
> >> > > > > > > physical memory check  (yarn.nodemanager.pmem-check-
> enabled)
> >> and
> >> > > ran
> >> > > > > jobs
> >> > > > > > > successfully.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> >> > > > > > kirupagaran.d@gmail.com
> >> > > > > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Unit tests passing and build was successful on the branch
> >> > > > > > > > "version-0.2.0-incubating-rc3"
> >> > > > > > > >
> >> > > > > > > > +1 from me
> >> > > > > > > >
> >> > > > > > > > Cheers,
> >> > > > > > > > Kirupa
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
> >> whisrael@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > +1 looks good to me
> >> > > > > > > > >
> >> > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> >> > > > nadavhartzvi@gmail.com
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > +1 approve. Tested multiple times and after a long
> >> round of
> >> > > > > fixing
> >> > > > > > > and
> >> > > > > > > > > > testing over and over.
> >> > > > > > > > > >
> >> > > > > > > > > > Cheers,
> >> > > > > > > > > > Nadav
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <
> >> yaniv@shinto.io>
> >> > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi everyone,
> >> > > > > > > > > > >
> >> > > > > > > > > > > We have fixed the legal issues, as well as a bug
> >> found by
> >> > > > > @Nadav
> >> > > > > > > > please
> >> > > > > > > > > > > review and vote on the release candidate #3 for the
> >> > version
> >> > > > > > > > > > > 0.2.0-incubating, as follows
> >> > > > > > > > > > >
> >> > > > > > > > > > > [ ] +1, Approve the release
> >> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
> >> > specific
> >> > > > > > > comments)
> >> > > > > > > > > > >
> >> > > > > > > > > > > The complete staging area is available for your
> >> review,
> >> > > which
> >> > > > > > > > includes:
> >> > > > > > > > > > >
> >> > > > > > > > > > > * JIRA release notes [1],
> >> > > > > > > > > > > * the official Apache source release to be deployed
> to
> >> > > > > > > > dist.apache.org
> >> > > > > > > > > > > [2],
> >> > > > > > > > > > > which is signed with the key with fingerprint [3],
> >> > > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3"
> [4],
> >> > > > > > > > > > > * Java artifacts were built with Gradle 3.1 and
> >> > > > OpenJDK/Oracle
> >> > > > > > JDK
> >> > > > > > > > > > > 1.8.0_151
> >> > > > > > > > > > >
> >> > > > > > > > > > > The vote will be open for at least 72 hours. It is
> >> > adopted
> >> > > by
> >> > > > > > > > majority
> >> > > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks,
> >> > > > > > > > > > > Yaniv
> >> > > > > > > > > > >
> >> > > > > > > > > > > [1] https://issues.apache.org/
> >> > jira/secure/ReleaseNote.jspa
> >> > > ?
> >> > > > > > > > > > > projectId=12321521&version=12342793
> >> > > > > > > > > > > [2] https://dist.apache.org/repos/
> >> > > > > dist/dev/incubator/amaterasu/
> >> > > > > > > > > 0.2.0rc3/
> >> > > > > > > > > > > [3]
> >> > > > > > > https://dist.apache.org/repos/
> dist/dev/incubator/amaterasu/K
> >> EYS
> >> > > > > > > > > > > [4] https://github.com/apache/incu
> >> bator-amaterasu/tags
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Yaniv Rodenski
> >> >
> >> > +61 477 778 405
> >> > yaniv@shinto.io
> >> >
> >>
> >
> >
> >
> > --
> > Yaniv Rodenski
> >
> > +61 477 778 405
> > yaniv@shinto.io
> >
>
>
>
> --
> Yaniv Rodenski
>
> +61 477 778 405
> yaniv@shinto.io
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Yaniv Rodenski <ya...@shinto.io>.
One thing to clarify, the tar.gz is the source code as I've seen this is
released in other projects. if that is not needed I'm happy to drop it from
future releases.

On Fri, Jun 1, 2018 at 5:01 PM, Yaniv Rodenski <ya...@shinto.io> wrote:

> Thanks Davor,
>
> We will fix those for the next release.
>
> Cheers,
> Yaniv
>
> On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci <db...@gmail.com> wrote:
>
>> Great improvement!
>>
>> license/notice/disclaimer: ok
>> headers: ~fine (missing in some non-code files, it would be great to add
>> those -- non-blocking)
>> binaries: there's a gradle binary, but that may be ruled as acceptable
>> (non-blocking from me)
>>
>> Not sure why we need both .tar and .tar.gz. I'd get rid of one of them.
>> Also, shipping private keys in the distribution looks weird as well.
>>
>> For the first release, this is good. So, +1 to send onwards for an
>> Incubator review, and polish a few of these things for a subsequent
>> release.
>>
>> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski <ya...@shinto.io> wrote:
>>
>> > Hi All,
>> >
>> > Just a kind reminder for our mentors, we need your vote on this one :)
>> >
>> > Cheers,
>> > Yaniv
>> >
>> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <ar...@arunma.com>
>> wrote:
>> >
>> > > Thanks a lot, Nadav. Will get home and spend some more time on this. I
>> > was
>> > > in a rush and did this poor workaround. My VM is just 8 GB.
>> > >
>> > > Cheers
>> > > Arun
>> > >
>> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com>
>> > wrote:
>> > >
>> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP
>> docker
>> > on
>> > > > AWS). Both Spark-Scala and PySpark.
>> > > > It worked on Yaniv's HDP cluster as well.
>> > > > Worth noting:
>> > > > 1. HDP 2.6.4
>> > > > 2. Cluster has total of 32GB memory available
>> > > > 3. Each container is allocated 1G memory.
>> > > > 4. Amaterasu.properties:
>> > > >
>> > > > zk=sandbox-hdp.hortonworks.com
>> > > > version=0.2.0-incubating-rc3
>> > > > master=192.168.33.11
>> > > > user=root
>> > > > mode=yarn
>> > > > webserver.port=8000
>> > > > webserver.root=dist
>> > > > spark.version=2.6.4.0-91
>> > > > yarn.queue=default
>> > > > yarn.jarspath=hdfs:///apps/amaterasu
>> > > > spark.home=/usr/hdp/current/spark2-client
>> > > >
>> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.
>> > cloudera2-1.cdh5.7.0.p0.
>> > > 171658/lib/spark2
>> > > > yarn.hadoop.home.dir=/etc/hadoop
>> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.
>> 6.4.0-91"
>> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
>> > > >
>> > > >
>> > > > Arun, please share:
>> > > > 1. YARN memory configurations
>> > > > 2. amaterasu.properties content
>> > > > 3. HDP version.
>> > > >
>> > > > Cheers,
>> > > > Nadav
>> > > >
>> > > >
>> > > > On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
>> > > >
>> > > > > The pmem disabling is just temporary. I'll do a detailed analysis
>> and
>> > > get
>> > > > > back with a proper solution.
>> > > > >
>> > > > > Any hints on this front is highly appreciated.
>> > > > >
>> > > > > Cheers
>> > > > > Arun
>> > > > >
>> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <
>> nadavhartzvi@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > > Yaniv, Eyal, this might be related to the same issue you faced
>> with
>> > > > HDP.
>> > > > > > Can you confirm?
>> > > > > >
>> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
>> > wrote:
>> > > > > >
>> > > > > > > +1 from me
>> > > > > > >
>> > > > > > > Unit Tests and Build ran fine.
>> > > > > > >
>> > > > > > > Tested on HDP (VM) but had trouble allocating containers
>> (didn't
>> > > have
>> > > > > > that
>> > > > > > > before).  Apparently Centos VMs are known to have this
>> problem.
>> > > > > Disabled
>> > > > > > > physical memory check  (yarn.nodemanager.pmem-check-enabled)
>> and
>> > > ran
>> > > > > jobs
>> > > > > > > successfully.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
>> > > > > > kirupagaran.d@gmail.com
>> > > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Unit tests passing and build was successful on the branch
>> > > > > > > > "version-0.2.0-incubating-rc3"
>> > > > > > > >
>> > > > > > > > +1 from me
>> > > > > > > >
>> > > > > > > > Cheers,
>> > > > > > > > Kirupa
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
>> whisrael@gmail.com
>> > >
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > +1 looks good to me
>> > > > > > > > >
>> > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
>> > > > nadavhartzvi@gmail.com
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > +1 approve. Tested multiple times and after a long
>> round of
>> > > > > fixing
>> > > > > > > and
>> > > > > > > > > > testing over and over.
>> > > > > > > > > >
>> > > > > > > > > > Cheers,
>> > > > > > > > > > Nadav
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <
>> yaniv@shinto.io>
>> > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi everyone,
>> > > > > > > > > > >
>> > > > > > > > > > > We have fixed the legal issues, as well as a bug
>> found by
>> > > > > @Nadav
>> > > > > > > > please
>> > > > > > > > > > > review and vote on the release candidate #3 for the
>> > version
>> > > > > > > > > > > 0.2.0-incubating, as follows
>> > > > > > > > > > >
>> > > > > > > > > > > [ ] +1, Approve the release
>> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
>> > specific
>> > > > > > > comments)
>> > > > > > > > > > >
>> > > > > > > > > > > The complete staging area is available for your
>> review,
>> > > which
>> > > > > > > > includes:
>> > > > > > > > > > >
>> > > > > > > > > > > * JIRA release notes [1],
>> > > > > > > > > > > * the official Apache source release to be deployed to
>> > > > > > > > dist.apache.org
>> > > > > > > > > > > [2],
>> > > > > > > > > > > which is signed with the key with fingerprint [3],
>> > > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
>> > > > > > > > > > > * Java artifacts were built with Gradle 3.1 and
>> > > > OpenJDK/Oracle
>> > > > > > JDK
>> > > > > > > > > > > 1.8.0_151
>> > > > > > > > > > >
>> > > > > > > > > > > The vote will be open for at least 72 hours. It is
>> > adopted
>> > > by
>> > > > > > > > majority
>> > > > > > > > > > > approval, with at least 3 PMC affirmative votes.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > > Yaniv
>> > > > > > > > > > >
>> > > > > > > > > > > [1] https://issues.apache.org/
>> > jira/secure/ReleaseNote.jspa
>> > > ?
>> > > > > > > > > > > projectId=12321521&version=12342793
>> > > > > > > > > > > [2] https://dist.apache.org/repos/
>> > > > > dist/dev/incubator/amaterasu/
>> > > > > > > > > 0.2.0rc3/
>> > > > > > > > > > > [3]
>> > > > > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/K
>> EYS
>> > > > > > > > > > > [4] https://github.com/apache/incu
>> bator-amaterasu/tags
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Yaniv Rodenski
>> >
>> > +61 477 778 405
>> > yaniv@shinto.io
>> >
>>
>
>
>
> --
> Yaniv Rodenski
>
> +61 477 778 405
> yaniv@shinto.io
>



-- 
Yaniv Rodenski

+61 477 778 405
yaniv@shinto.io

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Yaniv Rodenski <ya...@shinto.io>.
Thanks Davor,

We will fix those for the next release.

Cheers,
Yaniv

On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci <db...@gmail.com> wrote:

> Great improvement!
>
> license/notice/disclaimer: ok
> headers: ~fine (missing in some non-code files, it would be great to add
> those -- non-blocking)
> binaries: there's a gradle binary, but that may be ruled as acceptable
> (non-blocking from me)
>
> Not sure why we need both .tar and .tar.gz. I'd get rid of one of them.
> Also, shipping private keys in the distribution looks weird as well.
>
> For the first release, this is good. So, +1 to send onwards for an
> Incubator review, and polish a few of these things for a subsequent
> release.
>
> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski <ya...@shinto.io> wrote:
>
> > Hi All,
> >
> > Just a kind reminder for our mentors, we need your vote on this one :)
> >
> > Cheers,
> > Yaniv
> >
> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <ar...@arunma.com>
> wrote:
> >
> > > Thanks a lot, Nadav. Will get home and spend some more time on this. I
> > was
> > > in a rush and did this poor workaround. My VM is just 8 GB.
> > >
> > > Cheers
> > > Arun
> > >
> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com>
> > wrote:
> > >
> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP
> docker
> > on
> > > > AWS). Both Spark-Scala and PySpark.
> > > > It worked on Yaniv's HDP cluster as well.
> > > > Worth noting:
> > > > 1. HDP 2.6.4
> > > > 2. Cluster has total of 32GB memory available
> > > > 3. Each container is allocated 1G memory.
> > > > 4. Amaterasu.properties:
> > > >
> > > > zk=sandbox-hdp.hortonworks.com
> > > > version=0.2.0-incubating-rc3
> > > > master=192.168.33.11
> > > > user=root
> > > > mode=yarn
> > > > webserver.port=8000
> > > > webserver.root=dist
> > > > spark.version=2.6.4.0-91
> > > > yarn.queue=default
> > > > yarn.jarspath=hdfs:///apps/amaterasu
> > > > spark.home=/usr/hdp/current/spark2-client
> > > >
> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.
> > cloudera2-1.cdh5.7.0.p0.
> > > 171658/lib/spark2
> > > > yarn.hadoop.home.dir=/etc/hadoop
> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > > >
> > > >
> > > > Arun, please share:
> > > > 1. YARN memory configurations
> > > > 2. amaterasu.properties content
> > > > 3. HDP version.
> > > >
> > > > Cheers,
> > > > Nadav
> > > >
> > > >
> > > > On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
> > > >
> > > > > The pmem disabling is just temporary. I'll do a detailed analysis
> and
> > > get
> > > > > back with a proper solution.
> > > > >
> > > > > Any hints on this front is highly appreciated.
> > > > >
> > > > > Cheers
> > > > > Arun
> > > > >
> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <nadavhartzvi@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Yaniv, Eyal, this might be related to the same issue you faced
> with
> > > > HDP.
> > > > > > Can you confirm?
> > > > > >
> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
> > wrote:
> > > > > >
> > > > > > > +1 from me
> > > > > > >
> > > > > > > Unit Tests and Build ran fine.
> > > > > > >
> > > > > > > Tested on HDP (VM) but had trouble allocating containers
> (didn't
> > > have
> > > > > > that
> > > > > > > before).  Apparently Centos VMs are known to have this problem.
> > > > > Disabled
> > > > > > > physical memory check  (yarn.nodemanager.pmem-check-enabled)
> and
> > > ran
> > > > > jobs
> > > > > > > successfully.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > > > > > kirupagaran.d@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Unit tests passing and build was successful on the branch
> > > > > > > > "version-0.2.0-incubating-rc3"
> > > > > > > >
> > > > > > > > +1 from me
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Kirupa
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <
> whisrael@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 looks good to me
> > > > > > > > >
> > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> > > > nadavhartzvi@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 approve. Tested multiple times and after a long round
> of
> > > > > fixing
> > > > > > > and
> > > > > > > > > > testing over and over.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Nadav
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <yaniv@shinto.io
> >
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > We have fixed the legal issues, as well as a bug found
> by
> > > > > @Nadav
> > > > > > > > please
> > > > > > > > > > > review and vote on the release candidate #3 for the
> > version
> > > > > > > > > > > 0.2.0-incubating, as follows
> > > > > > > > > > >
> > > > > > > > > > > [ ] +1, Approve the release
> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
> > specific
> > > > > > > comments)
> > > > > > > > > > >
> > > > > > > > > > > The complete staging area is available for your review,
> > > which
> > > > > > > > includes:
> > > > > > > > > > >
> > > > > > > > > > > * JIRA release notes [1],
> > > > > > > > > > > * the official Apache source release to be deployed to
> > > > > > > > dist.apache.org
> > > > > > > > > > > [2],
> > > > > > > > > > > which is signed with the key with fingerprint [3],
> > > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > > > > > > > > > * Java artifacts were built with Gradle 3.1 and
> > > > OpenJDK/Oracle
> > > > > > JDK
> > > > > > > > > > > 1.8.0_151
> > > > > > > > > > >
> > > > > > > > > > > The vote will be open for at least 72 hours. It is
> > adopted
> > > by
> > > > > > > > majority
> > > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Yaniv
> > > > > > > > > > >
> > > > > > > > > > > [1] https://issues.apache.org/
> > jira/secure/ReleaseNote.jspa
> > > ?
> > > > > > > > > > > projectId=12321521&version=12342793
> > > > > > > > > > > [2] https://dist.apache.org/repos/
> > > > > dist/dev/incubator/amaterasu/
> > > > > > > > > 0.2.0rc3/
> > > > > > > > > > > [3]
> > > > > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/
> KEYS
> > > > > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Yaniv Rodenski
> >
> > +61 477 778 405
> > yaniv@shinto.io
> >
>



-- 
Yaniv Rodenski

+61 477 778 405
yaniv@shinto.io

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Davor Bonaci <db...@gmail.com>.
Great improvement!

license/notice/disclaimer: ok
headers: ~fine (missing in some non-code files, it would be great to add
those -- non-blocking)
binaries: there's a gradle binary, but that may be ruled as acceptable
(non-blocking from me)

Not sure why we need both .tar and .tar.gz. I'd get rid of one of them.
Also, shipping private keys in the distribution looks weird as well.

For the first release, this is good. So, +1 to send onwards for an
Incubator review, and polish a few of these things for a subsequent release.

On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski <ya...@shinto.io> wrote:

> Hi All,
>
> Just a kind reminder for our mentors, we need your vote on this one :)
>
> Cheers,
> Yaniv
>
> On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <ar...@arunma.com> wrote:
>
> > Thanks a lot, Nadav. Will get home and spend some more time on this. I
> was
> > in a rush and did this poor workaround. My VM is just 8 GB.
> >
> > Cheers
> > Arun
> >
> > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com>
> wrote:
> >
> > > Yaniv and I just tested it. It worked flawlessly on my end (HDP docker
> on
> > > AWS). Both Spark-Scala and PySpark.
> > > It worked on Yaniv's HDP cluster as well.
> > > Worth noting:
> > > 1. HDP 2.6.4
> > > 2. Cluster has total of 32GB memory available
> > > 3. Each container is allocated 1G memory.
> > > 4. Amaterasu.properties:
> > >
> > > zk=sandbox-hdp.hortonworks.com
> > > version=0.2.0-incubating-rc3
> > > master=192.168.33.11
> > > user=root
> > > mode=yarn
> > > webserver.port=8000
> > > webserver.root=dist
> > > spark.version=2.6.4.0-91
> > > yarn.queue=default
> > > yarn.jarspath=hdfs:///apps/amaterasu
> > > spark.home=/usr/hdp/current/spark2-client
> > >
> > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.
> cloudera2-1.cdh5.7.0.p0.
> > 171658/lib/spark2
> > > yarn.hadoop.home.dir=/etc/hadoop
> > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > >
> > >
> > > Arun, please share:
> > > 1. YARN memory configurations
> > > 2. amaterasu.properties content
> > > 3. HDP version.
> > >
> > > Cheers,
> > > Nadav
> > >
> > >
> > > On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
> > >
> > > > The pmem disabling is just temporary. I'll do a detailed analysis and
> > get
> > > > back with a proper solution.
> > > >
> > > > Any hints on this front is highly appreciated.
> > > >
> > > > Cheers
> > > > Arun
> > > >
> > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <na...@gmail.com>
> > > wrote:
> > > >
> > > > > Yaniv, Eyal, this might be related to the same issue you faced with
> > > HDP.
> > > > > Can you confirm?
> > > > >
> > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com>
> wrote:
> > > > >
> > > > > > +1 from me
> > > > > >
> > > > > > Unit Tests and Build ran fine.
> > > > > >
> > > > > > Tested on HDP (VM) but had trouble allocating containers (didn't
> > have
> > > > > that
> > > > > > before).  Apparently Centos VMs are known to have this problem.
> > > > Disabled
> > > > > > physical memory check  (yarn.nodemanager.pmem-check-enabled) and
> > ran
> > > > jobs
> > > > > > successfully.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > > > > kirupagaran.d@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Unit tests passing and build was successful on the branch
> > > > > > > "version-0.2.0-incubating-rc3"
> > > > > > >
> > > > > > > +1 from me
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Kirupa
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <whisrael@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > +1 looks good to me
> > > > > > > >
> > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> > > nadavhartzvi@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 approve. Tested multiple times and after a long round of
> > > > fixing
> > > > > > and
> > > > > > > > > testing over and over.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Nadav
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io>
> > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > We have fixed the legal issues, as well as a bug found by
> > > > @Nadav
> > > > > > > please
> > > > > > > > > > review and vote on the release candidate #3 for the
> version
> > > > > > > > > > 0.2.0-incubating, as follows
> > > > > > > > > >
> > > > > > > > > > [ ] +1, Approve the release
> > > > > > > > > > [ ] -1, Do not approve the release (please provide
> specific
> > > > > > comments)
> > > > > > > > > >
> > > > > > > > > > The complete staging area is available for your review,
> > which
> > > > > > > includes:
> > > > > > > > > >
> > > > > > > > > > * JIRA release notes [1],
> > > > > > > > > > * the official Apache source release to be deployed to
> > > > > > > dist.apache.org
> > > > > > > > > > [2],
> > > > > > > > > > which is signed with the key with fingerprint [3],
> > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > > > > > > > > * Java artifacts were built with Gradle 3.1 and
> > > OpenJDK/Oracle
> > > > > JDK
> > > > > > > > > > 1.8.0_151
> > > > > > > > > >
> > > > > > > > > > The vote will be open for at least 72 hours. It is
> adopted
> > by
> > > > > > > majority
> > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Yaniv
> > > > > > > > > >
> > > > > > > > > > [1] https://issues.apache.org/
> jira/secure/ReleaseNote.jspa
> > ?
> > > > > > > > > > projectId=12321521&version=12342793
> > > > > > > > > > [2] https://dist.apache.org/repos/
> > > > dist/dev/incubator/amaterasu/
> > > > > > > > 0.2.0rc3/
> > > > > > > > > > [3]
> > > > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > > > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Yaniv Rodenski
>
> +61 477 778 405
> yaniv@shinto.io
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Yaniv Rodenski <ya...@shinto.io>.
Hi All,

Just a kind reminder for our mentors, we need your vote on this one :)

Cheers,
Yaniv

On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <ar...@arunma.com> wrote:

> Thanks a lot, Nadav. Will get home and spend some more time on this. I was
> in a rush and did this poor workaround. My VM is just 8 GB.
>
> Cheers
> Arun
>
> On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com> wrote:
>
> > Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on
> > AWS). Both Spark-Scala and PySpark.
> > It worked on Yaniv's HDP cluster as well.
> > Worth noting:
> > 1. HDP 2.6.4
> > 2. Cluster has total of 32GB memory available
> > 3. Each container is allocated 1G memory.
> > 4. Amaterasu.properties:
> >
> > zk=sandbox-hdp.hortonworks.com
> > version=0.2.0-incubating-rc3
> > master=192.168.33.11
> > user=root
> > mode=yarn
> > webserver.port=8000
> > webserver.root=dist
> > spark.version=2.6.4.0-91
> > yarn.queue=default
> > yarn.jarspath=hdfs:///apps/amaterasu
> > spark.home=/usr/hdp/current/spark2-client
> >
> > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.
> 171658/lib/spark2
> > yarn.hadoop.home.dir=/etc/hadoop
> > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> >
> >
> > Arun, please share:
> > 1. YARN memory configurations
> > 2. amaterasu.properties content
> > 3. HDP version.
> >
> > Cheers,
> > Nadav
> >
> >
> > On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
> >
> > > The pmem disabling is just temporary. I'll do a detailed analysis and
> get
> > > back with a proper solution.
> > >
> > > Any hints on this front is highly appreciated.
> > >
> > > Cheers
> > > Arun
> > >
> > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <na...@gmail.com>
> > wrote:
> > >
> > > > Yaniv, Eyal, this might be related to the same issue you faced with
> > HDP.
> > > > Can you confirm?
> > > >
> > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com> wrote:
> > > >
> > > > > +1 from me
> > > > >
> > > > > Unit Tests and Build ran fine.
> > > > >
> > > > > Tested on HDP (VM) but had trouble allocating containers (didn't
> have
> > > > that
> > > > > before).  Apparently Centos VMs are known to have this problem.
> > > Disabled
> > > > > physical memory check  (yarn.nodemanager.pmem-check-enabled) and
> ran
> > > jobs
> > > > > successfully.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > > > kirupagaran.d@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Unit tests passing and build was successful on the branch
> > > > > > "version-0.2.0-incubating-rc3"
> > > > > >
> > > > > > +1 from me
> > > > > >
> > > > > > Cheers,
> > > > > > Kirupa
> > > > > >
> > > > > >
> > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <wh...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > +1 looks good to me
> > > > > > >
> > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> > nadavhartzvi@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > +1 approve. Tested multiple times and after a long round of
> > > fixing
> > > > > and
> > > > > > > > testing over and over.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Nadav
> > > > > > > >
> > > > > > > >
> > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io>
> > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > We have fixed the legal issues, as well as a bug found by
> > > @Nadav
> > > > > > please
> > > > > > > > > review and vote on the release candidate #3 for the version
> > > > > > > > > 0.2.0-incubating, as follows
> > > > > > > > >
> > > > > > > > > [ ] +1, Approve the release
> > > > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > > > comments)
> > > > > > > > >
> > > > > > > > > The complete staging area is available for your review,
> which
> > > > > > includes:
> > > > > > > > >
> > > > > > > > > * JIRA release notes [1],
> > > > > > > > > * the official Apache source release to be deployed to
> > > > > > dist.apache.org
> > > > > > > > > [2],
> > > > > > > > > which is signed with the key with fingerprint [3],
> > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > > > > > > > * Java artifacts were built with Gradle 3.1 and
> > OpenJDK/Oracle
> > > > JDK
> > > > > > > > > 1.8.0_151
> > > > > > > > >
> > > > > > > > > The vote will be open for at least 72 hours. It is adopted
> by
> > > > > > majority
> > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Yaniv
> > > > > > > > >
> > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa
> ?
> > > > > > > > > projectId=12321521&version=12342793
> > > > > > > > > [2] https://dist.apache.org/repos/
> > > dist/dev/incubator/amaterasu/
> > > > > > > 0.2.0rc3/
> > > > > > > > > [3]
> > > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Yaniv Rodenski

+61 477 778 405
yaniv@shinto.io

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Arun Manivannan <ar...@arunma.com>.
Thanks a lot, Nadav. Will get home and spend some more time on this. I was
in a rush and did this poor workaround. My VM is just 8 GB.

Cheers
Arun

On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <na...@gmail.com> wrote:

> Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on
> AWS). Both Spark-Scala and PySpark.
> It worked on Yaniv's HDP cluster as well.
> Worth noting:
> 1. HDP 2.6.4
> 2. Cluster has total of 32GB memory available
> 3. Each container is allocated 1G memory.
> 4. Amaterasu.properties:
>
> zk=sandbox-hdp.hortonworks.com
> version=0.2.0-incubating-rc3
> master=192.168.33.11
> user=root
> mode=yarn
> webserver.port=8000
> webserver.root=dist
> spark.version=2.6.4.0-91
> yarn.queue=default
> yarn.jarspath=hdfs:///apps/amaterasu
> spark.home=/usr/hdp/current/spark2-client
>
> #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2
> yarn.hadoop.home.dir=/etc/hadoop
> spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
>
>
> Arun, please share:
> 1. YARN memory configurations
> 2. amaterasu.properties content
> 3. HDP version.
>
> Cheers,
> Nadav
>
>
> On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:
>
> > The pmem disabling is just temporary. I'll do a detailed analysis and get
> > back with a proper solution.
> >
> > Any hints on this front is highly appreciated.
> >
> > Cheers
> > Arun
> >
> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <na...@gmail.com>
> wrote:
> >
> > > Yaniv, Eyal, this might be related to the same issue you faced with
> HDP.
> > > Can you confirm?
> > >
> > > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com> wrote:
> > >
> > > > +1 from me
> > > >
> > > > Unit Tests and Build ran fine.
> > > >
> > > > Tested on HDP (VM) but had trouble allocating containers (didn't have
> > > that
> > > > before).  Apparently Centos VMs are known to have this problem.
> > Disabled
> > > > physical memory check  (yarn.nodemanager.pmem-check-enabled) and ran
> > jobs
> > > > successfully.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > > kirupagaran.d@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Unit tests passing and build was successful on the branch
> > > > > "version-0.2.0-incubating-rc3"
> > > > >
> > > > > +1 from me
> > > > >
> > > > > Cheers,
> > > > > Kirupa
> > > > >
> > > > >
> > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <wh...@gmail.com>
> > wrote:
> > > > >
> > > > > > +1 looks good to me
> > > > > >
> > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <
> nadavhartzvi@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > +1 approve. Tested multiple times and after a long round of
> > fixing
> > > > and
> > > > > > > testing over and over.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Nadav
> > > > > > >
> > > > > > >
> > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io>
> wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > We have fixed the legal issues, as well as a bug found by
> > @Nadav
> > > > > please
> > > > > > > > review and vote on the release candidate #3 for the version
> > > > > > > > 0.2.0-incubating, as follows
> > > > > > > >
> > > > > > > > [ ] +1, Approve the release
> > > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > > comments)
> > > > > > > >
> > > > > > > > The complete staging area is available for your review, which
> > > > > includes:
> > > > > > > >
> > > > > > > > * JIRA release notes [1],
> > > > > > > > * the official Apache source release to be deployed to
> > > > > dist.apache.org
> > > > > > > > [2],
> > > > > > > > which is signed with the key with fingerprint [3],
> > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > > > > > > * Java artifacts were built with Gradle 3.1 and
> OpenJDK/Oracle
> > > JDK
> > > > > > > > 1.8.0_151
> > > > > > > >
> > > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > > > majority
> > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Yaniv
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > > > > > projectId=12321521&version=12342793
> > > > > > > > [2] https://dist.apache.org/repos/
> > dist/dev/incubator/amaterasu/
> > > > > > 0.2.0rc3/
> > > > > > > > [3]
> > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Nadav Har Tzvi <na...@gmail.com>.
Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on
AWS). Both Spark-Scala and PySpark.
It worked on Yaniv's HDP cluster as well.
Worth noting:
1. HDP 2.6.4
2. Cluster has total of 32GB memory available
3. Each container is allocated 1G memory.
4. Amaterasu.properties:

zk=sandbox-hdp.hortonworks.com
version=0.2.0-incubating-rc3
master=192.168.33.11
user=root
mode=yarn
webserver.port=8000
webserver.root=dist
spark.version=2.6.4.0-91
yarn.queue=default
yarn.jarspath=hdfs:///apps/amaterasu
spark.home=/usr/hdp/current/spark2-client
#spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2
yarn.hadoop.home.dir=/etc/hadoop
spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91"
spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91"


Arun, please share:
1. YARN memory configurations
2. amaterasu.properties content
3. HDP version.

Cheers,
Nadav


On 30 May 2018 at 07:11, Arun Manivannan <ar...@arunma.com> wrote:

> The pmem disabling is just temporary. I'll do a detailed analysis and get
> back with a proper solution.
>
> Any hints on this front is highly appreciated.
>
> Cheers
> Arun
>
> On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <na...@gmail.com> wrote:
>
> > Yaniv, Eyal, this might be related to the same issue you faced with HDP.
> > Can you confirm?
> >
> > On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com> wrote:
> >
> > > +1 from me
> > >
> > > Unit Tests and Build ran fine.
> > >
> > > Tested on HDP (VM) but had trouble allocating containers (didn't have
> > that
> > > before).  Apparently Centos VMs are known to have this problem.
> Disabled
> > > physical memory check  (yarn.nodemanager.pmem-check-enabled) and ran
> jobs
> > > successfully.
> > >
> > >
> > >
> > >
> > >
> > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> > kirupagaran.d@gmail.com
> > > >
> > > wrote:
> > >
> > > > Unit tests passing and build was successful on the branch
> > > > "version-0.2.0-incubating-rc3"
> > > >
> > > > +1 from me
> > > >
> > > > Cheers,
> > > > Kirupa
> > > >
> > > >
> > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <wh...@gmail.com>
> wrote:
> > > >
> > > > > +1 looks good to me
> > > > >
> > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <nadavhartzvi@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > +1 approve. Tested multiple times and after a long round of
> fixing
> > > and
> > > > > > testing over and over.
> > > > > >
> > > > > > Cheers,
> > > > > > Nadav
> > > > > >
> > > > > >
> > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io> wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > We have fixed the legal issues, as well as a bug found by
> @Nadav
> > > > please
> > > > > > > review and vote on the release candidate #3 for the version
> > > > > > > 0.2.0-incubating, as follows
> > > > > > >
> > > > > > > [ ] +1, Approve the release
> > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > comments)
> > > > > > >
> > > > > > > The complete staging area is available for your review, which
> > > > includes:
> > > > > > >
> > > > > > > * JIRA release notes [1],
> > > > > > > * the official Apache source release to be deployed to
> > > > dist.apache.org
> > > > > > > [2],
> > > > > > > which is signed with the key with fingerprint [3],
> > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle
> > JDK
> > > > > > > 1.8.0_151
> > > > > > >
> > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > > majority
> > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yaniv
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > > > > projectId=12321521&version=12342793
> > > > > > > [2] https://dist.apache.org/repos/
> dist/dev/incubator/amaterasu/
> > > > > 0.2.0rc3/
> > > > > > > [3]
> > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Arun Manivannan <ar...@arunma.com>.
The pmem disabling is just temporary. I'll do a detailed analysis and get
back with a proper solution.

Any hints on this front is highly appreciated.

Cheers
Arun

On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <na...@gmail.com> wrote:

> Yaniv, Eyal, this might be related to the same issue you faced with HDP.
> Can you confirm?
>
> On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com> wrote:
>
> > +1 from me
> >
> > Unit Tests and Build ran fine.
> >
> > Tested on HDP (VM) but had trouble allocating containers (didn't have
> that
> > before).  Apparently Centos VMs are known to have this problem. Disabled
> > physical memory check  (yarn.nodemanager.pmem-check-enabled) and ran jobs
> > successfully.
> >
> >
> >
> >
> >
> > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <
> kirupagaran.d@gmail.com
> > >
> > wrote:
> >
> > > Unit tests passing and build was successful on the branch
> > > "version-0.2.0-incubating-rc3"
> > >
> > > +1 from me
> > >
> > > Cheers,
> > > Kirupa
> > >
> > >
> > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <wh...@gmail.com> wrote:
> > >
> > > > +1 looks good to me
> > > >
> > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <na...@gmail.com>
> > > wrote:
> > > >
> > > > > +1 approve. Tested multiple times and after a long round of fixing
> > and
> > > > > testing over and over.
> > > > >
> > > > > Cheers,
> > > > > Nadav
> > > > >
> > > > >
> > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io> wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > We have fixed the legal issues, as well as a bug found by @Nadav
> > > please
> > > > > > review and vote on the release candidate #3 for the version
> > > > > > 0.2.0-incubating, as follows
> > > > > >
> > > > > > [ ] +1, Approve the release
> > > > > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > > >
> > > > > > The complete staging area is available for your review, which
> > > includes:
> > > > > >
> > > > > > * JIRA release notes [1],
> > > > > > * the official Apache source release to be deployed to
> > > dist.apache.org
> > > > > > [2],
> > > > > > which is signed with the key with fingerprint [3],
> > > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle
> JDK
> > > > > > 1.8.0_151
> > > > > >
> > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > majority
> > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > >
> > > > > > Thanks,
> > > > > > Yaniv
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > > > projectId=12321521&version=12342793
> > > > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/
> > > > 0.2.0rc3/
> > > > > > [3]
> > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > > > > > [4] https://github.com/apache/incubator-amaterasu/tags
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Nadav Har Tzvi <na...@gmail.com>.
Yaniv, Eyal, this might be related to the same issue you faced with HDP.
Can you confirm?

On Tue, May 29, 2018, 17:58 Arun Manivannan <ar...@arunma.com> wrote:

> +1 from me
>
> Unit Tests and Build ran fine.
>
> Tested on HDP (VM) but had trouble allocating containers (didn't have that
> before).  Apparently Centos VMs are known to have this problem. Disabled
> physical memory check  (yarn.nodemanager.pmem-check-enabled) and ran jobs
> successfully.
>
>
>
>
>
> On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <kirupagaran.d@gmail.com
> >
> wrote:
>
> > Unit tests passing and build was successful on the branch
> > "version-0.2.0-incubating-rc3"
> >
> > +1 from me
> >
> > Cheers,
> > Kirupa
> >
> >
> > On Tue, May 29, 2018 at 3:06 PM, guy peleg <wh...@gmail.com> wrote:
> >
> > > +1 looks good to me
> > >
> > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <na...@gmail.com>
> > wrote:
> > >
> > > > +1 approve. Tested multiple times and after a long round of fixing
> and
> > > > testing over and over.
> > > >
> > > > Cheers,
> > > > Nadav
> > > >
> > > >
> > > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io> wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > We have fixed the legal issues, as well as a bug found by @Nadav
> > please
> > > > > review and vote on the release candidate #3 for the version
> > > > > 0.2.0-incubating, as follows
> > > > >
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > > > >
> > > > > The complete staging area is available for your review, which
> > includes:
> > > > >
> > > > > * JIRA release notes [1],
> > > > > * the official Apache source release to be deployed to
> > dist.apache.org
> > > > > [2],
> > > > > which is signed with the key with fingerprint [3],
> > > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK
> > > > > 1.8.0_151
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > Thanks,
> > > > > Yaniv
> > > > >
> > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > > projectId=12321521&version=12342793
> > > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/
> > > 0.2.0rc3/
> > > > > [3]
> https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > > > > [4] https://github.com/apache/incubator-amaterasu/tags
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Arun Manivannan <ar...@arunma.com>.
+1 from me

Unit Tests and Build ran fine.

Tested on HDP (VM) but had trouble allocating containers (didn't have that
before).  Apparently Centos VMs are known to have this problem. Disabled
physical memory check  (yarn.nodemanager.pmem-check-enabled) and ran jobs
successfully.





On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan <ki...@gmail.com>
wrote:

> Unit tests passing and build was successful on the branch
> "version-0.2.0-incubating-rc3"
>
> +1 from me
>
> Cheers,
> Kirupa
>
>
> On Tue, May 29, 2018 at 3:06 PM, guy peleg <wh...@gmail.com> wrote:
>
> > +1 looks good to me
> >
> > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <na...@gmail.com>
> wrote:
> >
> > > +1 approve. Tested multiple times and after a long round of fixing and
> > > testing over and over.
> > >
> > > Cheers,
> > > Nadav
> > >
> > >
> > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > We have fixed the legal issues, as well as a bug found by @Nadav
> please
> > > > review and vote on the release candidate #3 for the version
> > > > 0.2.0-incubating, as follows
> > > >
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > >
> > > > * JIRA release notes [1],
> > > > * the official Apache source release to be deployed to
> dist.apache.org
> > > > [2],
> > > > which is signed with the key with fingerprint [3],
> > > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK
> > > > 1.8.0_151
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Yaniv
> > > >
> > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > projectId=12321521&version=12342793
> > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/
> > 0.2.0rc3/
> > > > [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > > > [4] https://github.com/apache/incubator-amaterasu/tags
> > > >
> > >
> >
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Kirupa Devarajan <ki...@gmail.com>.
Unit tests passing and build was successful on the branch
"version-0.2.0-incubating-rc3"

+1 from me

Cheers,
Kirupa


On Tue, May 29, 2018 at 3:06 PM, guy peleg <wh...@gmail.com> wrote:

> +1 looks good to me
>
> On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <na...@gmail.com> wrote:
>
> > +1 approve. Tested multiple times and after a long round of fixing and
> > testing over and over.
> >
> > Cheers,
> > Nadav
> >
> >
> > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io> wrote:
> >
> > > Hi everyone,
> > >
> > > We have fixed the legal issues, as well as a bug found by @Nadav please
> > > review and vote on the release candidate #3 for the version
> > > 0.2.0-incubating, as follows
> > >
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > >
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to dist.apache.org
> > > [2],
> > > which is signed with the key with fingerprint [3],
> > > * source code tag "version-0.2.0-incubating-rc3" [4],
> > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK
> > > 1.8.0_151
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Yaniv
> > >
> > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > projectId=12321521&version=12342793
> > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/
> 0.2.0rc3/
> > > [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > > [4] https://github.com/apache/incubator-amaterasu/tags
> > >
> >
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by guy peleg <wh...@gmail.com>.
+1 looks good to me

On Tue, May 29, 2018, 14:39 Nadav Har Tzvi <na...@gmail.com> wrote:

> +1 approve. Tested multiple times and after a long round of fixing and
> testing over and over.
>
> Cheers,
> Nadav
>
>
> On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io> wrote:
>
> > Hi everyone,
> >
> > We have fixed the legal issues, as well as a bug found by @Nadav please
> > review and vote on the release candidate #3 for the version
> > 0.2.0-incubating, as follows
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> >
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > [2],
> > which is signed with the key with fingerprint [3],
> > * source code tag "version-0.2.0-incubating-rc3" [4],
> > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK
> > 1.8.0_151
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Yaniv
> >
> > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12321521&version=12342793
> > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/0.2.0rc3/
> > [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> > [4] https://github.com/apache/incubator-amaterasu/tags
> >
>

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Posted by Nadav Har Tzvi <na...@gmail.com>.
+1 approve. Tested multiple times and after a long round of fixing and
testing over and over.

Cheers,
Nadav


On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io> wrote:

> Hi everyone,
>
> We have fixed the legal issues, as well as a bug found by @Nadav please
> review and vote on the release candidate #3 for the version
> 0.2.0-incubating, as follows
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
>
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2],
> which is signed with the key with fingerprint [3],
> * source code tag "version-0.2.0-incubating-rc3" [4],
> * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK
> 1.8.0_151
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Yaniv
>
> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12321521&version=12342793
> [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/0.2.0rc3/
> [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS
> [4] https://github.com/apache/incubator-amaterasu/tags
>