You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Byung-Gon Chun <bg...@gmail.com> on 2014/12/06 01:03:47 UTC
Tentative schedule of the first release: week of Dec. 15
Hi all,
This is a quick update on the first release.
The first release of Apache REEF will be done in the week of Dec. 15.
I will give you more details about the release process in Apache later.
Meanwhile, if you have features you want to include in the first release,
please do so before Dec. 15.
Cheers,
Gon
--
Byung-Gon Chun
Re: Tentative schedule of the first release: week of Dec. 15
Posted by John Yang <jo...@gmail.com>.
I registered an issue: https://issues.apache.org/jira/browse/REEF-80
John
On Thu, Dec 25, 2014 at 12:04 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
> John, thanks for the update.
>
> A class project team also reported the same problem.
>
> Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.RuntimeException: Cannot append by read-append-delete-create with
> exception.
> Caused by: java.lang.RuntimeException: Cannot append by
> read-append-delete-create with exception.
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): ID
> mismatch. Request id and saved id: 165643 , 165644 for file
> /ReefApplications/application_1418954976955_0112/evaluatorsChangesLog
>
> It'd be nice to create a test case to reproduce this error.
> Let's register an issue to track this bug.
>
> -Gon
>
>
>
> On Tue, Dec 23, 2014 at 2:52 PM, John Yang <jo...@gmail.com> wrote:
>
> > Hi Markus,
> >
> >
> > you were right on point that the additional jar dependency was the source
> > of the problem. Thank you so much. Now the classpath problem is gone when
> > we run the tests with the script.
> >
> > I've filed a PR that enables running the tests on YARN:
> > https://github.com/apache/incubator-reef/pull/38. Gon will merge it.
> >
> >
> > All tests pass on my local machine(1-node YARN cluster), however 1
> > test(reef-job-TEST_CloseEvaluatorTest) fails on our lab's cluster. I
> have 2
> > questions regarding the failure.
> >
> > 1. YarnContainerManager#appendByDeleteAndCreate seems to get called even
> > in the environments where append is supported. Am I missing something?
> >
> > 2. From the Driver log, it seems that an ID mismatch is the root of the
> > failure:
> >
> > Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> > java.lang.RuntimeException: Cannot append by read-append-delete-create
> with
> > exception.
> > Caused by: java.lang.RuntimeException: Cannot append by
> > read-append-delete-create with exception.
> > Caused by:
> > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): ID
> > mismatch. Request id and saved id: 165643 , 165644 for file
> > /ReefApplications/application_1418954976955_0112/evaluatorsChangesLog
> >
> > has anyone experienced similar problems or know why this happens? I've
> > attached the log. You should also be able to reproduce this with the PR I
> > filed.
> >
> > Also, I heard from Gon that a team of students who worked on developing a
> > REEF application reported similar incidences a few times recently.
> >
> >
> > Thanks,
> > John
> >
> >
> >
> >
> > On Mon, Dec 22, 2014 at 11:30 PM, Markus Weimer <ma...@weimo.de> wrote:
> >
> >> Hi,
> >>
> >> can you try *only* using the following jar?
> >>
> >> reef-tests-0.10-incubating-SNAPSHOT-test-jar-with-dependencies.jar
> >>
> >> This is the JAR we use with bin/runtests.ps1 on freshly created
> >> HDInsight clusters without any other installation or configuration
> needed.
> >>
> >> It contains all the dependencies we need but the ones marked as
> >> "provided" in the POMs. The latter is basically Hadoop. This JAR
> >> specifically contains all test dependencies as well, transitively (I
> >> hope). You should not need to add anything else (but Hadoop) to the
> >> CLASSPATH. For instance, the following command should succeed running
> >> HelloREEF on your cluster:
> >>
> >> yarn jar
> >> reef-tests-0.10-incubating-SNAPSHOT-test-jar-with-dependencies.jar
> >> org.apache.reef.examples.hello.HelloReefYarn
> >>
> >> Adding JARs might just as well produce the error you see: If the test
> >> class is in more than one JAR on the classpath, the JAR resolution
> >> scheme we use will pick up the first one. And that might just be one
> >> that does not contain all dependencies and therefore not the Launcher
> >> class.
> >>
> >> Hope this helps,
> >>
> >> Markus
> >>
> >>
> >>
> >>
> >>
> >> On 2014-12-22 14:41, John Yang wrote:
> >> > Hello everyone,
> >> >
> >> >
> >> > I'd like to share my findings on this.
> >> >
> >> >
> >> > 1. I've reproduced Brian's result(attached below) using his script
> >> > in our lab's Cluster. It seems that the tests that use the jar of
> >> > this.getClass() as the GLOBAL_LIBRARIES produce the Launcher not
> >> > found error.
> >> >
> >> > Error(Launcher not found):
> >> > .set(DriverConfiguration.GLOBAL_LIBRARIES,
> >> > EnvironmentUtils.getClassLocation(this.getClass()))
> >> >
> >> > Working: .set(DriverConfiguration.GLOBAL_LIBRARIES,
> >> > EnvironmentUtils.getClassLocation(EvaluatorReuseTestDriver.class))
> >> >
> >> >
> >> > 2. On my local computer, I get 1~2 failures(sometimes 1 and
> >> > sometimes 2), which are not caused by Launcher not found
> >> > error(NullPointer, etc) though.
> >> >
> >> >
> >> > 3. I've also tried "mvn clean install" and failed. The hardest part
> >> > is running the tests against a single shaded(or
> >> > jar-with-dependency) jar, which I did not succeed. The reason why
> >> > Launcher not found error occurs is that if the tests are not run
> >> > against a single jar with that contains all the dependencies, by
> >> > default "EnvironmentUtils.getClassLocation(CLASS_XX)", only include
> >> > the test classes located in /target/test-classes. Because only test
> >> > classes are sent to the Driver, the Driver cannot find Launcher.
> >> >
> >> >
> >> > Tomorrow, I'll try to find the source of the "this.getClass()"
> >> > problem and give running tests against a shaded jar another try.
> >> >
> >> >
> >> > Best regards, John
> >> >
> >> >
> >> >
> >> > [Attached]Brian's doc:
> >> >
> >> > ===
> >> >
> >> > # reef-tests on YARN
> >> >
> >> > This document contains the current status of trying to get
> >> > reef-tests to run on a YARN cluster. Running using maven, as the
> >> > local runtime does, is not working. Instead, the current attempt is
> >> > to run via JUnit.
> >> >
> >> > ## Running via JUnit - Prerequisites - Build reef by running `mvn
> >> > clean install -DskipTests` - Build reef-tests jar by running `mvn
> >> > jar:test-jar` from the reef-tests directory - Run reef-tests by
> >> > running JUnitCore. The bash script below runs with the reef-tests
> >> > jar, test-jar-with-dependencies, Hadoop conf dir, and Hadoop jars
> >> > in the classpath.
> >> >
> >> > ``` TEST_JAR=`echo
> >> > $REEF_HOME/reef-tests/target/reef-tests-*-tests.jar`
> >> > DEPENDENCY_JAR=`echo
> >> >
> >>
> $REEF_HOME/reef-tests/target/reef-tests-*-test-jar-with-dependencies.jar`
> >> >
> >> >
> >> >
> >>
> CLASSPATH=$YARN_HOME/share/hadoop/common/*:$YARN_HOME/share/hadoop/common/lib/*:$YARN_HOME/share/hadoop/yarn/*:$YARN_HOME/share/hadoop/hdfs/*:$YARN_HOME/share/hadoop/mapreduce/lib/*:$YARN_HOME/share/hadoop/mapreduce/*
> >> >
> >> > YARN_CONF_DIR=$YARN_HOME/etc/hadoop
> >> >
> >> > CMD="java -cp $YARN_CONF_DIR:$DEPENDENCY_JAR:$TEST_JAR:$CLASSPATH
> >> > org.junit.runner.JUnitCore org.apache.reef.tests.AllTestsSuite $*"
> >> > echo $CMD $CMD ```
> >> >
> >> > I’ve found that Hadoop configuration directory and jars must be
> >> > added to the classpath. (Hadoop jars are not found within
> >> > test-jar-with-dependencies.) If these are not added to the
> >> > classpath, the tests fail because the _client_ refuses to run with
> >> > a `NoClassDefFound` exception.
> >> >
> >> > ## Results
> >> >
> >> > ``` Tests run: 28, Failures: 13 ```
> >> >
> >> > Note that many jobs (`reef-job-TEST_EvaluatorSizeTest-777`,
> >> > `reef-job-TEST_FileResourceTest`, etc.) fail immediately on the
> >> > cluster, but the test runner can only proceed after the 5 minute
> >> > timeout. So even if the test runner appears stalled, it will
> >> > continue after 5 minutes. It may make sense to adjust
> >> > `YarnTestEnvironment.getTestTimeout` while debugging these jobs. I
> >> > believe the jobs that wait for timeout are unexpected failures,
> >> > while those that continue immediately are failures expected by the
> >> > tests.
> >> >
> >> > ## Some pointers
> >> >
> >> > The CMS cluster is setup with log aggregation. So, once a job has
> >> > finished, the log is available using the command `yarn logs
> >> > -applicationId <appId>`
> >> >
> >> > ## Debugging notes
> >> >
> >> > The logs reveal the job failures seem to stem from some form of
> >> > “class not found” errors. Some examples: - `Error: Could not find
> >> > or load main class org.apache.reef.runtime.common.Launcher` -
> >> > Caused by: java.lang.ClassNotFoundException:
> >> > org.apache.reef.tests.fail.FailDriverTest (seen once)
> >> >
> >> > `Launcher` is probably needed by all REEF apps, but it is only
> >> > sporadically not found. This is the most puzzling part so far.
> >> >
> >> >
> >> > On Mon, Dec 22, 2014 at 6:12 PM, Byung-Gon Chun <bg...@gmail.com>
> >> > wrote:
> >> >
> >> >> Brian's on honeymoon. :) So, John and I have taken over YARN
> >> >> tests in our Linux cluster.
> >> >>
> >> >> It'd be great if we can run YARN tests on the windows cluster,
> >> >> too.
> >> >>
> >> >> Thanks. -Gon
> >> >>
> >> >> On Thu, Dec 18, 2014 at 5:01 PM, Brian Cho <ch...@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> Yes, the branch passes tests locally.
> >> >>>
> >> >>> We'll have to fix YARN tests. I haven't figured it out quite
> >> >>> yet. The fix should be contributed to the main branch, which we
> >> >>> could then merge to release branch.
> >> >>>
> >> >>> -Brian
> >> >>>
> >> >>> On Thu, Dec 18, 2014 at 4:57 PM, Byung-Gon Chun
> >> >>> <bg...@gmail.com>
> >> >> wrote:
> >> >>>>
> >> >>>> Thanks, Brian.
> >> >>>>
> >> >>>> Did you test the branch locally?
> >> >>>>
> >> >>>> What should we do about YARN tests?
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Thu, Dec 18, 2014 at 4:31 PM, Brian Cho
> >> >>>> <ch...@gmail.com> wrote:
> >> >>>>>
> >> >>>>> On Thu, Dec 18, 2014 at 10:09 AM, Byung-Gon Chun
> >> >>>>> <bg...@gmail.com>
> >> >>>> wrote:
> >> >>>>>>
> >> >>>>>> Thanks for the input.
> >> >>>>>>
> >> >>>>>> Sure. I will publish the git hash of the version after
> >> >>>>>> creating a
> >> >>>> release
> >> >>>>>> branch (following apache incubation naming).
> >> >>>>>>
> >> >>>>>> I will update basic files on the master branch.
> >> >>>>>>
> >> >>>>>> Brian will do the removal of the reef-bridge in the
> >> >>>>>> release branch. Brian, please check the naming convention
> >> >>>>>> of the release branch in
> >> >>>>> Apache.
> >> >>>>>>
> >> >>>>>>
> >> >>>>> I've added the release branch (branch-0.10.0-incubating)
> >> >>>>> and pushed
> >> >> the
> >> >>>>> reef-bridge removal commit. I would have liked to issue a
> >> >>>>> pull
> >> >> request
> >> >>>> for
> >> >>>>> this commit, but it seems github mirroring for a new branch
> >> >>>>> only
> >> >> kicks
> >> >>> in
> >> >>>>> once there is a first commit. So, I didn't have a branch on
> >> >>>>> github to
> >> >>>> issue
> >> >>>>> the pull request for at the time of the commit. In future,
> >> >>>>> we may
> >> >> want
> >> >>> to
> >> >>>>> make a trivial commit after creating a branch (although
> >> >>>>> these hacks
> >> >>>> always
> >> >>>>> make me cringe).
> >> >>>>>
> >> >>>>> Thanks, Brian
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>> Thanks. -Gon
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Thu, Dec 18, 2014 at 2:32 AM, Markus Weimer
> >> >>>>>> <ma...@weimo.de>
> >> >>>> wrote:
> >> >>>>>>>
> >> >>>>>>> Hi,
> >> >>>>>>>
> >> >>>>>>> I'd like to make sure that we have the released version
> >> >>>>>>> in git as
> >> >>>> well.
> >> >>>>>>> That means that we should publish the git hash of the
> >> >>>>>>> version
> >> >> that
> >> >>>> was
> >> >>>>>>> used to make the tarball. That way, we can always go
> >> >>>>>>> back and
> >> >>>> backport
> >> >>>>>>> stuff as it becomes needed.
> >> >>>>>>>
> >> >>>>>>> For that, can we create a release branch? E.g.
> >> >>>>>>> "release-0.10" or
> >> >>>> such?
> >> >>>>>>> However, some of the work below (updating LICENSE,
> >> >>>>>>> DISCLAIMER,
> >> >>>> NOTICES,
> >> >>>>>>> add KEYS, ...) can be done immediately against the
> >> >>>>>>> master
> >> >> branch. I
> >> >>>>>>> believe the license headers are alright, given that
> >> >>>>>>> apache-rat
> >> >>>> passes.
> >> >>>>>>>
> >> >>>>>>> Some things will be subtractive and should only be done
> >> >>>>>>> in the
> >> >>>> branch.
> >> >>>>>>> First and foremost, that is the removal of reef-bridge
> >> >>>>>>> both from
> >> >>> git
> >> >>>>> and
> >> >>>>>>> the master pom. That code won't work in open source
> >> >>>>>>> until REEF-51
> >> >>> is
> >> >>>>>>> resolved anyhow. I'm not sure what else will have to be
> >> >>>>>>> removed.
> >> >>>>>>>
> >> >>>>>>> Markus
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> On 2014-12-17 08:01, Byung-Gon Chun wrote:
> >> >>>>>>>> Hi,
> >> >>>>>>>>
> >> >>>>>>>> Warning. This email is long.
> >> >>>>>>>>
> >> >>>>>>>> Release voting is done in two phases. First, the PPMC
> >> >>>>>>>> votes on a release once it's ready by the
> >> >> release
> >> >>>>>> manager
> >> >>>>>>>> (me this time). If that passes, we call a vote on the
> >> >>>>>>>> release;
> >> >>>>>> incubator
> >> >>>>>>>> PMC members vote.
> >> >>>>>>>>
> >> >>>>>>>> PPMC should check the following for the release. 1.
> >> >>>>>>>> checksums and PGP signatures are valid The release
> >> >>>>>>>> candidate is signed with PGP key ....., which is
> >> >>>> included
> >> >>>>>>>> in the repository's KEYS file: <url>
> >> >>>>>>>>
> >> >>>>>>>> checksums: sha1 and/or md5 checksums
> >> >>>>>>>>
> >> >>>>>>>> 2. Build is successful including automated tests
> >> >>>>>>>>
> >> >>>>>>>> 3. DISCLAIMER is correct. filenames include
> >> >>>>>>>> "incubating". The file name should be
> >> >>>>>>>> 0.10.0-incubating.
> >> >>>>>>>>
> >> >>>>>>>> We have to add DISCLAIMER.
> >> >>>>>>>>
> >> >>>>>>>> Podling web sites MUST include a clear disclaimer on
> >> >>>>>>>> their
> >> >>> website
> >> >>>>> and
> >> >>>>>> in
> >> >>>>>>>> all documentation (including releases) stating that
> >> >>>>>>>> they are in
> >> >>>>>>> incubation.
> >> >>>>>>>> Podlings SHOULD use the following text for all
> >> >>>>>>>> disclaimers
> >> >>> (replace
> >> >>>>> the
> >> >>>>>>>> underlined phrases as appropriate):
> >> >>>>>>>>
> >> >>>>>>>> Apache *Podling-Name* is an effort undergoing
> >> >>>>>>>> incubation at The
> >> >>>>> Apache
> >> >>>>>>>> Software Foundation (ASF), sponsored by the *name of
> >> >>>>>>>> Apache TLP
> >> >>>>>> sponsor*.
> >> >>>>>>>> Incubation is required of all newly accepted projects
> >> >>>>>>>> until a
> >> >>>> further
> >> >>>>>>>> review indicates that the infrastructure,
> >> >>>>>>>> communications, and
> >> >>>>> decision
> >> >>>>>>>> making process have stabilized in a manner consistent
> >> >>>>>>>> with
> >> >> other
> >> >>>>>>> successful
> >> >>>>>>>> ASF projects. While incubation status is not
> >> >>>>>>>> necessarily a
> >> >>>> reflection
> >> >>>>>> of
> >> >>>>>>>> the completeness or stability of the code, it does
> >> >>>>>>>> indicate
> >> >> that
> >> >>>> the
> >> >>>>>>>> project has yet to be fully endorsed by the ASF.
> >> >>>>>>>>
> >> >>>>>>>> Podlings wishing to use a different disclaimer
> >> >>>>>>>> message MUST
> >> >> have
> >> >>>> the
> >> >>>>>>>> disclaimer approved by the Incubator PMC prior to
> >> >>>>>>>> use.
> >> >>>>>>>>
> >> >>>>>>>> For releases, the text SHOULD be included in a
> >> >>>>>>>> separate
> >> >>> DISCLAIMER
> >> >>>>> file
> >> >>>>>>>> stored alongside the NOTICE and LICENSE files.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> 4. Top-level LICENSE and NOTICE are correct for each
> >> >>> distribution.
> >> >>>>>>>>
> >> >>>>>>>> I believe we don't have any bundled dependencies. In
> >> >>>>>>>> that case,
> >> >>> we
> >> >>>>> can
> >> >>>>>> go
> >> >>>>>>>> with an easy path.
> >> >>>>>>>>
> >> >>>>>>>> LICENSE should contain the text of the ALv2
> >> >>>>>>>> <http://apache.org/licenses/LICENSE-2.0.txt> -- no
> >> >>>>>>>> more, no
> >> >>> less.
> >> >>>>>>>>
> >> >>>>>>>> NOTICE should contain only the following text,
> >> >>>>>>>> adapted with the
> >> >>>>>> product's
> >> >>>>>>>> name and copyright dates.
> >> >>>>>>>>
> >> >>>>>>>> Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache
> >> >>>>>>>> Software
> >> >>>>>> Foundation
> >> >>>>>>>> This product includes software developed atThe Apache
> >> >>>>>>>> Software Foundation (http://www.apache.org/).
> >> >>>>>>>>
> >> >>>>>>>> If not, we should add bundled dependency
> >> >>>>>>>> information.
> >> >>>>>>>>
> >> >>>>>>>> 5. All source files have license headers where
> >> >>>>>>>> appropriate. I believe rat checks this.
> >> >>>>>>>>
> >> >>>>>>>> 6. The provenance of all source files is clear (ASF
> >> >>>>>>>> or software
> >> >>>>>> grants).
> >> >>>>>>>> We took care of SGA and CLA. Right? So, we just need
> >> >>>>>>>> to get approval from IPMC members by voting.
> >> >>>>>>>>
> >> >>>>>>>> 7. Dependency licenses are ok. We cannot have the
> >> >>>>>>>> following
> >> >>>>> dependency
> >> >>>>>>>> license.
> >> >>>>>>>>
> >> >>>>>>>> - Binary Code License (BCL) - Special exceptions to
> >> >>>>>>>> the GNU GPL (e.g. GNU Classpath) - GNU GPL 1, 2, 3 <
> >> >>>>>> http://www.opensource.org/licenses/gpl-license.php
> >> >>>>>>>>
> >> >>>>>>>> - GNU LGPL 2, 2.1, 3
> >> >>>>>>>> <http://www.opensource.org/licenses/lgpl-license.php>
> >> >>>>>>>>
> >> >>>>>>>>
> >> - Affero GPL 3 <
> >> >>> http://www.opensource.org/licenses/agpl-v3.html
> >> >>>>>
> >> >>>>>>>> - NPL 1.0/NPL 1.1 - QPL
> >> >>>>>>>> <http://doc.trolltech.com/3.0/license.html> -
> >> >>>>>>>> Sleepycat License <
> >> >>>>>> http://www.opensource.org/licenses/sleepycat.php
> >> >>>>>>>>
> >> >>>>>>>> - Microsoft Limited Public License
> >> >>>>>>>> <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
> >> >>>>>>>> - Code Project Open License (CPOL)
> >> >>>>>>>>
> >> >>>>>>>> 8. Release consists of source code only, no
> >> >>>>>>>> binaries. We should not have any dll, so, jar.
> >> >>>>>>>>
> >> >>>>>>>> * Some like to add notices (such as RELEASE_NOTES,
> >> >>>>>>>> READMEand
> >> >>>>> CHANGES)
> >> >>>>>>> for
> >> >>>>>>>> each release. CHANGES lists fixed JIRA issues.
> >> >>>>>>>>
> >> >>>>>>>> After addressing the above, I will create a source
> >> >>>>>>>> tarball
> >> >>>>> (*.tar.gz),
> >> >>>>>>>> signature (*.asc), checksum (*.sha,*.md5) and post
> >> >>>>>>>> them in my
> >> >>>> apache
> >> >>>>>>>> account, which is a typical way of posting RCs.
> >> >>>>>>>>
> >> >>>>>>>> I will send out the SHA1 checksum of the archive, the
> >> >>>>>>>> commit id to be voted upon, the tag to be voted upon,
> >> >>>>>>>> the
> >> >>> list
> >> >>>> of
> >> >>>>>>> fixed
> >> >>>>>>>> issues, a link to the keys to verify the signature of
> >> >>>>>>>> the
> >> >> release
> >> >>>>>>> artifact.
> >> >>>>>>>>
> >> >>>>>>>> Once the incubator PMC approves the release, we
> >> >>>>>>>> upload the
> >> >>> release
> >> >>>> to
> >> >>>>>> the
> >> >>>>>>>> incubator distribution directory, podling
> >> >>>>>>>> distribution
> >> >> directory,
> >> >>>> and
> >> >>>>>> our
> >> >>>>>>>> own web site.
> >> >>>>>>>>
> >> >>>>>>>> Let me know if you have any questions.
> >> >>>>>>>>
> >> >>>>>>>> -Gon
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <
> >> >>> bgchun@gmail.com>
> >> >>>>>>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>> Hi,
> >> >>>>>>>>>
> >> >>>>>>>>> The release management is a bit more complicated
> >> >>>>>>>>> than I
> >> >> thought.
> >> >>>>>>>>> There're things I am figuring out since this is the
> >> >>>>>>>>> first one. Hopefully I will send out an update
> >> >>>>>>>>> today. Stay tuned.
> >> >>>>>>>>>
> >> >>>>>>>>> -Gon
> >> >>>>>>>>>
> >> >>>>>>>>> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <
> >> >>> markus@weimo.de>
> >> >>>>>>> wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>> Hi,
> >> >>>>>>>>>>
> >> >>>>>>>>>> any updates on the schedule or task list for the
> >> >>>>>>>>>> rest of us
> >> >> to
> >> >>>> get
> >> >>>>>> this
> >> >>>>>>>>>> done?
> >> >>>>>>>>>>
> >> >>>>>>>>>> Thanks,
> >> >>>>>>>>>>
> >> >>>>>>>>>> Markus
> >> >>>>>>>>>>
> >> >>>>>>>>>> -----Original Message----- From: "Byung-Gon Chun"
> >> >>>>>>>>>> <bg...@gmail.com> Sent: 12/5/2014 16:06 To:
> >> >>>>>>>>>> "dev@reef.incubator.apache.org" <
> >> >>>> dev@reef.incubator.apache.org
> >> >>>>>>
> >> >>>>>>>>>> Subject: Tentative schedule of the first release:
> >> >>>>>>>>>> week of
> >> >> Dec.
> >> >>> 15
> >> >>>>>>>>>>
> >> >>>>>>>>>> Hi all,
> >> >>>>>>>>>>
> >> >>>>>>>>>> This is a quick update on the first release. The
> >> >>>>>>>>>> first release of Apache REEF will be done in the
> >> >>>>>>>>>> week of
> >> >>> Dec.
> >> >>>>> 15.
> >> >>>>>>>>>> I will give you more details about the release
> >> >>>>>>>>>> process in
> >> >>> Apache
> >> >>>>>> later.
> >> >>>>>>>>>>
> >> >>>>>>>>>> Meanwhile, if you have features you want to
> >> >>>>>>>>>> include in the
> >> >>> first
> >> >>>>>>> release,
> >> >>>>>>>>>> please do so before Dec. 15.
> >> >>>>>>>>>>
> >> >>>>>>>>>> Cheers, Gon
> >> >>>>>>>>>>
> >> >>>>>>>>>> -- Byung-Gon Chun
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>> -- Byung-Gon Chun
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> -- Byung-Gon Chun
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>> -- Byung-Gon Chun
> >> >>>>
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> -- Byung-Gon Chun
> >> >>
> >> >
> >>
> >
> >
>
>
> --
> Byung-Gon Chun
>
Re: Tentative schedule of the first release: week of Dec. 15
Posted by Byung-Gon Chun <bg...@gmail.com>.
John, thanks for the update.
A class project team also reported the same problem.
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: Cannot append by read-append-delete-create with
exception.
Caused by: java.lang.RuntimeException: Cannot append by
read-append-delete-create with exception.
Caused by:
org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): ID
mismatch. Request id and saved id: 165643 , 165644 for file
/ReefApplications/application_1418954976955_0112/evaluatorsChangesLog
It'd be nice to create a test case to reproduce this error.
Let's register an issue to track this bug.
-Gon
On Tue, Dec 23, 2014 at 2:52 PM, John Yang <jo...@gmail.com> wrote:
> Hi Markus,
>
>
> you were right on point that the additional jar dependency was the source
> of the problem. Thank you so much. Now the classpath problem is gone when
> we run the tests with the script.
>
> I've filed a PR that enables running the tests on YARN:
> https://github.com/apache/incubator-reef/pull/38. Gon will merge it.
>
>
> All tests pass on my local machine(1-node YARN cluster), however 1
> test(reef-job-TEST_CloseEvaluatorTest) fails on our lab's cluster. I have 2
> questions regarding the failure.
>
> 1. YarnContainerManager#appendByDeleteAndCreate seems to get called even
> in the environments where append is supported. Am I missing something?
>
> 2. From the Driver log, it seems that an ID mismatch is the root of the
> failure:
>
> Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.RuntimeException: Cannot append by read-append-delete-create with
> exception.
> Caused by: java.lang.RuntimeException: Cannot append by
> read-append-delete-create with exception.
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): ID
> mismatch. Request id and saved id: 165643 , 165644 for file
> /ReefApplications/application_1418954976955_0112/evaluatorsChangesLog
>
> has anyone experienced similar problems or know why this happens? I've
> attached the log. You should also be able to reproduce this with the PR I
> filed.
>
> Also, I heard from Gon that a team of students who worked on developing a
> REEF application reported similar incidences a few times recently.
>
>
> Thanks,
> John
>
>
>
>
> On Mon, Dec 22, 2014 at 11:30 PM, Markus Weimer <ma...@weimo.de> wrote:
>
>> Hi,
>>
>> can you try *only* using the following jar?
>>
>> reef-tests-0.10-incubating-SNAPSHOT-test-jar-with-dependencies.jar
>>
>> This is the JAR we use with bin/runtests.ps1 on freshly created
>> HDInsight clusters without any other installation or configuration needed.
>>
>> It contains all the dependencies we need but the ones marked as
>> "provided" in the POMs. The latter is basically Hadoop. This JAR
>> specifically contains all test dependencies as well, transitively (I
>> hope). You should not need to add anything else (but Hadoop) to the
>> CLASSPATH. For instance, the following command should succeed running
>> HelloREEF on your cluster:
>>
>> yarn jar
>> reef-tests-0.10-incubating-SNAPSHOT-test-jar-with-dependencies.jar
>> org.apache.reef.examples.hello.HelloReefYarn
>>
>> Adding JARs might just as well produce the error you see: If the test
>> class is in more than one JAR on the classpath, the JAR resolution
>> scheme we use will pick up the first one. And that might just be one
>> that does not contain all dependencies and therefore not the Launcher
>> class.
>>
>> Hope this helps,
>>
>> Markus
>>
>>
>>
>>
>>
>> On 2014-12-22 14:41, John Yang wrote:
>> > Hello everyone,
>> >
>> >
>> > I'd like to share my findings on this.
>> >
>> >
>> > 1. I've reproduced Brian's result(attached below) using his script
>> > in our lab's Cluster. It seems that the tests that use the jar of
>> > this.getClass() as the GLOBAL_LIBRARIES produce the Launcher not
>> > found error.
>> >
>> > Error(Launcher not found):
>> > .set(DriverConfiguration.GLOBAL_LIBRARIES,
>> > EnvironmentUtils.getClassLocation(this.getClass()))
>> >
>> > Working: .set(DriverConfiguration.GLOBAL_LIBRARIES,
>> > EnvironmentUtils.getClassLocation(EvaluatorReuseTestDriver.class))
>> >
>> >
>> > 2. On my local computer, I get 1~2 failures(sometimes 1 and
>> > sometimes 2), which are not caused by Launcher not found
>> > error(NullPointer, etc) though.
>> >
>> >
>> > 3. I've also tried "mvn clean install" and failed. The hardest part
>> > is running the tests against a single shaded(or
>> > jar-with-dependency) jar, which I did not succeed. The reason why
>> > Launcher not found error occurs is that if the tests are not run
>> > against a single jar with that contains all the dependencies, by
>> > default "EnvironmentUtils.getClassLocation(CLASS_XX)", only include
>> > the test classes located in /target/test-classes. Because only test
>> > classes are sent to the Driver, the Driver cannot find Launcher.
>> >
>> >
>> > Tomorrow, I'll try to find the source of the "this.getClass()"
>> > problem and give running tests against a shaded jar another try.
>> >
>> >
>> > Best regards, John
>> >
>> >
>> >
>> > [Attached]Brian's doc:
>> >
>> > ===
>> >
>> > # reef-tests on YARN
>> >
>> > This document contains the current status of trying to get
>> > reef-tests to run on a YARN cluster. Running using maven, as the
>> > local runtime does, is not working. Instead, the current attempt is
>> > to run via JUnit.
>> >
>> > ## Running via JUnit - Prerequisites - Build reef by running `mvn
>> > clean install -DskipTests` - Build reef-tests jar by running `mvn
>> > jar:test-jar` from the reef-tests directory - Run reef-tests by
>> > running JUnitCore. The bash script below runs with the reef-tests
>> > jar, test-jar-with-dependencies, Hadoop conf dir, and Hadoop jars
>> > in the classpath.
>> >
>> > ``` TEST_JAR=`echo
>> > $REEF_HOME/reef-tests/target/reef-tests-*-tests.jar`
>> > DEPENDENCY_JAR=`echo
>> >
>> $REEF_HOME/reef-tests/target/reef-tests-*-test-jar-with-dependencies.jar`
>> >
>> >
>> >
>> CLASSPATH=$YARN_HOME/share/hadoop/common/*:$YARN_HOME/share/hadoop/common/lib/*:$YARN_HOME/share/hadoop/yarn/*:$YARN_HOME/share/hadoop/hdfs/*:$YARN_HOME/share/hadoop/mapreduce/lib/*:$YARN_HOME/share/hadoop/mapreduce/*
>> >
>> > YARN_CONF_DIR=$YARN_HOME/etc/hadoop
>> >
>> > CMD="java -cp $YARN_CONF_DIR:$DEPENDENCY_JAR:$TEST_JAR:$CLASSPATH
>> > org.junit.runner.JUnitCore org.apache.reef.tests.AllTestsSuite $*"
>> > echo $CMD $CMD ```
>> >
>> > I’ve found that Hadoop configuration directory and jars must be
>> > added to the classpath. (Hadoop jars are not found within
>> > test-jar-with-dependencies.) If these are not added to the
>> > classpath, the tests fail because the _client_ refuses to run with
>> > a `NoClassDefFound` exception.
>> >
>> > ## Results
>> >
>> > ``` Tests run: 28, Failures: 13 ```
>> >
>> > Note that many jobs (`reef-job-TEST_EvaluatorSizeTest-777`,
>> > `reef-job-TEST_FileResourceTest`, etc.) fail immediately on the
>> > cluster, but the test runner can only proceed after the 5 minute
>> > timeout. So even if the test runner appears stalled, it will
>> > continue after 5 minutes. It may make sense to adjust
>> > `YarnTestEnvironment.getTestTimeout` while debugging these jobs. I
>> > believe the jobs that wait for timeout are unexpected failures,
>> > while those that continue immediately are failures expected by the
>> > tests.
>> >
>> > ## Some pointers
>> >
>> > The CMS cluster is setup with log aggregation. So, once a job has
>> > finished, the log is available using the command `yarn logs
>> > -applicationId <appId>`
>> >
>> > ## Debugging notes
>> >
>> > The logs reveal the job failures seem to stem from some form of
>> > “class not found” errors. Some examples: - `Error: Could not find
>> > or load main class org.apache.reef.runtime.common.Launcher` -
>> > Caused by: java.lang.ClassNotFoundException:
>> > org.apache.reef.tests.fail.FailDriverTest (seen once)
>> >
>> > `Launcher` is probably needed by all REEF apps, but it is only
>> > sporadically not found. This is the most puzzling part so far.
>> >
>> >
>> > On Mon, Dec 22, 2014 at 6:12 PM, Byung-Gon Chun <bg...@gmail.com>
>> > wrote:
>> >
>> >> Brian's on honeymoon. :) So, John and I have taken over YARN
>> >> tests in our Linux cluster.
>> >>
>> >> It'd be great if we can run YARN tests on the windows cluster,
>> >> too.
>> >>
>> >> Thanks. -Gon
>> >>
>> >> On Thu, Dec 18, 2014 at 5:01 PM, Brian Cho <ch...@gmail.com>
>> >> wrote:
>> >>
>> >>> Yes, the branch passes tests locally.
>> >>>
>> >>> We'll have to fix YARN tests. I haven't figured it out quite
>> >>> yet. The fix should be contributed to the main branch, which we
>> >>> could then merge to release branch.
>> >>>
>> >>> -Brian
>> >>>
>> >>> On Thu, Dec 18, 2014 at 4:57 PM, Byung-Gon Chun
>> >>> <bg...@gmail.com>
>> >> wrote:
>> >>>>
>> >>>> Thanks, Brian.
>> >>>>
>> >>>> Did you test the branch locally?
>> >>>>
>> >>>> What should we do about YARN tests?
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Dec 18, 2014 at 4:31 PM, Brian Cho
>> >>>> <ch...@gmail.com> wrote:
>> >>>>>
>> >>>>> On Thu, Dec 18, 2014 at 10:09 AM, Byung-Gon Chun
>> >>>>> <bg...@gmail.com>
>> >>>> wrote:
>> >>>>>>
>> >>>>>> Thanks for the input.
>> >>>>>>
>> >>>>>> Sure. I will publish the git hash of the version after
>> >>>>>> creating a
>> >>>> release
>> >>>>>> branch (following apache incubation naming).
>> >>>>>>
>> >>>>>> I will update basic files on the master branch.
>> >>>>>>
>> >>>>>> Brian will do the removal of the reef-bridge in the
>> >>>>>> release branch. Brian, please check the naming convention
>> >>>>>> of the release branch in
>> >>>>> Apache.
>> >>>>>>
>> >>>>>>
>> >>>>> I've added the release branch (branch-0.10.0-incubating)
>> >>>>> and pushed
>> >> the
>> >>>>> reef-bridge removal commit. I would have liked to issue a
>> >>>>> pull
>> >> request
>> >>>> for
>> >>>>> this commit, but it seems github mirroring for a new branch
>> >>>>> only
>> >> kicks
>> >>> in
>> >>>>> once there is a first commit. So, I didn't have a branch on
>> >>>>> github to
>> >>>> issue
>> >>>>> the pull request for at the time of the commit. In future,
>> >>>>> we may
>> >> want
>> >>> to
>> >>>>> make a trivial commit after creating a branch (although
>> >>>>> these hacks
>> >>>> always
>> >>>>> make me cringe).
>> >>>>>
>> >>>>> Thanks, Brian
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>> Thanks. -Gon
>> >>>>>>
>> >>>>>>
>> >>>>>> On Thu, Dec 18, 2014 at 2:32 AM, Markus Weimer
>> >>>>>> <ma...@weimo.de>
>> >>>> wrote:
>> >>>>>>>
>> >>>>>>> Hi,
>> >>>>>>>
>> >>>>>>> I'd like to make sure that we have the released version
>> >>>>>>> in git as
>> >>>> well.
>> >>>>>>> That means that we should publish the git hash of the
>> >>>>>>> version
>> >> that
>> >>>> was
>> >>>>>>> used to make the tarball. That way, we can always go
>> >>>>>>> back and
>> >>>> backport
>> >>>>>>> stuff as it becomes needed.
>> >>>>>>>
>> >>>>>>> For that, can we create a release branch? E.g.
>> >>>>>>> "release-0.10" or
>> >>>> such?
>> >>>>>>> However, some of the work below (updating LICENSE,
>> >>>>>>> DISCLAIMER,
>> >>>> NOTICES,
>> >>>>>>> add KEYS, ...) can be done immediately against the
>> >>>>>>> master
>> >> branch. I
>> >>>>>>> believe the license headers are alright, given that
>> >>>>>>> apache-rat
>> >>>> passes.
>> >>>>>>>
>> >>>>>>> Some things will be subtractive and should only be done
>> >>>>>>> in the
>> >>>> branch.
>> >>>>>>> First and foremost, that is the removal of reef-bridge
>> >>>>>>> both from
>> >>> git
>> >>>>> and
>> >>>>>>> the master pom. That code won't work in open source
>> >>>>>>> until REEF-51
>> >>> is
>> >>>>>>> resolved anyhow. I'm not sure what else will have to be
>> >>>>>>> removed.
>> >>>>>>>
>> >>>>>>> Markus
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On 2014-12-17 08:01, Byung-Gon Chun wrote:
>> >>>>>>>> Hi,
>> >>>>>>>>
>> >>>>>>>> Warning. This email is long.
>> >>>>>>>>
>> >>>>>>>> Release voting is done in two phases. First, the PPMC
>> >>>>>>>> votes on a release once it's ready by the
>> >> release
>> >>>>>> manager
>> >>>>>>>> (me this time). If that passes, we call a vote on the
>> >>>>>>>> release;
>> >>>>>> incubator
>> >>>>>>>> PMC members vote.
>> >>>>>>>>
>> >>>>>>>> PPMC should check the following for the release. 1.
>> >>>>>>>> checksums and PGP signatures are valid The release
>> >>>>>>>> candidate is signed with PGP key ....., which is
>> >>>> included
>> >>>>>>>> in the repository's KEYS file: <url>
>> >>>>>>>>
>> >>>>>>>> checksums: sha1 and/or md5 checksums
>> >>>>>>>>
>> >>>>>>>> 2. Build is successful including automated tests
>> >>>>>>>>
>> >>>>>>>> 3. DISCLAIMER is correct. filenames include
>> >>>>>>>> "incubating". The file name should be
>> >>>>>>>> 0.10.0-incubating.
>> >>>>>>>>
>> >>>>>>>> We have to add DISCLAIMER.
>> >>>>>>>>
>> >>>>>>>> Podling web sites MUST include a clear disclaimer on
>> >>>>>>>> their
>> >>> website
>> >>>>> and
>> >>>>>> in
>> >>>>>>>> all documentation (including releases) stating that
>> >>>>>>>> they are in
>> >>>>>>> incubation.
>> >>>>>>>> Podlings SHOULD use the following text for all
>> >>>>>>>> disclaimers
>> >>> (replace
>> >>>>> the
>> >>>>>>>> underlined phrases as appropriate):
>> >>>>>>>>
>> >>>>>>>> Apache *Podling-Name* is an effort undergoing
>> >>>>>>>> incubation at The
>> >>>>> Apache
>> >>>>>>>> Software Foundation (ASF), sponsored by the *name of
>> >>>>>>>> Apache TLP
>> >>>>>> sponsor*.
>> >>>>>>>> Incubation is required of all newly accepted projects
>> >>>>>>>> until a
>> >>>> further
>> >>>>>>>> review indicates that the infrastructure,
>> >>>>>>>> communications, and
>> >>>>> decision
>> >>>>>>>> making process have stabilized in a manner consistent
>> >>>>>>>> with
>> >> other
>> >>>>>>> successful
>> >>>>>>>> ASF projects. While incubation status is not
>> >>>>>>>> necessarily a
>> >>>> reflection
>> >>>>>> of
>> >>>>>>>> the completeness or stability of the code, it does
>> >>>>>>>> indicate
>> >> that
>> >>>> the
>> >>>>>>>> project has yet to be fully endorsed by the ASF.
>> >>>>>>>>
>> >>>>>>>> Podlings wishing to use a different disclaimer
>> >>>>>>>> message MUST
>> >> have
>> >>>> the
>> >>>>>>>> disclaimer approved by the Incubator PMC prior to
>> >>>>>>>> use.
>> >>>>>>>>
>> >>>>>>>> For releases, the text SHOULD be included in a
>> >>>>>>>> separate
>> >>> DISCLAIMER
>> >>>>> file
>> >>>>>>>> stored alongside the NOTICE and LICENSE files.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> 4. Top-level LICENSE and NOTICE are correct for each
>> >>> distribution.
>> >>>>>>>>
>> >>>>>>>> I believe we don't have any bundled dependencies. In
>> >>>>>>>> that case,
>> >>> we
>> >>>>> can
>> >>>>>> go
>> >>>>>>>> with an easy path.
>> >>>>>>>>
>> >>>>>>>> LICENSE should contain the text of the ALv2
>> >>>>>>>> <http://apache.org/licenses/LICENSE-2.0.txt> -- no
>> >>>>>>>> more, no
>> >>> less.
>> >>>>>>>>
>> >>>>>>>> NOTICE should contain only the following text,
>> >>>>>>>> adapted with the
>> >>>>>> product's
>> >>>>>>>> name and copyright dates.
>> >>>>>>>>
>> >>>>>>>> Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache
>> >>>>>>>> Software
>> >>>>>> Foundation
>> >>>>>>>> This product includes software developed atThe Apache
>> >>>>>>>> Software Foundation (http://www.apache.org/).
>> >>>>>>>>
>> >>>>>>>> If not, we should add bundled dependency
>> >>>>>>>> information.
>> >>>>>>>>
>> >>>>>>>> 5. All source files have license headers where
>> >>>>>>>> appropriate. I believe rat checks this.
>> >>>>>>>>
>> >>>>>>>> 6. The provenance of all source files is clear (ASF
>> >>>>>>>> or software
>> >>>>>> grants).
>> >>>>>>>> We took care of SGA and CLA. Right? So, we just need
>> >>>>>>>> to get approval from IPMC members by voting.
>> >>>>>>>>
>> >>>>>>>> 7. Dependency licenses are ok. We cannot have the
>> >>>>>>>> following
>> >>>>> dependency
>> >>>>>>>> license.
>> >>>>>>>>
>> >>>>>>>> - Binary Code License (BCL) - Special exceptions to
>> >>>>>>>> the GNU GPL (e.g. GNU Classpath) - GNU GPL 1, 2, 3 <
>> >>>>>> http://www.opensource.org/licenses/gpl-license.php
>> >>>>>>>>
>> >>>>>>>> - GNU LGPL 2, 2.1, 3
>> >>>>>>>> <http://www.opensource.org/licenses/lgpl-license.php>
>> >>>>>>>>
>> >>>>>>>>
>> - Affero GPL 3 <
>> >>> http://www.opensource.org/licenses/agpl-v3.html
>> >>>>>
>> >>>>>>>> - NPL 1.0/NPL 1.1 - QPL
>> >>>>>>>> <http://doc.trolltech.com/3.0/license.html> -
>> >>>>>>>> Sleepycat License <
>> >>>>>> http://www.opensource.org/licenses/sleepycat.php
>> >>>>>>>>
>> >>>>>>>> - Microsoft Limited Public License
>> >>>>>>>> <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
>> >>>>>>>> - Code Project Open License (CPOL)
>> >>>>>>>>
>> >>>>>>>> 8. Release consists of source code only, no
>> >>>>>>>> binaries. We should not have any dll, so, jar.
>> >>>>>>>>
>> >>>>>>>> * Some like to add notices (such as RELEASE_NOTES,
>> >>>>>>>> READMEand
>> >>>>> CHANGES)
>> >>>>>>> for
>> >>>>>>>> each release. CHANGES lists fixed JIRA issues.
>> >>>>>>>>
>> >>>>>>>> After addressing the above, I will create a source
>> >>>>>>>> tarball
>> >>>>> (*.tar.gz),
>> >>>>>>>> signature (*.asc), checksum (*.sha,*.md5) and post
>> >>>>>>>> them in my
>> >>>> apache
>> >>>>>>>> account, which is a typical way of posting RCs.
>> >>>>>>>>
>> >>>>>>>> I will send out the SHA1 checksum of the archive, the
>> >>>>>>>> commit id to be voted upon, the tag to be voted upon,
>> >>>>>>>> the
>> >>> list
>> >>>> of
>> >>>>>>> fixed
>> >>>>>>>> issues, a link to the keys to verify the signature of
>> >>>>>>>> the
>> >> release
>> >>>>>>> artifact.
>> >>>>>>>>
>> >>>>>>>> Once the incubator PMC approves the release, we
>> >>>>>>>> upload the
>> >>> release
>> >>>> to
>> >>>>>> the
>> >>>>>>>> incubator distribution directory, podling
>> >>>>>>>> distribution
>> >> directory,
>> >>>> and
>> >>>>>> our
>> >>>>>>>> own web site.
>> >>>>>>>>
>> >>>>>>>> Let me know if you have any questions.
>> >>>>>>>>
>> >>>>>>>> -Gon
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <
>> >>> bgchun@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Hi,
>> >>>>>>>>>
>> >>>>>>>>> The release management is a bit more complicated
>> >>>>>>>>> than I
>> >> thought.
>> >>>>>>>>> There're things I am figuring out since this is the
>> >>>>>>>>> first one. Hopefully I will send out an update
>> >>>>>>>>> today. Stay tuned.
>> >>>>>>>>>
>> >>>>>>>>> -Gon
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <
>> >>> markus@weimo.de>
>> >>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> Hi,
>> >>>>>>>>>>
>> >>>>>>>>>> any updates on the schedule or task list for the
>> >>>>>>>>>> rest of us
>> >> to
>> >>>> get
>> >>>>>> this
>> >>>>>>>>>> done?
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks,
>> >>>>>>>>>>
>> >>>>>>>>>> Markus
>> >>>>>>>>>>
>> >>>>>>>>>> -----Original Message----- From: "Byung-Gon Chun"
>> >>>>>>>>>> <bg...@gmail.com> Sent: 12/5/2014 16:06 To:
>> >>>>>>>>>> "dev@reef.incubator.apache.org" <
>> >>>> dev@reef.incubator.apache.org
>> >>>>>>
>> >>>>>>>>>> Subject: Tentative schedule of the first release:
>> >>>>>>>>>> week of
>> >> Dec.
>> >>> 15
>> >>>>>>>>>>
>> >>>>>>>>>> Hi all,
>> >>>>>>>>>>
>> >>>>>>>>>> This is a quick update on the first release. The
>> >>>>>>>>>> first release of Apache REEF will be done in the
>> >>>>>>>>>> week of
>> >>> Dec.
>> >>>>> 15.
>> >>>>>>>>>> I will give you more details about the release
>> >>>>>>>>>> process in
>> >>> Apache
>> >>>>>> later.
>> >>>>>>>>>>
>> >>>>>>>>>> Meanwhile, if you have features you want to
>> >>>>>>>>>> include in the
>> >>> first
>> >>>>>>> release,
>> >>>>>>>>>> please do so before Dec. 15.
>> >>>>>>>>>>
>> >>>>>>>>>> Cheers, Gon
>> >>>>>>>>>>
>> >>>>>>>>>> -- Byung-Gon Chun
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> -- Byung-Gon Chun
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> -- Byung-Gon Chun
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> -- Byung-Gon Chun
>> >>>>
>> >>>
>> >>
>> >>
>> >>
>> >> -- Byung-Gon Chun
>> >>
>> >
>>
>
>
--
Byung-Gon Chun
Re: Tentative schedule of the first release: week of Dec. 15
Posted by John Yang <jo...@gmail.com>.
Hi Markus,
you were right on point that the additional jar dependency was the source
of the problem. Thank you so much. Now the classpath problem is gone when
we run the tests with the script.
I've filed a PR that enables running the tests on YARN:
https://github.com/apache/incubator-reef/pull/38. Gon will merge it.
All tests pass on my local machine(1-node YARN cluster), however 1
test(reef-job-TEST_CloseEvaluatorTest) fails on our lab's cluster. I have 2
questions regarding the failure.
1. YarnContainerManager#appendByDeleteAndCreate seems to get called even in
the environments where append is supported. Am I missing something?
2. From the Driver log, it seems that an ID mismatch is the root of the
failure:
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: Cannot append by read-append-delete-create with
exception.
Caused by: java.lang.RuntimeException: Cannot append by
read-append-delete-create with exception.
Caused by:
org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): ID
mismatch. Request id and saved id: 165643 , 165644 for file
/ReefApplications/application_1418954976955_0112/evaluatorsChangesLog
has anyone experienced similar problems or know why this happens? I've
attached the log. You should also be able to reproduce this with the PR I
filed.
Also, I heard from Gon that a team of students who worked on developing a
REEF application reported similar incidences a few times recently.
Thanks,
John
On Mon, Dec 22, 2014 at 11:30 PM, Markus Weimer <ma...@weimo.de> wrote:
> Hi,
>
> can you try *only* using the following jar?
>
> reef-tests-0.10-incubating-SNAPSHOT-test-jar-with-dependencies.jar
>
> This is the JAR we use with bin/runtests.ps1 on freshly created
> HDInsight clusters without any other installation or configuration needed.
>
> It contains all the dependencies we need but the ones marked as
> "provided" in the POMs. The latter is basically Hadoop. This JAR
> specifically contains all test dependencies as well, transitively (I
> hope). You should not need to add anything else (but Hadoop) to the
> CLASSPATH. For instance, the following command should succeed running
> HelloREEF on your cluster:
>
> yarn jar
> reef-tests-0.10-incubating-SNAPSHOT-test-jar-with-dependencies.jar
> org.apache.reef.examples.hello.HelloReefYarn
>
> Adding JARs might just as well produce the error you see: If the test
> class is in more than one JAR on the classpath, the JAR resolution
> scheme we use will pick up the first one. And that might just be one
> that does not contain all dependencies and therefore not the Launcher
> class.
>
> Hope this helps,
>
> Markus
>
>
>
>
>
> On 2014-12-22 14:41, John Yang wrote:
> > Hello everyone,
> >
> >
> > I'd like to share my findings on this.
> >
> >
> > 1. I've reproduced Brian's result(attached below) using his script
> > in our lab's Cluster. It seems that the tests that use the jar of
> > this.getClass() as the GLOBAL_LIBRARIES produce the Launcher not
> > found error.
> >
> > Error(Launcher not found):
> > .set(DriverConfiguration.GLOBAL_LIBRARIES,
> > EnvironmentUtils.getClassLocation(this.getClass()))
> >
> > Working: .set(DriverConfiguration.GLOBAL_LIBRARIES,
> > EnvironmentUtils.getClassLocation(EvaluatorReuseTestDriver.class))
> >
> >
> > 2. On my local computer, I get 1~2 failures(sometimes 1 and
> > sometimes 2), which are not caused by Launcher not found
> > error(NullPointer, etc) though.
> >
> >
> > 3. I've also tried "mvn clean install" and failed. The hardest part
> > is running the tests against a single shaded(or
> > jar-with-dependency) jar, which I did not succeed. The reason why
> > Launcher not found error occurs is that if the tests are not run
> > against a single jar with that contains all the dependencies, by
> > default "EnvironmentUtils.getClassLocation(CLASS_XX)", only include
> > the test classes located in /target/test-classes. Because only test
> > classes are sent to the Driver, the Driver cannot find Launcher.
> >
> >
> > Tomorrow, I'll try to find the source of the "this.getClass()"
> > problem and give running tests against a shaded jar another try.
> >
> >
> > Best regards, John
> >
> >
> >
> > [Attached]Brian's doc:
> >
> > ===
> >
> > # reef-tests on YARN
> >
> > This document contains the current status of trying to get
> > reef-tests to run on a YARN cluster. Running using maven, as the
> > local runtime does, is not working. Instead, the current attempt is
> > to run via JUnit.
> >
> > ## Running via JUnit - Prerequisites - Build reef by running `mvn
> > clean install -DskipTests` - Build reef-tests jar by running `mvn
> > jar:test-jar` from the reef-tests directory - Run reef-tests by
> > running JUnitCore. The bash script below runs with the reef-tests
> > jar, test-jar-with-dependencies, Hadoop conf dir, and Hadoop jars
> > in the classpath.
> >
> > ``` TEST_JAR=`echo
> > $REEF_HOME/reef-tests/target/reef-tests-*-tests.jar`
> > DEPENDENCY_JAR=`echo
> > $REEF_HOME/reef-tests/target/reef-tests-*-test-jar-with-dependencies.jar`
> >
> >
> >
> CLASSPATH=$YARN_HOME/share/hadoop/common/*:$YARN_HOME/share/hadoop/common/lib/*:$YARN_HOME/share/hadoop/yarn/*:$YARN_HOME/share/hadoop/hdfs/*:$YARN_HOME/share/hadoop/mapreduce/lib/*:$YARN_HOME/share/hadoop/mapreduce/*
> >
> > YARN_CONF_DIR=$YARN_HOME/etc/hadoop
> >
> > CMD="java -cp $YARN_CONF_DIR:$DEPENDENCY_JAR:$TEST_JAR:$CLASSPATH
> > org.junit.runner.JUnitCore org.apache.reef.tests.AllTestsSuite $*"
> > echo $CMD $CMD ```
> >
> > I’ve found that Hadoop configuration directory and jars must be
> > added to the classpath. (Hadoop jars are not found within
> > test-jar-with-dependencies.) If these are not added to the
> > classpath, the tests fail because the _client_ refuses to run with
> > a `NoClassDefFound` exception.
> >
> > ## Results
> >
> > ``` Tests run: 28, Failures: 13 ```
> >
> > Note that many jobs (`reef-job-TEST_EvaluatorSizeTest-777`,
> > `reef-job-TEST_FileResourceTest`, etc.) fail immediately on the
> > cluster, but the test runner can only proceed after the 5 minute
> > timeout. So even if the test runner appears stalled, it will
> > continue after 5 minutes. It may make sense to adjust
> > `YarnTestEnvironment.getTestTimeout` while debugging these jobs. I
> > believe the jobs that wait for timeout are unexpected failures,
> > while those that continue immediately are failures expected by the
> > tests.
> >
> > ## Some pointers
> >
> > The CMS cluster is setup with log aggregation. So, once a job has
> > finished, the log is available using the command `yarn logs
> > -applicationId <appId>`
> >
> > ## Debugging notes
> >
> > The logs reveal the job failures seem to stem from some form of
> > “class not found” errors. Some examples: - `Error: Could not find
> > or load main class org.apache.reef.runtime.common.Launcher` -
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.reef.tests.fail.FailDriverTest (seen once)
> >
> > `Launcher` is probably needed by all REEF apps, but it is only
> > sporadically not found. This is the most puzzling part so far.
> >
> >
> > On Mon, Dec 22, 2014 at 6:12 PM, Byung-Gon Chun <bg...@gmail.com>
> > wrote:
> >
> >> Brian's on honeymoon. :) So, John and I have taken over YARN
> >> tests in our Linux cluster.
> >>
> >> It'd be great if we can run YARN tests on the windows cluster,
> >> too.
> >>
> >> Thanks. -Gon
> >>
> >> On Thu, Dec 18, 2014 at 5:01 PM, Brian Cho <ch...@gmail.com>
> >> wrote:
> >>
> >>> Yes, the branch passes tests locally.
> >>>
> >>> We'll have to fix YARN tests. I haven't figured it out quite
> >>> yet. The fix should be contributed to the main branch, which we
> >>> could then merge to release branch.
> >>>
> >>> -Brian
> >>>
> >>> On Thu, Dec 18, 2014 at 4:57 PM, Byung-Gon Chun
> >>> <bg...@gmail.com>
> >> wrote:
> >>>>
> >>>> Thanks, Brian.
> >>>>
> >>>> Did you test the branch locally?
> >>>>
> >>>> What should we do about YARN tests?
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Dec 18, 2014 at 4:31 PM, Brian Cho
> >>>> <ch...@gmail.com> wrote:
> >>>>>
> >>>>> On Thu, Dec 18, 2014 at 10:09 AM, Byung-Gon Chun
> >>>>> <bg...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>> Thanks for the input.
> >>>>>>
> >>>>>> Sure. I will publish the git hash of the version after
> >>>>>> creating a
> >>>> release
> >>>>>> branch (following apache incubation naming).
> >>>>>>
> >>>>>> I will update basic files on the master branch.
> >>>>>>
> >>>>>> Brian will do the removal of the reef-bridge in the
> >>>>>> release branch. Brian, please check the naming convention
> >>>>>> of the release branch in
> >>>>> Apache.
> >>>>>>
> >>>>>>
> >>>>> I've added the release branch (branch-0.10.0-incubating)
> >>>>> and pushed
> >> the
> >>>>> reef-bridge removal commit. I would have liked to issue a
> >>>>> pull
> >> request
> >>>> for
> >>>>> this commit, but it seems github mirroring for a new branch
> >>>>> only
> >> kicks
> >>> in
> >>>>> once there is a first commit. So, I didn't have a branch on
> >>>>> github to
> >>>> issue
> >>>>> the pull request for at the time of the commit. In future,
> >>>>> we may
> >> want
> >>> to
> >>>>> make a trivial commit after creating a branch (although
> >>>>> these hacks
> >>>> always
> >>>>> make me cringe).
> >>>>>
> >>>>> Thanks, Brian
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Thanks. -Gon
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Dec 18, 2014 at 2:32 AM, Markus Weimer
> >>>>>> <ma...@weimo.de>
> >>>> wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I'd like to make sure that we have the released version
> >>>>>>> in git as
> >>>> well.
> >>>>>>> That means that we should publish the git hash of the
> >>>>>>> version
> >> that
> >>>> was
> >>>>>>> used to make the tarball. That way, we can always go
> >>>>>>> back and
> >>>> backport
> >>>>>>> stuff as it becomes needed.
> >>>>>>>
> >>>>>>> For that, can we create a release branch? E.g.
> >>>>>>> "release-0.10" or
> >>>> such?
> >>>>>>> However, some of the work below (updating LICENSE,
> >>>>>>> DISCLAIMER,
> >>>> NOTICES,
> >>>>>>> add KEYS, ...) can be done immediately against the
> >>>>>>> master
> >> branch. I
> >>>>>>> believe the license headers are alright, given that
> >>>>>>> apache-rat
> >>>> passes.
> >>>>>>>
> >>>>>>> Some things will be subtractive and should only be done
> >>>>>>> in the
> >>>> branch.
> >>>>>>> First and foremost, that is the removal of reef-bridge
> >>>>>>> both from
> >>> git
> >>>>> and
> >>>>>>> the master pom. That code won't work in open source
> >>>>>>> until REEF-51
> >>> is
> >>>>>>> resolved anyhow. I'm not sure what else will have to be
> >>>>>>> removed.
> >>>>>>>
> >>>>>>> Markus
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 2014-12-17 08:01, Byung-Gon Chun wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> Warning. This email is long.
> >>>>>>>>
> >>>>>>>> Release voting is done in two phases. First, the PPMC
> >>>>>>>> votes on a release once it's ready by the
> >> release
> >>>>>> manager
> >>>>>>>> (me this time). If that passes, we call a vote on the
> >>>>>>>> release;
> >>>>>> incubator
> >>>>>>>> PMC members vote.
> >>>>>>>>
> >>>>>>>> PPMC should check the following for the release. 1.
> >>>>>>>> checksums and PGP signatures are valid The release
> >>>>>>>> candidate is signed with PGP key ....., which is
> >>>> included
> >>>>>>>> in the repository's KEYS file: <url>
> >>>>>>>>
> >>>>>>>> checksums: sha1 and/or md5 checksums
> >>>>>>>>
> >>>>>>>> 2. Build is successful including automated tests
> >>>>>>>>
> >>>>>>>> 3. DISCLAIMER is correct. filenames include
> >>>>>>>> "incubating". The file name should be
> >>>>>>>> 0.10.0-incubating.
> >>>>>>>>
> >>>>>>>> We have to add DISCLAIMER.
> >>>>>>>>
> >>>>>>>> Podling web sites MUST include a clear disclaimer on
> >>>>>>>> their
> >>> website
> >>>>> and
> >>>>>> in
> >>>>>>>> all documentation (including releases) stating that
> >>>>>>>> they are in
> >>>>>>> incubation.
> >>>>>>>> Podlings SHOULD use the following text for all
> >>>>>>>> disclaimers
> >>> (replace
> >>>>> the
> >>>>>>>> underlined phrases as appropriate):
> >>>>>>>>
> >>>>>>>> Apache *Podling-Name* is an effort undergoing
> >>>>>>>> incubation at The
> >>>>> Apache
> >>>>>>>> Software Foundation (ASF), sponsored by the *name of
> >>>>>>>> Apache TLP
> >>>>>> sponsor*.
> >>>>>>>> Incubation is required of all newly accepted projects
> >>>>>>>> until a
> >>>> further
> >>>>>>>> review indicates that the infrastructure,
> >>>>>>>> communications, and
> >>>>> decision
> >>>>>>>> making process have stabilized in a manner consistent
> >>>>>>>> with
> >> other
> >>>>>>> successful
> >>>>>>>> ASF projects. While incubation status is not
> >>>>>>>> necessarily a
> >>>> reflection
> >>>>>> of
> >>>>>>>> the completeness or stability of the code, it does
> >>>>>>>> indicate
> >> that
> >>>> the
> >>>>>>>> project has yet to be fully endorsed by the ASF.
> >>>>>>>>
> >>>>>>>> Podlings wishing to use a different disclaimer
> >>>>>>>> message MUST
> >> have
> >>>> the
> >>>>>>>> disclaimer approved by the Incubator PMC prior to
> >>>>>>>> use.
> >>>>>>>>
> >>>>>>>> For releases, the text SHOULD be included in a
> >>>>>>>> separate
> >>> DISCLAIMER
> >>>>> file
> >>>>>>>> stored alongside the NOTICE and LICENSE files.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 4. Top-level LICENSE and NOTICE are correct for each
> >>> distribution.
> >>>>>>>>
> >>>>>>>> I believe we don't have any bundled dependencies. In
> >>>>>>>> that case,
> >>> we
> >>>>> can
> >>>>>> go
> >>>>>>>> with an easy path.
> >>>>>>>>
> >>>>>>>> LICENSE should contain the text of the ALv2
> >>>>>>>> <http://apache.org/licenses/LICENSE-2.0.txt> -- no
> >>>>>>>> more, no
> >>> less.
> >>>>>>>>
> >>>>>>>> NOTICE should contain only the following text,
> >>>>>>>> adapted with the
> >>>>>> product's
> >>>>>>>> name and copyright dates.
> >>>>>>>>
> >>>>>>>> Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache
> >>>>>>>> Software
> >>>>>> Foundation
> >>>>>>>> This product includes software developed atThe Apache
> >>>>>>>> Software Foundation (http://www.apache.org/).
> >>>>>>>>
> >>>>>>>> If not, we should add bundled dependency
> >>>>>>>> information.
> >>>>>>>>
> >>>>>>>> 5. All source files have license headers where
> >>>>>>>> appropriate. I believe rat checks this.
> >>>>>>>>
> >>>>>>>> 6. The provenance of all source files is clear (ASF
> >>>>>>>> or software
> >>>>>> grants).
> >>>>>>>> We took care of SGA and CLA. Right? So, we just need
> >>>>>>>> to get approval from IPMC members by voting.
> >>>>>>>>
> >>>>>>>> 7. Dependency licenses are ok. We cannot have the
> >>>>>>>> following
> >>>>> dependency
> >>>>>>>> license.
> >>>>>>>>
> >>>>>>>> - Binary Code License (BCL) - Special exceptions to
> >>>>>>>> the GNU GPL (e.g. GNU Classpath) - GNU GPL 1, 2, 3 <
> >>>>>> http://www.opensource.org/licenses/gpl-license.php
> >>>>>>>>
> >>>>>>>> - GNU LGPL 2, 2.1, 3
> >>>>>>>> <http://www.opensource.org/licenses/lgpl-license.php>
> >>>>>>>>
> >>>>>>>>
> - Affero GPL 3 <
> >>> http://www.opensource.org/licenses/agpl-v3.html
> >>>>>
> >>>>>>>> - NPL 1.0/NPL 1.1 - QPL
> >>>>>>>> <http://doc.trolltech.com/3.0/license.html> -
> >>>>>>>> Sleepycat License <
> >>>>>> http://www.opensource.org/licenses/sleepycat.php
> >>>>>>>>
> >>>>>>>> - Microsoft Limited Public License
> >>>>>>>> <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
> >>>>>>>> - Code Project Open License (CPOL)
> >>>>>>>>
> >>>>>>>> 8. Release consists of source code only, no
> >>>>>>>> binaries. We should not have any dll, so, jar.
> >>>>>>>>
> >>>>>>>> * Some like to add notices (such as RELEASE_NOTES,
> >>>>>>>> READMEand
> >>>>> CHANGES)
> >>>>>>> for
> >>>>>>>> each release. CHANGES lists fixed JIRA issues.
> >>>>>>>>
> >>>>>>>> After addressing the above, I will create a source
> >>>>>>>> tarball
> >>>>> (*.tar.gz),
> >>>>>>>> signature (*.asc), checksum (*.sha,*.md5) and post
> >>>>>>>> them in my
> >>>> apache
> >>>>>>>> account, which is a typical way of posting RCs.
> >>>>>>>>
> >>>>>>>> I will send out the SHA1 checksum of the archive, the
> >>>>>>>> commit id to be voted upon, the tag to be voted upon,
> >>>>>>>> the
> >>> list
> >>>> of
> >>>>>>> fixed
> >>>>>>>> issues, a link to the keys to verify the signature of
> >>>>>>>> the
> >> release
> >>>>>>> artifact.
> >>>>>>>>
> >>>>>>>> Once the incubator PMC approves the release, we
> >>>>>>>> upload the
> >>> release
> >>>> to
> >>>>>> the
> >>>>>>>> incubator distribution directory, podling
> >>>>>>>> distribution
> >> directory,
> >>>> and
> >>>>>> our
> >>>>>>>> own web site.
> >>>>>>>>
> >>>>>>>> Let me know if you have any questions.
> >>>>>>>>
> >>>>>>>> -Gon
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <
> >>> bgchun@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> The release management is a bit more complicated
> >>>>>>>>> than I
> >> thought.
> >>>>>>>>> There're things I am figuring out since this is the
> >>>>>>>>> first one. Hopefully I will send out an update
> >>>>>>>>> today. Stay tuned.
> >>>>>>>>>
> >>>>>>>>> -Gon
> >>>>>>>>>
> >>>>>>>>> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <
> >>> markus@weimo.de>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> any updates on the schedule or task list for the
> >>>>>>>>>> rest of us
> >> to
> >>>> get
> >>>>>> this
> >>>>>>>>>> done?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>>
> >>>>>>>>>> Markus
> >>>>>>>>>>
> >>>>>>>>>> -----Original Message----- From: "Byung-Gon Chun"
> >>>>>>>>>> <bg...@gmail.com> Sent: 12/5/2014 16:06 To:
> >>>>>>>>>> "dev@reef.incubator.apache.org" <
> >>>> dev@reef.incubator.apache.org
> >>>>>>
> >>>>>>>>>> Subject: Tentative schedule of the first release:
> >>>>>>>>>> week of
> >> Dec.
> >>> 15
> >>>>>>>>>>
> >>>>>>>>>> Hi all,
> >>>>>>>>>>
> >>>>>>>>>> This is a quick update on the first release. The
> >>>>>>>>>> first release of Apache REEF will be done in the
> >>>>>>>>>> week of
> >>> Dec.
> >>>>> 15.
> >>>>>>>>>> I will give you more details about the release
> >>>>>>>>>> process in
> >>> Apache
> >>>>>> later.
> >>>>>>>>>>
> >>>>>>>>>> Meanwhile, if you have features you want to
> >>>>>>>>>> include in the
> >>> first
> >>>>>>> release,
> >>>>>>>>>> please do so before Dec. 15.
> >>>>>>>>>>
> >>>>>>>>>> Cheers, Gon
> >>>>>>>>>>
> >>>>>>>>>> -- Byung-Gon Chun
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -- Byung-Gon Chun
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -- Byung-Gon Chun
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> -- Byung-Gon Chun
> >>>>
> >>>
> >>
> >>
> >>
> >> -- Byung-Gon Chun
> >>
> >
>
Re: Tentative schedule of the first release: week of Dec. 15
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
can you try *only* using the following jar?
reef-tests-0.10-incubating-SNAPSHOT-test-jar-with-dependencies.jar
This is the JAR we use with bin/runtests.ps1 on freshly created
HDInsight clusters without any other installation or configuration needed.
It contains all the dependencies we need but the ones marked as
"provided" in the POMs. The latter is basically Hadoop. This JAR
specifically contains all test dependencies as well, transitively (I
hope). You should not need to add anything else (but Hadoop) to the
CLASSPATH. For instance, the following command should succeed running
HelloREEF on your cluster:
yarn jar
reef-tests-0.10-incubating-SNAPSHOT-test-jar-with-dependencies.jar
org.apache.reef.examples.hello.HelloReefYarn
Adding JARs might just as well produce the error you see: If the test
class is in more than one JAR on the classpath, the JAR resolution
scheme we use will pick up the first one. And that might just be one
that does not contain all dependencies and therefore not the Launcher
class.
Hope this helps,
Markus
On 2014-12-22 14:41, John Yang wrote:
> Hello everyone,
>
>
> I'd like to share my findings on this.
>
>
> 1. I've reproduced Brian's result(attached below) using his script
> in our lab's Cluster. It seems that the tests that use the jar of
> this.getClass() as the GLOBAL_LIBRARIES produce the Launcher not
> found error.
>
> Error(Launcher not found):
> .set(DriverConfiguration.GLOBAL_LIBRARIES,
> EnvironmentUtils.getClassLocation(this.getClass()))
>
> Working: .set(DriverConfiguration.GLOBAL_LIBRARIES,
> EnvironmentUtils.getClassLocation(EvaluatorReuseTestDriver.class))
>
>
> 2. On my local computer, I get 1~2 failures(sometimes 1 and
> sometimes 2), which are not caused by Launcher not found
> error(NullPointer, etc) though.
>
>
> 3. I've also tried "mvn clean install" and failed. The hardest part
> is running the tests against a single shaded(or
> jar-with-dependency) jar, which I did not succeed. The reason why
> Launcher not found error occurs is that if the tests are not run
> against a single jar with that contains all the dependencies, by
> default "EnvironmentUtils.getClassLocation(CLASS_XX)", only include
> the test classes located in /target/test-classes. Because only test
> classes are sent to the Driver, the Driver cannot find Launcher.
>
>
> Tomorrow, I'll try to find the source of the "this.getClass()"
> problem and give running tests against a shaded jar another try.
>
>
> Best regards, John
>
>
>
> [Attached]Brian's doc:
>
> ===
>
> # reef-tests on YARN
>
> This document contains the current status of trying to get
> reef-tests to run on a YARN cluster. Running using maven, as the
> local runtime does, is not working. Instead, the current attempt is
> to run via JUnit.
>
> ## Running via JUnit - Prerequisites - Build reef by running `mvn
> clean install -DskipTests` - Build reef-tests jar by running `mvn
> jar:test-jar` from the reef-tests directory - Run reef-tests by
> running JUnitCore. The bash script below runs with the reef-tests
> jar, test-jar-with-dependencies, Hadoop conf dir, and Hadoop jars
> in the classpath.
>
> ``` TEST_JAR=`echo
> $REEF_HOME/reef-tests/target/reef-tests-*-tests.jar`
> DEPENDENCY_JAR=`echo
> $REEF_HOME/reef-tests/target/reef-tests-*-test-jar-with-dependencies.jar`
>
>
> CLASSPATH=$YARN_HOME/share/hadoop/common/*:$YARN_HOME/share/hadoop/common/lib/*:$YARN_HOME/share/hadoop/yarn/*:$YARN_HOME/share/hadoop/hdfs/*:$YARN_HOME/share/hadoop/mapreduce/lib/*:$YARN_HOME/share/hadoop/mapreduce/*
>
> YARN_CONF_DIR=$YARN_HOME/etc/hadoop
>
> CMD="java -cp $YARN_CONF_DIR:$DEPENDENCY_JAR:$TEST_JAR:$CLASSPATH
> org.junit.runner.JUnitCore org.apache.reef.tests.AllTestsSuite $*"
> echo $CMD $CMD ```
>
> I’ve found that Hadoop configuration directory and jars must be
> added to the classpath. (Hadoop jars are not found within
> test-jar-with-dependencies.) If these are not added to the
> classpath, the tests fail because the _client_ refuses to run with
> a `NoClassDefFound` exception.
>
> ## Results
>
> ``` Tests run: 28, Failures: 13 ```
>
> Note that many jobs (`reef-job-TEST_EvaluatorSizeTest-777`,
> `reef-job-TEST_FileResourceTest`, etc.) fail immediately on the
> cluster, but the test runner can only proceed after the 5 minute
> timeout. So even if the test runner appears stalled, it will
> continue after 5 minutes. It may make sense to adjust
> `YarnTestEnvironment.getTestTimeout` while debugging these jobs. I
> believe the jobs that wait for timeout are unexpected failures,
> while those that continue immediately are failures expected by the
> tests.
>
> ## Some pointers
>
> The CMS cluster is setup with log aggregation. So, once a job has
> finished, the log is available using the command `yarn logs
> -applicationId <appId>`
>
> ## Debugging notes
>
> The logs reveal the job failures seem to stem from some form of
> “class not found” errors. Some examples: - `Error: Could not find
> or load main class org.apache.reef.runtime.common.Launcher` -
> Caused by: java.lang.ClassNotFoundException:
> org.apache.reef.tests.fail.FailDriverTest (seen once)
>
> `Launcher` is probably needed by all REEF apps, but it is only
> sporadically not found. This is the most puzzling part so far.
>
>
> On Mon, Dec 22, 2014 at 6:12 PM, Byung-Gon Chun <bg...@gmail.com>
> wrote:
>
>> Brian's on honeymoon. :) So, John and I have taken over YARN
>> tests in our Linux cluster.
>>
>> It'd be great if we can run YARN tests on the windows cluster,
>> too.
>>
>> Thanks. -Gon
>>
>> On Thu, Dec 18, 2014 at 5:01 PM, Brian Cho <ch...@gmail.com>
>> wrote:
>>
>>> Yes, the branch passes tests locally.
>>>
>>> We'll have to fix YARN tests. I haven't figured it out quite
>>> yet. The fix should be contributed to the main branch, which we
>>> could then merge to release branch.
>>>
>>> -Brian
>>>
>>> On Thu, Dec 18, 2014 at 4:57 PM, Byung-Gon Chun
>>> <bg...@gmail.com>
>> wrote:
>>>>
>>>> Thanks, Brian.
>>>>
>>>> Did you test the branch locally?
>>>>
>>>> What should we do about YARN tests?
>>>>
>>>>
>>>>
>>>> On Thu, Dec 18, 2014 at 4:31 PM, Brian Cho
>>>> <ch...@gmail.com> wrote:
>>>>>
>>>>> On Thu, Dec 18, 2014 at 10:09 AM, Byung-Gon Chun
>>>>> <bg...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Thanks for the input.
>>>>>>
>>>>>> Sure. I will publish the git hash of the version after
>>>>>> creating a
>>>> release
>>>>>> branch (following apache incubation naming).
>>>>>>
>>>>>> I will update basic files on the master branch.
>>>>>>
>>>>>> Brian will do the removal of the reef-bridge in the
>>>>>> release branch. Brian, please check the naming convention
>>>>>> of the release branch in
>>>>> Apache.
>>>>>>
>>>>>>
>>>>> I've added the release branch (branch-0.10.0-incubating)
>>>>> and pushed
>> the
>>>>> reef-bridge removal commit. I would have liked to issue a
>>>>> pull
>> request
>>>> for
>>>>> this commit, but it seems github mirroring for a new branch
>>>>> only
>> kicks
>>> in
>>>>> once there is a first commit. So, I didn't have a branch on
>>>>> github to
>>>> issue
>>>>> the pull request for at the time of the commit. In future,
>>>>> we may
>> want
>>> to
>>>>> make a trivial commit after creating a branch (although
>>>>> these hacks
>>>> always
>>>>> make me cringe).
>>>>>
>>>>> Thanks, Brian
>>>>>
>>>>>
>>>>>
>>>>>> Thanks. -Gon
>>>>>>
>>>>>>
>>>>>> On Thu, Dec 18, 2014 at 2:32 AM, Markus Weimer
>>>>>> <ma...@weimo.de>
>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'd like to make sure that we have the released version
>>>>>>> in git as
>>>> well.
>>>>>>> That means that we should publish the git hash of the
>>>>>>> version
>> that
>>>> was
>>>>>>> used to make the tarball. That way, we can always go
>>>>>>> back and
>>>> backport
>>>>>>> stuff as it becomes needed.
>>>>>>>
>>>>>>> For that, can we create a release branch? E.g.
>>>>>>> "release-0.10" or
>>>> such?
>>>>>>> However, some of the work below (updating LICENSE,
>>>>>>> DISCLAIMER,
>>>> NOTICES,
>>>>>>> add KEYS, ...) can be done immediately against the
>>>>>>> master
>> branch. I
>>>>>>> believe the license headers are alright, given that
>>>>>>> apache-rat
>>>> passes.
>>>>>>>
>>>>>>> Some things will be subtractive and should only be done
>>>>>>> in the
>>>> branch.
>>>>>>> First and foremost, that is the removal of reef-bridge
>>>>>>> both from
>>> git
>>>>> and
>>>>>>> the master pom. That code won't work in open source
>>>>>>> until REEF-51
>>> is
>>>>>>> resolved anyhow. I'm not sure what else will have to be
>>>>>>> removed.
>>>>>>>
>>>>>>> Markus
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2014-12-17 08:01, Byung-Gon Chun wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Warning. This email is long.
>>>>>>>>
>>>>>>>> Release voting is done in two phases. First, the PPMC
>>>>>>>> votes on a release once it's ready by the
>> release
>>>>>> manager
>>>>>>>> (me this time). If that passes, we call a vote on the
>>>>>>>> release;
>>>>>> incubator
>>>>>>>> PMC members vote.
>>>>>>>>
>>>>>>>> PPMC should check the following for the release. 1.
>>>>>>>> checksums and PGP signatures are valid The release
>>>>>>>> candidate is signed with PGP key ....., which is
>>>> included
>>>>>>>> in the repository's KEYS file: <url>
>>>>>>>>
>>>>>>>> checksums: sha1 and/or md5 checksums
>>>>>>>>
>>>>>>>> 2. Build is successful including automated tests
>>>>>>>>
>>>>>>>> 3. DISCLAIMER is correct. filenames include
>>>>>>>> "incubating". The file name should be
>>>>>>>> 0.10.0-incubating.
>>>>>>>>
>>>>>>>> We have to add DISCLAIMER.
>>>>>>>>
>>>>>>>> Podling web sites MUST include a clear disclaimer on
>>>>>>>> their
>>> website
>>>>> and
>>>>>> in
>>>>>>>> all documentation (including releases) stating that
>>>>>>>> they are in
>>>>>>> incubation.
>>>>>>>> Podlings SHOULD use the following text for all
>>>>>>>> disclaimers
>>> (replace
>>>>> the
>>>>>>>> underlined phrases as appropriate):
>>>>>>>>
>>>>>>>> Apache *Podling-Name* is an effort undergoing
>>>>>>>> incubation at The
>>>>> Apache
>>>>>>>> Software Foundation (ASF), sponsored by the *name of
>>>>>>>> Apache TLP
>>>>>> sponsor*.
>>>>>>>> Incubation is required of all newly accepted projects
>>>>>>>> until a
>>>> further
>>>>>>>> review indicates that the infrastructure,
>>>>>>>> communications, and
>>>>> decision
>>>>>>>> making process have stabilized in a manner consistent
>>>>>>>> with
>> other
>>>>>>> successful
>>>>>>>> ASF projects. While incubation status is not
>>>>>>>> necessarily a
>>>> reflection
>>>>>> of
>>>>>>>> the completeness or stability of the code, it does
>>>>>>>> indicate
>> that
>>>> the
>>>>>>>> project has yet to be fully endorsed by the ASF.
>>>>>>>>
>>>>>>>> Podlings wishing to use a different disclaimer
>>>>>>>> message MUST
>> have
>>>> the
>>>>>>>> disclaimer approved by the Incubator PMC prior to
>>>>>>>> use.
>>>>>>>>
>>>>>>>> For releases, the text SHOULD be included in a
>>>>>>>> separate
>>> DISCLAIMER
>>>>> file
>>>>>>>> stored alongside the NOTICE and LICENSE files.
>>>>>>>>
>>>>>>>>
>>>>>>>> 4. Top-level LICENSE and NOTICE are correct for each
>>> distribution.
>>>>>>>>
>>>>>>>> I believe we don't have any bundled dependencies. In
>>>>>>>> that case,
>>> we
>>>>> can
>>>>>> go
>>>>>>>> with an easy path.
>>>>>>>>
>>>>>>>> LICENSE should contain the text of the ALv2
>>>>>>>> <http://apache.org/licenses/LICENSE-2.0.txt> -- no
>>>>>>>> more, no
>>> less.
>>>>>>>>
>>>>>>>> NOTICE should contain only the following text,
>>>>>>>> adapted with the
>>>>>> product's
>>>>>>>> name and copyright dates.
>>>>>>>>
>>>>>>>> Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache
>>>>>>>> Software
>>>>>> Foundation
>>>>>>>> This product includes software developed atThe Apache
>>>>>>>> Software Foundation (http://www.apache.org/).
>>>>>>>>
>>>>>>>> If not, we should add bundled dependency
>>>>>>>> information.
>>>>>>>>
>>>>>>>> 5. All source files have license headers where
>>>>>>>> appropriate. I believe rat checks this.
>>>>>>>>
>>>>>>>> 6. The provenance of all source files is clear (ASF
>>>>>>>> or software
>>>>>> grants).
>>>>>>>> We took care of SGA and CLA. Right? So, we just need
>>>>>>>> to get approval from IPMC members by voting.
>>>>>>>>
>>>>>>>> 7. Dependency licenses are ok. We cannot have the
>>>>>>>> following
>>>>> dependency
>>>>>>>> license.
>>>>>>>>
>>>>>>>> - Binary Code License (BCL) - Special exceptions to
>>>>>>>> the GNU GPL (e.g. GNU Classpath) - GNU GPL 1, 2, 3 <
>>>>>> http://www.opensource.org/licenses/gpl-license.php
>>>>>>>>
>>>>>>>> - GNU LGPL 2, 2.1, 3
>>>>>>>> <http://www.opensource.org/licenses/lgpl-license.php>
>>>>>>>>
>>>>>>>>
- Affero GPL 3 <
>>> http://www.opensource.org/licenses/agpl-v3.html
>>>>>
>>>>>>>> - NPL 1.0/NPL 1.1 - QPL
>>>>>>>> <http://doc.trolltech.com/3.0/license.html> -
>>>>>>>> Sleepycat License <
>>>>>> http://www.opensource.org/licenses/sleepycat.php
>>>>>>>>
>>>>>>>> - Microsoft Limited Public License
>>>>>>>> <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
>>>>>>>> - Code Project Open License (CPOL)
>>>>>>>>
>>>>>>>> 8. Release consists of source code only, no
>>>>>>>> binaries. We should not have any dll, so, jar.
>>>>>>>>
>>>>>>>> * Some like to add notices (such as RELEASE_NOTES,
>>>>>>>> READMEand
>>>>> CHANGES)
>>>>>>> for
>>>>>>>> each release. CHANGES lists fixed JIRA issues.
>>>>>>>>
>>>>>>>> After addressing the above, I will create a source
>>>>>>>> tarball
>>>>> (*.tar.gz),
>>>>>>>> signature (*.asc), checksum (*.sha,*.md5) and post
>>>>>>>> them in my
>>>> apache
>>>>>>>> account, which is a typical way of posting RCs.
>>>>>>>>
>>>>>>>> I will send out the SHA1 checksum of the archive, the
>>>>>>>> commit id to be voted upon, the tag to be voted upon,
>>>>>>>> the
>>> list
>>>> of
>>>>>>> fixed
>>>>>>>> issues, a link to the keys to verify the signature of
>>>>>>>> the
>> release
>>>>>>> artifact.
>>>>>>>>
>>>>>>>> Once the incubator PMC approves the release, we
>>>>>>>> upload the
>>> release
>>>> to
>>>>>> the
>>>>>>>> incubator distribution directory, podling
>>>>>>>> distribution
>> directory,
>>>> and
>>>>>> our
>>>>>>>> own web site.
>>>>>>>>
>>>>>>>> Let me know if you have any questions.
>>>>>>>>
>>>>>>>> -Gon
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <
>>> bgchun@gmail.com>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> The release management is a bit more complicated
>>>>>>>>> than I
>> thought.
>>>>>>>>> There're things I am figuring out since this is the
>>>>>>>>> first one. Hopefully I will send out an update
>>>>>>>>> today. Stay tuned.
>>>>>>>>>
>>>>>>>>> -Gon
>>>>>>>>>
>>>>>>>>> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <
>>> markus@weimo.de>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> any updates on the schedule or task list for the
>>>>>>>>>> rest of us
>> to
>>>> get
>>>>>> this
>>>>>>>>>> done?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Markus
>>>>>>>>>>
>>>>>>>>>> -----Original Message----- From: "Byung-Gon Chun"
>>>>>>>>>> <bg...@gmail.com> Sent: 12/5/2014 16:06 To:
>>>>>>>>>> "dev@reef.incubator.apache.org" <
>>>> dev@reef.incubator.apache.org
>>>>>>
>>>>>>>>>> Subject: Tentative schedule of the first release:
>>>>>>>>>> week of
>> Dec.
>>> 15
>>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> This is a quick update on the first release. The
>>>>>>>>>> first release of Apache REEF will be done in the
>>>>>>>>>> week of
>>> Dec.
>>>>> 15.
>>>>>>>>>> I will give you more details about the release
>>>>>>>>>> process in
>>> Apache
>>>>>> later.
>>>>>>>>>>
>>>>>>>>>> Meanwhile, if you have features you want to
>>>>>>>>>> include in the
>>> first
>>>>>>> release,
>>>>>>>>>> please do so before Dec. 15.
>>>>>>>>>>
>>>>>>>>>> Cheers, Gon
>>>>>>>>>>
>>>>>>>>>> -- Byung-Gon Chun
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- Byung-Gon Chun
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> -- Byung-Gon Chun
>>>>>>
>>>>>
>>>>
>>>>
>>>> -- Byung-Gon Chun
>>>>
>>>
>>
>>
>>
>> -- Byung-Gon Chun
>>
>
Re: Tentative schedule of the first release: week of Dec. 15
Posted by John Yang <jo...@gmail.com>.
Hello everyone,
I'd like to share my findings on this.
1.
I've reproduced Brian's result(attached below) using his script in our
lab's Cluster. It seems that the tests that use the jar of this.getClass()
as the GLOBAL_LIBRARIES produce the Launcher not found error.
Error(Launcher not found):
.set(DriverConfiguration.GLOBAL_LIBRARIES,
EnvironmentUtils.getClassLocation(this.getClass()))
Working:
.set(DriverConfiguration.GLOBAL_LIBRARIES,
EnvironmentUtils.getClassLocation(EvaluatorReuseTestDriver.class))
2.
On my local computer, I get 1~2 failures(sometimes 1 and sometimes 2),
which are not caused by Launcher not found error(NullPointer, etc) though.
3.
I've also tried "mvn clean install" and failed. The hardest part is running
the tests against a single shaded(or jar-with-dependency) jar, which I did
not succeed. The reason why Launcher not found error occurs is that if the
tests are not run against a single jar with that contains all the
dependencies, by default "EnvironmentUtils.getClassLocation(CLASS_XX)",
only include the test classes located in /target/test-classes. Because only
test classes are sent to the Driver, the Driver cannot find Launcher.
Tomorrow, I'll try to find the source of the "this.getClass()" problem and
give running tests against a shaded jar another try.
Best regards,
John
[Attached]Brian's doc:
===
# reef-tests on YARN
This document contains the current status of trying to get reef-tests to
run on a YARN cluster. Running using maven, as the local runtime does, is
not working. Instead, the current attempt is to run via JUnit.
## Running via JUnit
- Prerequisites
- Build reef by running
`mvn clean install -DskipTests`
- Build reef-tests jar by running
`mvn jar:test-jar`
from the reef-tests directory
- Run reef-tests by running JUnitCore. The bash script below runs with the
reef-tests jar, test-jar-with-dependencies, Hadoop conf dir, and Hadoop
jars in the classpath.
```
TEST_JAR=`echo $REEF_HOME/reef-tests/target/reef-tests-*-tests.jar`
DEPENDENCY_JAR=`echo
$REEF_HOME/reef-tests/target/reef-tests-*-test-jar-with-dependencies.jar`
CLASSPATH=$YARN_HOME/share/hadoop/common/*:$YARN_HOME/share/hadoop/common/lib/*:$YARN_HOME/share/hadoop/yarn/*:$YARN_HOME/share/hadoop/hdfs/*:$YARN_HOME/share/hadoop/mapreduce/lib/*:$YARN_HOME/share/hadoop/mapreduce/*
YARN_CONF_DIR=$YARN_HOME/etc/hadoop
CMD="java -cp $YARN_CONF_DIR:$DEPENDENCY_JAR:$TEST_JAR:$CLASSPATH
org.junit.runner.JUnitCore org.apache.reef.tests.AllTestsSuite $*"
echo $CMD
$CMD
```
I’ve found that Hadoop configuration directory and jars must be added to
the classpath. (Hadoop jars are not found within
test-jar-with-dependencies.) If these are not added to the classpath, the
tests fail because the _client_ refuses to run with a `NoClassDefFound`
exception.
## Results
```
Tests run: 28, Failures: 13
```
Note that many jobs (`reef-job-TEST_EvaluatorSizeTest-777`,
`reef-job-TEST_FileResourceTest`, etc.) fail immediately on the cluster,
but the test runner can only proceed after the 5 minute timeout. So even if
the test runner appears stalled, it will continue after 5 minutes. It may
make sense to adjust `YarnTestEnvironment.getTestTimeout` while debugging
these jobs. I believe the jobs that wait for timeout are unexpected
failures, while those that continue immediately are failures expected by
the tests.
## Some pointers
The CMS cluster is setup with log aggregation. So, once a job has finished,
the log is available using the command
`yarn logs -applicationId <appId>`
## Debugging notes
The logs reveal the job failures seem to stem from some form of “class not
found” errors. Some examples:
- `Error: Could not find or load main class
org.apache.reef.runtime.common.Launcher`
- Caused by: java.lang.ClassNotFoundException:
org.apache.reef.tests.fail.FailDriverTest (seen once)
`Launcher` is probably needed by all REEF apps, but it is only sporadically
not found. This is the most puzzling part so far.
On Mon, Dec 22, 2014 at 6:12 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
> Brian's on honeymoon. :)
> So, John and I have taken over YARN tests in our Linux cluster.
>
> It'd be great if we can run YARN tests on the windows cluster, too.
>
> Thanks.
> -Gon
>
> On Thu, Dec 18, 2014 at 5:01 PM, Brian Cho <ch...@gmail.com> wrote:
>
> > Yes, the branch passes tests locally.
> >
> > We'll have to fix YARN tests. I haven't figured it out quite yet. The fix
> > should be contributed to the main branch, which we could then merge to
> > release branch.
> >
> > -Brian
> >
> > On Thu, Dec 18, 2014 at 4:57 PM, Byung-Gon Chun <bg...@gmail.com>
> wrote:
> > >
> > > Thanks, Brian.
> > >
> > > Did you test the branch locally?
> > >
> > > What should we do about YARN tests?
> > >
> > >
> > >
> > > On Thu, Dec 18, 2014 at 4:31 PM, Brian Cho <ch...@gmail.com> wrote:
> > > >
> > > > On Thu, Dec 18, 2014 at 10:09 AM, Byung-Gon Chun <bg...@gmail.com>
> > > wrote:
> > > > >
> > > > > Thanks for the input.
> > > > >
> > > > > Sure. I will publish the git hash of the version after creating a
> > > release
> > > > > branch (following apache incubation naming).
> > > > >
> > > > > I will update basic files on the master branch.
> > > > >
> > > > > Brian will do the removal of the reef-bridge in the release branch.
> > > > > Brian, please check the naming convention of the release branch in
> > > > Apache.
> > > > >
> > > > >
> > > > I've added the release branch (branch-0.10.0-incubating) and pushed
> the
> > > > reef-bridge removal commit. I would have liked to issue a pull
> request
> > > for
> > > > this commit, but it seems github mirroring for a new branch only
> kicks
> > in
> > > > once there is a first commit. So, I didn't have a branch on github to
> > > issue
> > > > the pull request for at the time of the commit. In future, we may
> want
> > to
> > > > make a trivial commit after creating a branch (although these hacks
> > > always
> > > > make me cringe).
> > > >
> > > > Thanks,
> > > > Brian
> > > >
> > > >
> > > >
> > > > > Thanks.
> > > > > -Gon
> > > > >
> > > > >
> > > > > On Thu, Dec 18, 2014 at 2:32 AM, Markus Weimer <ma...@weimo.de>
> > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'd like to make sure that we have the released version in git as
> > > well.
> > > > > > That means that we should publish the git hash of the version
> that
> > > was
> > > > > > used to make the tarball. That way, we can always go back and
> > > backport
> > > > > > stuff as it becomes needed.
> > > > > >
> > > > > > For that, can we create a release branch? E.g. "release-0.10" or
> > > such?
> > > > > > However, some of the work below (updating LICENSE, DISCLAIMER,
> > > NOTICES,
> > > > > > add KEYS, ...) can be done immediately against the master
> branch. I
> > > > > > believe the license headers are alright, given that apache-rat
> > > passes.
> > > > > >
> > > > > > Some things will be subtractive and should only be done in the
> > > branch.
> > > > > > First and foremost, that is the removal of reef-bridge both from
> > git
> > > > and
> > > > > > the master pom. That code won't work in open source until REEF-51
> > is
> > > > > > resolved anyhow. I'm not sure what else will have to be removed.
> > > > > >
> > > > > > Markus
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 2014-12-17 08:01, Byung-Gon Chun wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > Warning. This email is long.
> > > > > > >
> > > > > > > Release voting is done in two phases.
> > > > > > > First, the PPMC votes on a release once it's ready by the
> release
> > > > > manager
> > > > > > > (me this time). If that passes, we call a vote on the release;
> > > > > incubator
> > > > > > > PMC members vote.
> > > > > > >
> > > > > > > PPMC should check the following for the release.
> > > > > > > 1. checksums and PGP signatures are valid
> > > > > > > The release candidate is signed with PGP key ....., which is
> > > included
> > > > > > > in the repository's KEYS file: <url>
> > > > > > >
> > > > > > > checksums: sha1 and/or md5 checksums
> > > > > > >
> > > > > > > 2. Build is successful including automated tests
> > > > > > >
> > > > > > > 3. DISCLAIMER is correct. filenames include "incubating".
> > > > > > > The file name should be 0.10.0-incubating.
> > > > > > >
> > > > > > > We have to add DISCLAIMER.
> > > > > > >
> > > > > > > Podling web sites MUST include a clear disclaimer on their
> > website
> > > > and
> > > > > in
> > > > > > > all documentation (including releases) stating that they are in
> > > > > > incubation.
> > > > > > > Podlings SHOULD use the following text for all disclaimers
> > (replace
> > > > the
> > > > > > > underlined phrases as appropriate):
> > > > > > >
> > > > > > > Apache *Podling-Name* is an effort undergoing incubation at The
> > > > Apache
> > > > > > > Software Foundation (ASF), sponsored by the *name of Apache TLP
> > > > > sponsor*.
> > > > > > > Incubation is required of all newly accepted projects until a
> > > further
> > > > > > > review indicates that the infrastructure, communications, and
> > > > decision
> > > > > > > making process have stabilized in a manner consistent with
> other
> > > > > > successful
> > > > > > > ASF projects. While incubation status is not necessarily a
> > > reflection
> > > > > of
> > > > > > > the completeness or stability of the code, it does indicate
> that
> > > the
> > > > > > > project has yet to be fully endorsed by the ASF.
> > > > > > >
> > > > > > > Podlings wishing to use a different disclaimer message MUST
> have
> > > the
> > > > > > > disclaimer approved by the Incubator PMC prior to use.
> > > > > > >
> > > > > > > For releases, the text SHOULD be included in a separate
> > DISCLAIMER
> > > > file
> > > > > > > stored alongside the NOTICE and LICENSE files.
> > > > > > >
> > > > > > >
> > > > > > > 4. Top-level LICENSE and NOTICE are correct for each
> > distribution.
> > > > > > >
> > > > > > > I believe we don't have any bundled dependencies. In that case,
> > we
> > > > can
> > > > > go
> > > > > > > with an easy path.
> > > > > > >
> > > > > > > LICENSE should contain the text of the ALv2
> > > > > > > <http://apache.org/licenses/LICENSE-2.0.txt> -- no more, no
> > less.
> > > > > > >
> > > > > > > NOTICE should contain only the following text, adapted with the
> > > > > product's
> > > > > > > name and copyright dates.
> > > > > > >
> > > > > > > Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache Software
> > > > > Foundation
> > > > > > > This product includes software developed atThe Apache Software
> > > > > > > Foundation (http://www.apache.org/).
> > > > > > >
> > > > > > > If not, we should add bundled dependency information.
> > > > > > >
> > > > > > > 5. All source files have license headers where appropriate.
> > > > > > > I believe rat checks this.
> > > > > > >
> > > > > > > 6. The provenance of all source files is clear (ASF or software
> > > > > grants).
> > > > > > > We took care of SGA and CLA. Right?
> > > > > > > So, we just need to get approval from IPMC members by voting.
> > > > > > >
> > > > > > > 7. Dependency licenses are ok. We cannot have the following
> > > > dependency
> > > > > > > license.
> > > > > > >
> > > > > > > - Binary Code License (BCL)
> > > > > > > - Special exceptions to the GNU GPL (e.g. GNU Classpath)
> > > > > > > - GNU GPL 1, 2, 3 <
> > > > > http://www.opensource.org/licenses/gpl-license.php
> > > > > > >
> > > > > > > - GNU LGPL 2, 2.1, 3
> > > > > > > <http://www.opensource.org/licenses/lgpl-license.php>
> > > > > > > - Affero GPL 3 <
> > http://www.opensource.org/licenses/agpl-v3.html
> > > >
> > > > > > > - NPL 1.0/NPL 1.1
> > > > > > > - QPL <http://doc.trolltech.com/3.0/license.html>
> > > > > > > - Sleepycat License <
> > > > > http://www.opensource.org/licenses/sleepycat.php
> > > > > > >
> > > > > > > - Microsoft Limited Public License
> > > > > > > <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
> > > > > > > - Code Project Open License (CPOL)
> > > > > > >
> > > > > > > 8. Release consists of source code only, no binaries.
> > > > > > > We should not have any dll, so, jar.
> > > > > > >
> > > > > > > * Some like to add notices (such as RELEASE_NOTES, READMEand
> > > > CHANGES)
> > > > > > for
> > > > > > > each release.
> > > > > > > CHANGES lists fixed JIRA issues.
> > > > > > >
> > > > > > > After addressing the above, I will create a source tarball
> > > > (*.tar.gz),
> > > > > > > signature (*.asc), checksum (*.sha,*.md5) and post them in my
> > > apache
> > > > > > > account, which is a typical way of posting RCs.
> > > > > > >
> > > > > > > I will send out the SHA1 checksum of the archive,
> > > > > > > the commit id to be voted upon, the tag to be voted upon, the
> > list
> > > of
> > > > > > fixed
> > > > > > > issues, a link to the keys to verify the signature of the
> release
> > > > > > artifact.
> > > > > > >
> > > > > > > Once the incubator PMC approves the release, we upload the
> > release
> > > to
> > > > > the
> > > > > > > incubator distribution directory, podling distribution
> directory,
> > > and
> > > > > our
> > > > > > > own web site.
> > > > > > >
> > > > > > > Let me know if you have any questions.
> > > > > > >
> > > > > > > -Gon
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <
> > bgchun@gmail.com>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> The release management is a bit more complicated than I
> thought.
> > > > > > >> There're things I am figuring out since this is the first one.
> > > > > > >> Hopefully I will send out an update today. Stay tuned.
> > > > > > >>
> > > > > > >> -Gon
> > > > > > >>
> > > > > > >> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <
> > markus@weimo.de>
> > > > > > wrote:
> > > > > > >>>
> > > > > > >>> Hi,
> > > > > > >>>
> > > > > > >>> any updates on the schedule or task list for the rest of us
> to
> > > get
> > > > > this
> > > > > > >>> done?
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>>
> > > > > > >>> Markus
> > > > > > >>>
> > > > > > >>> -----Original Message-----
> > > > > > >>> From: "Byung-Gon Chun" <bg...@gmail.com>
> > > > > > >>> Sent: 12/5/2014 16:06
> > > > > > >>> To: "dev@reef.incubator.apache.org" <
> > > dev@reef.incubator.apache.org
> > > > >
> > > > > > >>> Subject: Tentative schedule of the first release: week of
> Dec.
> > 15
> > > > > > >>>
> > > > > > >>> Hi all,
> > > > > > >>>
> > > > > > >>> This is a quick update on the first release.
> > > > > > >>> The first release of Apache REEF will be done in the week of
> > Dec.
> > > > 15.
> > > > > > >>> I will give you more details about the release process in
> > Apache
> > > > > later.
> > > > > > >>>
> > > > > > >>> Meanwhile, if you have features you want to include in the
> > first
> > > > > > release,
> > > > > > >>> please do so before Dec. 15.
> > > > > > >>>
> > > > > > >>> Cheers,
> > > > > > >>> Gon
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> Byung-Gon Chun
> > > > > > >>>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Byung-Gon Chun
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Byung-Gon Chun
> > > > >
> > > >
> > >
> > >
> > > --
> > > Byung-Gon Chun
> > >
> >
>
>
>
> --
> Byung-Gon Chun
>
Re: Tentative schedule of the first release: week of Dec. 15
Posted by Byung-Gon Chun <bg...@gmail.com>.
Brian's on honeymoon. :)
So, John and I have taken over YARN tests in our Linux cluster.
It'd be great if we can run YARN tests on the windows cluster, too.
Thanks.
-Gon
On Thu, Dec 18, 2014 at 5:01 PM, Brian Cho <ch...@gmail.com> wrote:
> Yes, the branch passes tests locally.
>
> We'll have to fix YARN tests. I haven't figured it out quite yet. The fix
> should be contributed to the main branch, which we could then merge to
> release branch.
>
> -Brian
>
> On Thu, Dec 18, 2014 at 4:57 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
> >
> > Thanks, Brian.
> >
> > Did you test the branch locally?
> >
> > What should we do about YARN tests?
> >
> >
> >
> > On Thu, Dec 18, 2014 at 4:31 PM, Brian Cho <ch...@gmail.com> wrote:
> > >
> > > On Thu, Dec 18, 2014 at 10:09 AM, Byung-Gon Chun <bg...@gmail.com>
> > wrote:
> > > >
> > > > Thanks for the input.
> > > >
> > > > Sure. I will publish the git hash of the version after creating a
> > release
> > > > branch (following apache incubation naming).
> > > >
> > > > I will update basic files on the master branch.
> > > >
> > > > Brian will do the removal of the reef-bridge in the release branch.
> > > > Brian, please check the naming convention of the release branch in
> > > Apache.
> > > >
> > > >
> > > I've added the release branch (branch-0.10.0-incubating) and pushed the
> > > reef-bridge removal commit. I would have liked to issue a pull request
> > for
> > > this commit, but it seems github mirroring for a new branch only kicks
> in
> > > once there is a first commit. So, I didn't have a branch on github to
> > issue
> > > the pull request for at the time of the commit. In future, we may want
> to
> > > make a trivial commit after creating a branch (although these hacks
> > always
> > > make me cringe).
> > >
> > > Thanks,
> > > Brian
> > >
> > >
> > >
> > > > Thanks.
> > > > -Gon
> > > >
> > > >
> > > > On Thu, Dec 18, 2014 at 2:32 AM, Markus Weimer <ma...@weimo.de>
> > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'd like to make sure that we have the released version in git as
> > well.
> > > > > That means that we should publish the git hash of the version that
> > was
> > > > > used to make the tarball. That way, we can always go back and
> > backport
> > > > > stuff as it becomes needed.
> > > > >
> > > > > For that, can we create a release branch? E.g. "release-0.10" or
> > such?
> > > > > However, some of the work below (updating LICENSE, DISCLAIMER,
> > NOTICES,
> > > > > add KEYS, ...) can be done immediately against the master branch. I
> > > > > believe the license headers are alright, given that apache-rat
> > passes.
> > > > >
> > > > > Some things will be subtractive and should only be done in the
> > branch.
> > > > > First and foremost, that is the removal of reef-bridge both from
> git
> > > and
> > > > > the master pom. That code won't work in open source until REEF-51
> is
> > > > > resolved anyhow. I'm not sure what else will have to be removed.
> > > > >
> > > > > Markus
> > > > >
> > > > >
> > > > >
> > > > > On 2014-12-17 08:01, Byung-Gon Chun wrote:
> > > > > > Hi,
> > > > > >
> > > > > > Warning. This email is long.
> > > > > >
> > > > > > Release voting is done in two phases.
> > > > > > First, the PPMC votes on a release once it's ready by the release
> > > > manager
> > > > > > (me this time). If that passes, we call a vote on the release;
> > > > incubator
> > > > > > PMC members vote.
> > > > > >
> > > > > > PPMC should check the following for the release.
> > > > > > 1. checksums and PGP signatures are valid
> > > > > > The release candidate is signed with PGP key ....., which is
> > included
> > > > > > in the repository's KEYS file: <url>
> > > > > >
> > > > > > checksums: sha1 and/or md5 checksums
> > > > > >
> > > > > > 2. Build is successful including automated tests
> > > > > >
> > > > > > 3. DISCLAIMER is correct. filenames include "incubating".
> > > > > > The file name should be 0.10.0-incubating.
> > > > > >
> > > > > > We have to add DISCLAIMER.
> > > > > >
> > > > > > Podling web sites MUST include a clear disclaimer on their
> website
> > > and
> > > > in
> > > > > > all documentation (including releases) stating that they are in
> > > > > incubation.
> > > > > > Podlings SHOULD use the following text for all disclaimers
> (replace
> > > the
> > > > > > underlined phrases as appropriate):
> > > > > >
> > > > > > Apache *Podling-Name* is an effort undergoing incubation at The
> > > Apache
> > > > > > Software Foundation (ASF), sponsored by the *name of Apache TLP
> > > > sponsor*.
> > > > > > Incubation is required of all newly accepted projects until a
> > further
> > > > > > review indicates that the infrastructure, communications, and
> > > decision
> > > > > > making process have stabilized in a manner consistent with other
> > > > > successful
> > > > > > ASF projects. While incubation status is not necessarily a
> > reflection
> > > > of
> > > > > > the completeness or stability of the code, it does indicate that
> > the
> > > > > > project has yet to be fully endorsed by the ASF.
> > > > > >
> > > > > > Podlings wishing to use a different disclaimer message MUST have
> > the
> > > > > > disclaimer approved by the Incubator PMC prior to use.
> > > > > >
> > > > > > For releases, the text SHOULD be included in a separate
> DISCLAIMER
> > > file
> > > > > > stored alongside the NOTICE and LICENSE files.
> > > > > >
> > > > > >
> > > > > > 4. Top-level LICENSE and NOTICE are correct for each
> distribution.
> > > > > >
> > > > > > I believe we don't have any bundled dependencies. In that case,
> we
> > > can
> > > > go
> > > > > > with an easy path.
> > > > > >
> > > > > > LICENSE should contain the text of the ALv2
> > > > > > <http://apache.org/licenses/LICENSE-2.0.txt> -- no more, no
> less.
> > > > > >
> > > > > > NOTICE should contain only the following text, adapted with the
> > > > product's
> > > > > > name and copyright dates.
> > > > > >
> > > > > > Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache Software
> > > > Foundation
> > > > > > This product includes software developed atThe Apache Software
> > > > > > Foundation (http://www.apache.org/).
> > > > > >
> > > > > > If not, we should add bundled dependency information.
> > > > > >
> > > > > > 5. All source files have license headers where appropriate.
> > > > > > I believe rat checks this.
> > > > > >
> > > > > > 6. The provenance of all source files is clear (ASF or software
> > > > grants).
> > > > > > We took care of SGA and CLA. Right?
> > > > > > So, we just need to get approval from IPMC members by voting.
> > > > > >
> > > > > > 7. Dependency licenses are ok. We cannot have the following
> > > dependency
> > > > > > license.
> > > > > >
> > > > > > - Binary Code License (BCL)
> > > > > > - Special exceptions to the GNU GPL (e.g. GNU Classpath)
> > > > > > - GNU GPL 1, 2, 3 <
> > > > http://www.opensource.org/licenses/gpl-license.php
> > > > > >
> > > > > > - GNU LGPL 2, 2.1, 3
> > > > > > <http://www.opensource.org/licenses/lgpl-license.php>
> > > > > > - Affero GPL 3 <
> http://www.opensource.org/licenses/agpl-v3.html
> > >
> > > > > > - NPL 1.0/NPL 1.1
> > > > > > - QPL <http://doc.trolltech.com/3.0/license.html>
> > > > > > - Sleepycat License <
> > > > http://www.opensource.org/licenses/sleepycat.php
> > > > > >
> > > > > > - Microsoft Limited Public License
> > > > > > <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
> > > > > > - Code Project Open License (CPOL)
> > > > > >
> > > > > > 8. Release consists of source code only, no binaries.
> > > > > > We should not have any dll, so, jar.
> > > > > >
> > > > > > * Some like to add notices (such as RELEASE_NOTES, READMEand
> > > CHANGES)
> > > > > for
> > > > > > each release.
> > > > > > CHANGES lists fixed JIRA issues.
> > > > > >
> > > > > > After addressing the above, I will create a source tarball
> > > (*.tar.gz),
> > > > > > signature (*.asc), checksum (*.sha,*.md5) and post them in my
> > apache
> > > > > > account, which is a typical way of posting RCs.
> > > > > >
> > > > > > I will send out the SHA1 checksum of the archive,
> > > > > > the commit id to be voted upon, the tag to be voted upon, the
> list
> > of
> > > > > fixed
> > > > > > issues, a link to the keys to verify the signature of the release
> > > > > artifact.
> > > > > >
> > > > > > Once the incubator PMC approves the release, we upload the
> release
> > to
> > > > the
> > > > > > incubator distribution directory, podling distribution directory,
> > and
> > > > our
> > > > > > own web site.
> > > > > >
> > > > > > Let me know if you have any questions.
> > > > > >
> > > > > > -Gon
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <
> bgchun@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> Hi,
> > > > > >>
> > > > > >> The release management is a bit more complicated than I thought.
> > > > > >> There're things I am figuring out since this is the first one.
> > > > > >> Hopefully I will send out an update today. Stay tuned.
> > > > > >>
> > > > > >> -Gon
> > > > > >>
> > > > > >> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <
> markus@weimo.de>
> > > > > wrote:
> > > > > >>>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> any updates on the schedule or task list for the rest of us to
> > get
> > > > this
> > > > > >>> done?
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>>
> > > > > >>> Markus
> > > > > >>>
> > > > > >>> -----Original Message-----
> > > > > >>> From: "Byung-Gon Chun" <bg...@gmail.com>
> > > > > >>> Sent: 12/5/2014 16:06
> > > > > >>> To: "dev@reef.incubator.apache.org" <
> > dev@reef.incubator.apache.org
> > > >
> > > > > >>> Subject: Tentative schedule of the first release: week of Dec.
> 15
> > > > > >>>
> > > > > >>> Hi all,
> > > > > >>>
> > > > > >>> This is a quick update on the first release.
> > > > > >>> The first release of Apache REEF will be done in the week of
> Dec.
> > > 15.
> > > > > >>> I will give you more details about the release process in
> Apache
> > > > later.
> > > > > >>>
> > > > > >>> Meanwhile, if you have features you want to include in the
> first
> > > > > release,
> > > > > >>> please do so before Dec. 15.
> > > > > >>>
> > > > > >>> Cheers,
> > > > > >>> Gon
> > > > > >>>
> > > > > >>> --
> > > > > >>> Byung-Gon Chun
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Byung-Gon Chun
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Byung-Gon Chun
> > > >
> > >
> >
> >
> > --
> > Byung-Gon Chun
> >
>
--
Byung-Gon Chun
Re: Tentative schedule of the first release: week of Dec. 15
Posted by Brian Cho <ch...@gmail.com>.
Yes, the branch passes tests locally.
We'll have to fix YARN tests. I haven't figured it out quite yet. The fix
should be contributed to the main branch, which we could then merge to
release branch.
-Brian
On Thu, Dec 18, 2014 at 4:57 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
>
> Thanks, Brian.
>
> Did you test the branch locally?
>
> What should we do about YARN tests?
>
>
>
> On Thu, Dec 18, 2014 at 4:31 PM, Brian Cho <ch...@gmail.com> wrote:
> >
> > On Thu, Dec 18, 2014 at 10:09 AM, Byung-Gon Chun <bg...@gmail.com>
> wrote:
> > >
> > > Thanks for the input.
> > >
> > > Sure. I will publish the git hash of the version after creating a
> release
> > > branch (following apache incubation naming).
> > >
> > > I will update basic files on the master branch.
> > >
> > > Brian will do the removal of the reef-bridge in the release branch.
> > > Brian, please check the naming convention of the release branch in
> > Apache.
> > >
> > >
> > I've added the release branch (branch-0.10.0-incubating) and pushed the
> > reef-bridge removal commit. I would have liked to issue a pull request
> for
> > this commit, but it seems github mirroring for a new branch only kicks in
> > once there is a first commit. So, I didn't have a branch on github to
> issue
> > the pull request for at the time of the commit. In future, we may want to
> > make a trivial commit after creating a branch (although these hacks
> always
> > make me cringe).
> >
> > Thanks,
> > Brian
> >
> >
> >
> > > Thanks.
> > > -Gon
> > >
> > >
> > > On Thu, Dec 18, 2014 at 2:32 AM, Markus Weimer <ma...@weimo.de>
> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'd like to make sure that we have the released version in git as
> well.
> > > > That means that we should publish the git hash of the version that
> was
> > > > used to make the tarball. That way, we can always go back and
> backport
> > > > stuff as it becomes needed.
> > > >
> > > > For that, can we create a release branch? E.g. "release-0.10" or
> such?
> > > > However, some of the work below (updating LICENSE, DISCLAIMER,
> NOTICES,
> > > > add KEYS, ...) can be done immediately against the master branch. I
> > > > believe the license headers are alright, given that apache-rat
> passes.
> > > >
> > > > Some things will be subtractive and should only be done in the
> branch.
> > > > First and foremost, that is the removal of reef-bridge both from git
> > and
> > > > the master pom. That code won't work in open source until REEF-51 is
> > > > resolved anyhow. I'm not sure what else will have to be removed.
> > > >
> > > > Markus
> > > >
> > > >
> > > >
> > > > On 2014-12-17 08:01, Byung-Gon Chun wrote:
> > > > > Hi,
> > > > >
> > > > > Warning. This email is long.
> > > > >
> > > > > Release voting is done in two phases.
> > > > > First, the PPMC votes on a release once it's ready by the release
> > > manager
> > > > > (me this time). If that passes, we call a vote on the release;
> > > incubator
> > > > > PMC members vote.
> > > > >
> > > > > PPMC should check the following for the release.
> > > > > 1. checksums and PGP signatures are valid
> > > > > The release candidate is signed with PGP key ....., which is
> included
> > > > > in the repository's KEYS file: <url>
> > > > >
> > > > > checksums: sha1 and/or md5 checksums
> > > > >
> > > > > 2. Build is successful including automated tests
> > > > >
> > > > > 3. DISCLAIMER is correct. filenames include "incubating".
> > > > > The file name should be 0.10.0-incubating.
> > > > >
> > > > > We have to add DISCLAIMER.
> > > > >
> > > > > Podling web sites MUST include a clear disclaimer on their website
> > and
> > > in
> > > > > all documentation (including releases) stating that they are in
> > > > incubation.
> > > > > Podlings SHOULD use the following text for all disclaimers (replace
> > the
> > > > > underlined phrases as appropriate):
> > > > >
> > > > > Apache *Podling-Name* is an effort undergoing incubation at The
> > Apache
> > > > > Software Foundation (ASF), sponsored by the *name of Apache TLP
> > > sponsor*.
> > > > > Incubation is required of all newly accepted projects until a
> further
> > > > > review indicates that the infrastructure, communications, and
> > decision
> > > > > making process have stabilized in a manner consistent with other
> > > > successful
> > > > > ASF projects. While incubation status is not necessarily a
> reflection
> > > of
> > > > > the completeness or stability of the code, it does indicate that
> the
> > > > > project has yet to be fully endorsed by the ASF.
> > > > >
> > > > > Podlings wishing to use a different disclaimer message MUST have
> the
> > > > > disclaimer approved by the Incubator PMC prior to use.
> > > > >
> > > > > For releases, the text SHOULD be included in a separate DISCLAIMER
> > file
> > > > > stored alongside the NOTICE and LICENSE files.
> > > > >
> > > > >
> > > > > 4. Top-level LICENSE and NOTICE are correct for each distribution.
> > > > >
> > > > > I believe we don't have any bundled dependencies. In that case, we
> > can
> > > go
> > > > > with an easy path.
> > > > >
> > > > > LICENSE should contain the text of the ALv2
> > > > > <http://apache.org/licenses/LICENSE-2.0.txt> -- no more, no less.
> > > > >
> > > > > NOTICE should contain only the following text, adapted with the
> > > product's
> > > > > name and copyright dates.
> > > > >
> > > > > Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache Software
> > > Foundation
> > > > > This product includes software developed atThe Apache Software
> > > > > Foundation (http://www.apache.org/).
> > > > >
> > > > > If not, we should add bundled dependency information.
> > > > >
> > > > > 5. All source files have license headers where appropriate.
> > > > > I believe rat checks this.
> > > > >
> > > > > 6. The provenance of all source files is clear (ASF or software
> > > grants).
> > > > > We took care of SGA and CLA. Right?
> > > > > So, we just need to get approval from IPMC members by voting.
> > > > >
> > > > > 7. Dependency licenses are ok. We cannot have the following
> > dependency
> > > > > license.
> > > > >
> > > > > - Binary Code License (BCL)
> > > > > - Special exceptions to the GNU GPL (e.g. GNU Classpath)
> > > > > - GNU GPL 1, 2, 3 <
> > > http://www.opensource.org/licenses/gpl-license.php
> > > > >
> > > > > - GNU LGPL 2, 2.1, 3
> > > > > <http://www.opensource.org/licenses/lgpl-license.php>
> > > > > - Affero GPL 3 <http://www.opensource.org/licenses/agpl-v3.html
> >
> > > > > - NPL 1.0/NPL 1.1
> > > > > - QPL <http://doc.trolltech.com/3.0/license.html>
> > > > > - Sleepycat License <
> > > http://www.opensource.org/licenses/sleepycat.php
> > > > >
> > > > > - Microsoft Limited Public License
> > > > > <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
> > > > > - Code Project Open License (CPOL)
> > > > >
> > > > > 8. Release consists of source code only, no binaries.
> > > > > We should not have any dll, so, jar.
> > > > >
> > > > > * Some like to add notices (such as RELEASE_NOTES, READMEand
> > CHANGES)
> > > > for
> > > > > each release.
> > > > > CHANGES lists fixed JIRA issues.
> > > > >
> > > > > After addressing the above, I will create a source tarball
> > (*.tar.gz),
> > > > > signature (*.asc), checksum (*.sha,*.md5) and post them in my
> apache
> > > > > account, which is a typical way of posting RCs.
> > > > >
> > > > > I will send out the SHA1 checksum of the archive,
> > > > > the commit id to be voted upon, the tag to be voted upon, the list
> of
> > > > fixed
> > > > > issues, a link to the keys to verify the signature of the release
> > > > artifact.
> > > > >
> > > > > Once the incubator PMC approves the release, we upload the release
> to
> > > the
> > > > > incubator distribution directory, podling distribution directory,
> and
> > > our
> > > > > own web site.
> > > > >
> > > > > Let me know if you have any questions.
> > > > >
> > > > > -Gon
> > > > >
> > > > >
> > > > > On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <bg...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> Hi,
> > > > >>
> > > > >> The release management is a bit more complicated than I thought.
> > > > >> There're things I am figuring out since this is the first one.
> > > > >> Hopefully I will send out an update today. Stay tuned.
> > > > >>
> > > > >> -Gon
> > > > >>
> > > > >> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <ma...@weimo.de>
> > > > wrote:
> > > > >>>
> > > > >>> Hi,
> > > > >>>
> > > > >>> any updates on the schedule or task list for the rest of us to
> get
> > > this
> > > > >>> done?
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> Markus
> > > > >>>
> > > > >>> -----Original Message-----
> > > > >>> From: "Byung-Gon Chun" <bg...@gmail.com>
> > > > >>> Sent: 12/5/2014 16:06
> > > > >>> To: "dev@reef.incubator.apache.org" <
> dev@reef.incubator.apache.org
> > >
> > > > >>> Subject: Tentative schedule of the first release: week of Dec. 15
> > > > >>>
> > > > >>> Hi all,
> > > > >>>
> > > > >>> This is a quick update on the first release.
> > > > >>> The first release of Apache REEF will be done in the week of Dec.
> > 15.
> > > > >>> I will give you more details about the release process in Apache
> > > later.
> > > > >>>
> > > > >>> Meanwhile, if you have features you want to include in the first
> > > > release,
> > > > >>> please do so before Dec. 15.
> > > > >>>
> > > > >>> Cheers,
> > > > >>> Gon
> > > > >>>
> > > > >>> --
> > > > >>> Byung-Gon Chun
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Byung-Gon Chun
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Byung-Gon Chun
> > >
> >
>
>
> --
> Byung-Gon Chun
>
Re: Tentative schedule of the first release: week of Dec. 15
Posted by Byung-Gon Chun <bg...@gmail.com>.
Thanks, Brian.
Did you test the branch locally?
What should we do about YARN tests?
On Thu, Dec 18, 2014 at 4:31 PM, Brian Cho <ch...@gmail.com> wrote:
>
> On Thu, Dec 18, 2014 at 10:09 AM, Byung-Gon Chun <bg...@gmail.com> wrote:
> >
> > Thanks for the input.
> >
> > Sure. I will publish the git hash of the version after creating a release
> > branch (following apache incubation naming).
> >
> > I will update basic files on the master branch.
> >
> > Brian will do the removal of the reef-bridge in the release branch.
> > Brian, please check the naming convention of the release branch in
> Apache.
> >
> >
> I've added the release branch (branch-0.10.0-incubating) and pushed the
> reef-bridge removal commit. I would have liked to issue a pull request for
> this commit, but it seems github mirroring for a new branch only kicks in
> once there is a first commit. So, I didn't have a branch on github to issue
> the pull request for at the time of the commit. In future, we may want to
> make a trivial commit after creating a branch (although these hacks always
> make me cringe).
>
> Thanks,
> Brian
>
>
>
> > Thanks.
> > -Gon
> >
> >
> > On Thu, Dec 18, 2014 at 2:32 AM, Markus Weimer <ma...@weimo.de> wrote:
> > >
> > > Hi,
> > >
> > > I'd like to make sure that we have the released version in git as well.
> > > That means that we should publish the git hash of the version that was
> > > used to make the tarball. That way, we can always go back and backport
> > > stuff as it becomes needed.
> > >
> > > For that, can we create a release branch? E.g. "release-0.10" or such?
> > > However, some of the work below (updating LICENSE, DISCLAIMER, NOTICES,
> > > add KEYS, ...) can be done immediately against the master branch. I
> > > believe the license headers are alright, given that apache-rat passes.
> > >
> > > Some things will be subtractive and should only be done in the branch.
> > > First and foremost, that is the removal of reef-bridge both from git
> and
> > > the master pom. That code won't work in open source until REEF-51 is
> > > resolved anyhow. I'm not sure what else will have to be removed.
> > >
> > > Markus
> > >
> > >
> > >
> > > On 2014-12-17 08:01, Byung-Gon Chun wrote:
> > > > Hi,
> > > >
> > > > Warning. This email is long.
> > > >
> > > > Release voting is done in two phases.
> > > > First, the PPMC votes on a release once it's ready by the release
> > manager
> > > > (me this time). If that passes, we call a vote on the release;
> > incubator
> > > > PMC members vote.
> > > >
> > > > PPMC should check the following for the release.
> > > > 1. checksums and PGP signatures are valid
> > > > The release candidate is signed with PGP key ....., which is included
> > > > in the repository's KEYS file: <url>
> > > >
> > > > checksums: sha1 and/or md5 checksums
> > > >
> > > > 2. Build is successful including automated tests
> > > >
> > > > 3. DISCLAIMER is correct. filenames include "incubating".
> > > > The file name should be 0.10.0-incubating.
> > > >
> > > > We have to add DISCLAIMER.
> > > >
> > > > Podling web sites MUST include a clear disclaimer on their website
> and
> > in
> > > > all documentation (including releases) stating that they are in
> > > incubation.
> > > > Podlings SHOULD use the following text for all disclaimers (replace
> the
> > > > underlined phrases as appropriate):
> > > >
> > > > Apache *Podling-Name* is an effort undergoing incubation at The
> Apache
> > > > Software Foundation (ASF), sponsored by the *name of Apache TLP
> > sponsor*.
> > > > Incubation is required of all newly accepted projects until a further
> > > > review indicates that the infrastructure, communications, and
> decision
> > > > making process have stabilized in a manner consistent with other
> > > successful
> > > > ASF projects. While incubation status is not necessarily a reflection
> > of
> > > > the completeness or stability of the code, it does indicate that the
> > > > project has yet to be fully endorsed by the ASF.
> > > >
> > > > Podlings wishing to use a different disclaimer message MUST have the
> > > > disclaimer approved by the Incubator PMC prior to use.
> > > >
> > > > For releases, the text SHOULD be included in a separate DISCLAIMER
> file
> > > > stored alongside the NOTICE and LICENSE files.
> > > >
> > > >
> > > > 4. Top-level LICENSE and NOTICE are correct for each distribution.
> > > >
> > > > I believe we don't have any bundled dependencies. In that case, we
> can
> > go
> > > > with an easy path.
> > > >
> > > > LICENSE should contain the text of the ALv2
> > > > <http://apache.org/licenses/LICENSE-2.0.txt> -- no more, no less.
> > > >
> > > > NOTICE should contain only the following text, adapted with the
> > product's
> > > > name and copyright dates.
> > > >
> > > > Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache Software
> > Foundation
> > > > This product includes software developed atThe Apache Software
> > > > Foundation (http://www.apache.org/).
> > > >
> > > > If not, we should add bundled dependency information.
> > > >
> > > > 5. All source files have license headers where appropriate.
> > > > I believe rat checks this.
> > > >
> > > > 6. The provenance of all source files is clear (ASF or software
> > grants).
> > > > We took care of SGA and CLA. Right?
> > > > So, we just need to get approval from IPMC members by voting.
> > > >
> > > > 7. Dependency licenses are ok. We cannot have the following
> dependency
> > > > license.
> > > >
> > > > - Binary Code License (BCL)
> > > > - Special exceptions to the GNU GPL (e.g. GNU Classpath)
> > > > - GNU GPL 1, 2, 3 <
> > http://www.opensource.org/licenses/gpl-license.php
> > > >
> > > > - GNU LGPL 2, 2.1, 3
> > > > <http://www.opensource.org/licenses/lgpl-license.php>
> > > > - Affero GPL 3 <http://www.opensource.org/licenses/agpl-v3.html>
> > > > - NPL 1.0/NPL 1.1
> > > > - QPL <http://doc.trolltech.com/3.0/license.html>
> > > > - Sleepycat License <
> > http://www.opensource.org/licenses/sleepycat.php
> > > >
> > > > - Microsoft Limited Public License
> > > > <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
> > > > - Code Project Open License (CPOL)
> > > >
> > > > 8. Release consists of source code only, no binaries.
> > > > We should not have any dll, so, jar.
> > > >
> > > > * Some like to add notices (such as RELEASE_NOTES, READMEand
> CHANGES)
> > > for
> > > > each release.
> > > > CHANGES lists fixed JIRA issues.
> > > >
> > > > After addressing the above, I will create a source tarball
> (*.tar.gz),
> > > > signature (*.asc), checksum (*.sha,*.md5) and post them in my apache
> > > > account, which is a typical way of posting RCs.
> > > >
> > > > I will send out the SHA1 checksum of the archive,
> > > > the commit id to be voted upon, the tag to be voted upon, the list of
> > > fixed
> > > > issues, a link to the keys to verify the signature of the release
> > > artifact.
> > > >
> > > > Once the incubator PMC approves the release, we upload the release to
> > the
> > > > incubator distribution directory, podling distribution directory, and
> > our
> > > > own web site.
> > > >
> > > > Let me know if you have any questions.
> > > >
> > > > -Gon
> > > >
> > > >
> > > > On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <bg...@gmail.com>
> > > wrote:
> > > >>
> > > >> Hi,
> > > >>
> > > >> The release management is a bit more complicated than I thought.
> > > >> There're things I am figuring out since this is the first one.
> > > >> Hopefully I will send out an update today. Stay tuned.
> > > >>
> > > >> -Gon
> > > >>
> > > >> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <ma...@weimo.de>
> > > wrote:
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> any updates on the schedule or task list for the rest of us to get
> > this
> > > >>> done?
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Markus
> > > >>>
> > > >>> -----Original Message-----
> > > >>> From: "Byung-Gon Chun" <bg...@gmail.com>
> > > >>> Sent: 12/5/2014 16:06
> > > >>> To: "dev@reef.incubator.apache.org" <dev@reef.incubator.apache.org
> >
> > > >>> Subject: Tentative schedule of the first release: week of Dec. 15
> > > >>>
> > > >>> Hi all,
> > > >>>
> > > >>> This is a quick update on the first release.
> > > >>> The first release of Apache REEF will be done in the week of Dec.
> 15.
> > > >>> I will give you more details about the release process in Apache
> > later.
> > > >>>
> > > >>> Meanwhile, if you have features you want to include in the first
> > > release,
> > > >>> please do so before Dec. 15.
> > > >>>
> > > >>> Cheers,
> > > >>> Gon
> > > >>>
> > > >>> --
> > > >>> Byung-Gon Chun
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Byung-Gon Chun
> > > >>
> > > >
> > > >
> > >
> >
> >
> > --
> > Byung-Gon Chun
> >
>
--
Byung-Gon Chun
Re: Tentative schedule of the first release: week of Dec. 15
Posted by Brian Cho <ch...@gmail.com>.
On Thu, Dec 18, 2014 at 10:09 AM, Byung-Gon Chun <bg...@gmail.com> wrote:
>
> Thanks for the input.
>
> Sure. I will publish the git hash of the version after creating a release
> branch (following apache incubation naming).
>
> I will update basic files on the master branch.
>
> Brian will do the removal of the reef-bridge in the release branch.
> Brian, please check the naming convention of the release branch in Apache.
>
>
I've added the release branch (branch-0.10.0-incubating) and pushed the
reef-bridge removal commit. I would have liked to issue a pull request for
this commit, but it seems github mirroring for a new branch only kicks in
once there is a first commit. So, I didn't have a branch on github to issue
the pull request for at the time of the commit. In future, we may want to
make a trivial commit after creating a branch (although these hacks always
make me cringe).
Thanks,
Brian
> Thanks.
> -Gon
>
>
> On Thu, Dec 18, 2014 at 2:32 AM, Markus Weimer <ma...@weimo.de> wrote:
> >
> > Hi,
> >
> > I'd like to make sure that we have the released version in git as well.
> > That means that we should publish the git hash of the version that was
> > used to make the tarball. That way, we can always go back and backport
> > stuff as it becomes needed.
> >
> > For that, can we create a release branch? E.g. "release-0.10" or such?
> > However, some of the work below (updating LICENSE, DISCLAIMER, NOTICES,
> > add KEYS, ...) can be done immediately against the master branch. I
> > believe the license headers are alright, given that apache-rat passes.
> >
> > Some things will be subtractive and should only be done in the branch.
> > First and foremost, that is the removal of reef-bridge both from git and
> > the master pom. That code won't work in open source until REEF-51 is
> > resolved anyhow. I'm not sure what else will have to be removed.
> >
> > Markus
> >
> >
> >
> > On 2014-12-17 08:01, Byung-Gon Chun wrote:
> > > Hi,
> > >
> > > Warning. This email is long.
> > >
> > > Release voting is done in two phases.
> > > First, the PPMC votes on a release once it's ready by the release
> manager
> > > (me this time). If that passes, we call a vote on the release;
> incubator
> > > PMC members vote.
> > >
> > > PPMC should check the following for the release.
> > > 1. checksums and PGP signatures are valid
> > > The release candidate is signed with PGP key ....., which is included
> > > in the repository's KEYS file: <url>
> > >
> > > checksums: sha1 and/or md5 checksums
> > >
> > > 2. Build is successful including automated tests
> > >
> > > 3. DISCLAIMER is correct. filenames include "incubating".
> > > The file name should be 0.10.0-incubating.
> > >
> > > We have to add DISCLAIMER.
> > >
> > > Podling web sites MUST include a clear disclaimer on their website and
> in
> > > all documentation (including releases) stating that they are in
> > incubation.
> > > Podlings SHOULD use the following text for all disclaimers (replace the
> > > underlined phrases as appropriate):
> > >
> > > Apache *Podling-Name* is an effort undergoing incubation at The Apache
> > > Software Foundation (ASF), sponsored by the *name of Apache TLP
> sponsor*.
> > > Incubation is required of all newly accepted projects until a further
> > > review indicates that the infrastructure, communications, and decision
> > > making process have stabilized in a manner consistent with other
> > successful
> > > ASF projects. While incubation status is not necessarily a reflection
> of
> > > the completeness or stability of the code, it does indicate that the
> > > project has yet to be fully endorsed by the ASF.
> > >
> > > Podlings wishing to use a different disclaimer message MUST have the
> > > disclaimer approved by the Incubator PMC prior to use.
> > >
> > > For releases, the text SHOULD be included in a separate DISCLAIMER file
> > > stored alongside the NOTICE and LICENSE files.
> > >
> > >
> > > 4. Top-level LICENSE and NOTICE are correct for each distribution.
> > >
> > > I believe we don't have any bundled dependencies. In that case, we can
> go
> > > with an easy path.
> > >
> > > LICENSE should contain the text of the ALv2
> > > <http://apache.org/licenses/LICENSE-2.0.txt> -- no more, no less.
> > >
> > > NOTICE should contain only the following text, adapted with the
> product's
> > > name and copyright dates.
> > >
> > > Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache Software
> Foundation
> > > This product includes software developed atThe Apache Software
> > > Foundation (http://www.apache.org/).
> > >
> > > If not, we should add bundled dependency information.
> > >
> > > 5. All source files have license headers where appropriate.
> > > I believe rat checks this.
> > >
> > > 6. The provenance of all source files is clear (ASF or software
> grants).
> > > We took care of SGA and CLA. Right?
> > > So, we just need to get approval from IPMC members by voting.
> > >
> > > 7. Dependency licenses are ok. We cannot have the following dependency
> > > license.
> > >
> > > - Binary Code License (BCL)
> > > - Special exceptions to the GNU GPL (e.g. GNU Classpath)
> > > - GNU GPL 1, 2, 3 <
> http://www.opensource.org/licenses/gpl-license.php
> > >
> > > - GNU LGPL 2, 2.1, 3
> > > <http://www.opensource.org/licenses/lgpl-license.php>
> > > - Affero GPL 3 <http://www.opensource.org/licenses/agpl-v3.html>
> > > - NPL 1.0/NPL 1.1
> > > - QPL <http://doc.trolltech.com/3.0/license.html>
> > > - Sleepycat License <
> http://www.opensource.org/licenses/sleepycat.php
> > >
> > > - Microsoft Limited Public License
> > > <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
> > > - Code Project Open License (CPOL)
> > >
> > > 8. Release consists of source code only, no binaries.
> > > We should not have any dll, so, jar.
> > >
> > > * Some like to add notices (such as RELEASE_NOTES, READMEand CHANGES)
> > for
> > > each release.
> > > CHANGES lists fixed JIRA issues.
> > >
> > > After addressing the above, I will create a source tarball (*.tar.gz),
> > > signature (*.asc), checksum (*.sha,*.md5) and post them in my apache
> > > account, which is a typical way of posting RCs.
> > >
> > > I will send out the SHA1 checksum of the archive,
> > > the commit id to be voted upon, the tag to be voted upon, the list of
> > fixed
> > > issues, a link to the keys to verify the signature of the release
> > artifact.
> > >
> > > Once the incubator PMC approves the release, we upload the release to
> the
> > > incubator distribution directory, podling distribution directory, and
> our
> > > own web site.
> > >
> > > Let me know if you have any questions.
> > >
> > > -Gon
> > >
> > >
> > > On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <bg...@gmail.com>
> > wrote:
> > >>
> > >> Hi,
> > >>
> > >> The release management is a bit more complicated than I thought.
> > >> There're things I am figuring out since this is the first one.
> > >> Hopefully I will send out an update today. Stay tuned.
> > >>
> > >> -Gon
> > >>
> > >> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <ma...@weimo.de>
> > wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> any updates on the schedule or task list for the rest of us to get
> this
> > >>> done?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Markus
> > >>>
> > >>> -----Original Message-----
> > >>> From: "Byung-Gon Chun" <bg...@gmail.com>
> > >>> Sent: 12/5/2014 16:06
> > >>> To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
> > >>> Subject: Tentative schedule of the first release: week of Dec. 15
> > >>>
> > >>> Hi all,
> > >>>
> > >>> This is a quick update on the first release.
> > >>> The first release of Apache REEF will be done in the week of Dec. 15.
> > >>> I will give you more details about the release process in Apache
> later.
> > >>>
> > >>> Meanwhile, if you have features you want to include in the first
> > release,
> > >>> please do so before Dec. 15.
> > >>>
> > >>> Cheers,
> > >>> Gon
> > >>>
> > >>> --
> > >>> Byung-Gon Chun
> > >>>
> > >>
> > >>
> > >> --
> > >> Byung-Gon Chun
> > >>
> > >
> > >
> >
>
>
> --
> Byung-Gon Chun
>
Re: Tentative schedule of the first release: week of Dec. 15
Posted by Byung-Gon Chun <bg...@gmail.com>.
Thanks for the input.
Sure. I will publish the git hash of the version after creating a release
branch (following apache incubation naming).
I will update basic files on the master branch.
Brian will do the removal of the reef-bridge in the release branch.
Brian, please check the naming convention of the release branch in Apache.
Thanks.
-Gon
On Thu, Dec 18, 2014 at 2:32 AM, Markus Weimer <ma...@weimo.de> wrote:
>
> Hi,
>
> I'd like to make sure that we have the released version in git as well.
> That means that we should publish the git hash of the version that was
> used to make the tarball. That way, we can always go back and backport
> stuff as it becomes needed.
>
> For that, can we create a release branch? E.g. "release-0.10" or such?
> However, some of the work below (updating LICENSE, DISCLAIMER, NOTICES,
> add KEYS, ...) can be done immediately against the master branch. I
> believe the license headers are alright, given that apache-rat passes.
>
> Some things will be subtractive and should only be done in the branch.
> First and foremost, that is the removal of reef-bridge both from git and
> the master pom. That code won't work in open source until REEF-51 is
> resolved anyhow. I'm not sure what else will have to be removed.
>
> Markus
>
>
>
> On 2014-12-17 08:01, Byung-Gon Chun wrote:
> > Hi,
> >
> > Warning. This email is long.
> >
> > Release voting is done in two phases.
> > First, the PPMC votes on a release once it's ready by the release manager
> > (me this time). If that passes, we call a vote on the release; incubator
> > PMC members vote.
> >
> > PPMC should check the following for the release.
> > 1. checksums and PGP signatures are valid
> > The release candidate is signed with PGP key ....., which is included
> > in the repository's KEYS file: <url>
> >
> > checksums: sha1 and/or md5 checksums
> >
> > 2. Build is successful including automated tests
> >
> > 3. DISCLAIMER is correct. filenames include "incubating".
> > The file name should be 0.10.0-incubating.
> >
> > We have to add DISCLAIMER.
> >
> > Podling web sites MUST include a clear disclaimer on their website and in
> > all documentation (including releases) stating that they are in
> incubation.
> > Podlings SHOULD use the following text for all disclaimers (replace the
> > underlined phrases as appropriate):
> >
> > Apache *Podling-Name* is an effort undergoing incubation at The Apache
> > Software Foundation (ASF), sponsored by the *name of Apache TLP sponsor*.
> > Incubation is required of all newly accepted projects until a further
> > review indicates that the infrastructure, communications, and decision
> > making process have stabilized in a manner consistent with other
> successful
> > ASF projects. While incubation status is not necessarily a reflection of
> > the completeness or stability of the code, it does indicate that the
> > project has yet to be fully endorsed by the ASF.
> >
> > Podlings wishing to use a different disclaimer message MUST have the
> > disclaimer approved by the Incubator PMC prior to use.
> >
> > For releases, the text SHOULD be included in a separate DISCLAIMER file
> > stored alongside the NOTICE and LICENSE files.
> >
> >
> > 4. Top-level LICENSE and NOTICE are correct for each distribution.
> >
> > I believe we don't have any bundled dependencies. In that case, we can go
> > with an easy path.
> >
> > LICENSE should contain the text of the ALv2
> > <http://apache.org/licenses/LICENSE-2.0.txt> -- no more, no less.
> >
> > NOTICE should contain only the following text, adapted with the product's
> > name and copyright dates.
> >
> > Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache Software Foundation
> > This product includes software developed atThe Apache Software
> > Foundation (http://www.apache.org/).
> >
> > If not, we should add bundled dependency information.
> >
> > 5. All source files have license headers where appropriate.
> > I believe rat checks this.
> >
> > 6. The provenance of all source files is clear (ASF or software grants).
> > We took care of SGA and CLA. Right?
> > So, we just need to get approval from IPMC members by voting.
> >
> > 7. Dependency licenses are ok. We cannot have the following dependency
> > license.
> >
> > - Binary Code License (BCL)
> > - Special exceptions to the GNU GPL (e.g. GNU Classpath)
> > - GNU GPL 1, 2, 3 <http://www.opensource.org/licenses/gpl-license.php
> >
> > - GNU LGPL 2, 2.1, 3
> > <http://www.opensource.org/licenses/lgpl-license.php>
> > - Affero GPL 3 <http://www.opensource.org/licenses/agpl-v3.html>
> > - NPL 1.0/NPL 1.1
> > - QPL <http://doc.trolltech.com/3.0/license.html>
> > - Sleepycat License <http://www.opensource.org/licenses/sleepycat.php
> >
> > - Microsoft Limited Public License
> > <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
> > - Code Project Open License (CPOL)
> >
> > 8. Release consists of source code only, no binaries.
> > We should not have any dll, so, jar.
> >
> > * Some like to add notices (such as RELEASE_NOTES, READMEand CHANGES)
> for
> > each release.
> > CHANGES lists fixed JIRA issues.
> >
> > After addressing the above, I will create a source tarball (*.tar.gz),
> > signature (*.asc), checksum (*.sha,*.md5) and post them in my apache
> > account, which is a typical way of posting RCs.
> >
> > I will send out the SHA1 checksum of the archive,
> > the commit id to be voted upon, the tag to be voted upon, the list of
> fixed
> > issues, a link to the keys to verify the signature of the release
> artifact.
> >
> > Once the incubator PMC approves the release, we upload the release to the
> > incubator distribution directory, podling distribution directory, and our
> > own web site.
> >
> > Let me know if you have any questions.
> >
> > -Gon
> >
> >
> > On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <bg...@gmail.com>
> wrote:
> >>
> >> Hi,
> >>
> >> The release management is a bit more complicated than I thought.
> >> There're things I am figuring out since this is the first one.
> >> Hopefully I will send out an update today. Stay tuned.
> >>
> >> -Gon
> >>
> >> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <ma...@weimo.de>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> any updates on the schedule or task list for the rest of us to get this
> >>> done?
> >>>
> >>> Thanks,
> >>>
> >>> Markus
> >>>
> >>> -----Original Message-----
> >>> From: "Byung-Gon Chun" <bg...@gmail.com>
> >>> Sent: 12/5/2014 16:06
> >>> To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
> >>> Subject: Tentative schedule of the first release: week of Dec. 15
> >>>
> >>> Hi all,
> >>>
> >>> This is a quick update on the first release.
> >>> The first release of Apache REEF will be done in the week of Dec. 15.
> >>> I will give you more details about the release process in Apache later.
> >>>
> >>> Meanwhile, if you have features you want to include in the first
> release,
> >>> please do so before Dec. 15.
> >>>
> >>> Cheers,
> >>> Gon
> >>>
> >>> --
> >>> Byung-Gon Chun
> >>>
> >>
> >>
> >> --
> >> Byung-Gon Chun
> >>
> >
> >
>
--
Byung-Gon Chun
Re: Tentative schedule of the first release: week of Dec. 15
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
I'd like to make sure that we have the released version in git as well.
That means that we should publish the git hash of the version that was
used to make the tarball. That way, we can always go back and backport
stuff as it becomes needed.
For that, can we create a release branch? E.g. "release-0.10" or such?
However, some of the work below (updating LICENSE, DISCLAIMER, NOTICES,
add KEYS, ...) can be done immediately against the master branch. I
believe the license headers are alright, given that apache-rat passes.
Some things will be subtractive and should only be done in the branch.
First and foremost, that is the removal of reef-bridge both from git and
the master pom. That code won't work in open source until REEF-51 is
resolved anyhow. I'm not sure what else will have to be removed.
Markus
On 2014-12-17 08:01, Byung-Gon Chun wrote:
> Hi,
>
> Warning. This email is long.
>
> Release voting is done in two phases.
> First, the PPMC votes on a release once it's ready by the release manager
> (me this time). If that passes, we call a vote on the release; incubator
> PMC members vote.
>
> PPMC should check the following for the release.
> 1. checksums and PGP signatures are valid
> The release candidate is signed with PGP key ....., which is included
> in the repository's KEYS file: <url>
>
> checksums: sha1 and/or md5 checksums
>
> 2. Build is successful including automated tests
>
> 3. DISCLAIMER is correct. filenames include "incubating".
> The file name should be 0.10.0-incubating.
>
> We have to add DISCLAIMER.
>
> Podling web sites MUST include a clear disclaimer on their website and in
> all documentation (including releases) stating that they are in incubation.
> Podlings SHOULD use the following text for all disclaimers (replace the
> underlined phrases as appropriate):
>
> Apache *Podling-Name* is an effort undergoing incubation at The Apache
> Software Foundation (ASF), sponsored by the *name of Apache TLP sponsor*.
> Incubation is required of all newly accepted projects until a further
> review indicates that the infrastructure, communications, and decision
> making process have stabilized in a manner consistent with other successful
> ASF projects. While incubation status is not necessarily a reflection of
> the completeness or stability of the code, it does indicate that the
> project has yet to be fully endorsed by the ASF.
>
> Podlings wishing to use a different disclaimer message MUST have the
> disclaimer approved by the Incubator PMC prior to use.
>
> For releases, the text SHOULD be included in a separate DISCLAIMER file
> stored alongside the NOTICE and LICENSE files.
>
>
> 4. Top-level LICENSE and NOTICE are correct for each distribution.
>
> I believe we don't have any bundled dependencies. In that case, we can go
> with an easy path.
>
> LICENSE should contain the text of the ALv2
> <http://apache.org/licenses/LICENSE-2.0.txt> -- no more, no less.
>
> NOTICE should contain only the following text, adapted with the product's
> name and copyright dates.
>
> Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache Software Foundation
> This product includes software developed atThe Apache Software
> Foundation (http://www.apache.org/).
>
> If not, we should add bundled dependency information.
>
> 5. All source files have license headers where appropriate.
> I believe rat checks this.
>
> 6. The provenance of all source files is clear (ASF or software grants).
> We took care of SGA and CLA. Right?
> So, we just need to get approval from IPMC members by voting.
>
> 7. Dependency licenses are ok. We cannot have the following dependency
> license.
>
> - Binary Code License (BCL)
> - Special exceptions to the GNU GPL (e.g. GNU Classpath)
> - GNU GPL 1, 2, 3 <http://www.opensource.org/licenses/gpl-license.php>
> - GNU LGPL 2, 2.1, 3
> <http://www.opensource.org/licenses/lgpl-license.php>
> - Affero GPL 3 <http://www.opensource.org/licenses/agpl-v3.html>
> - NPL 1.0/NPL 1.1
> - QPL <http://doc.trolltech.com/3.0/license.html>
> - Sleepycat License <http://www.opensource.org/licenses/sleepycat.php>
> - Microsoft Limited Public License
> <http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
> - Code Project Open License (CPOL)
>
> 8. Release consists of source code only, no binaries.
> We should not have any dll, so, jar.
>
> * Some like to add notices (such as RELEASE_NOTES, READMEand CHANGES) for
> each release.
> CHANGES lists fixed JIRA issues.
>
> After addressing the above, I will create a source tarball (*.tar.gz),
> signature (*.asc), checksum (*.sha,*.md5) and post them in my apache
> account, which is a typical way of posting RCs.
>
> I will send out the SHA1 checksum of the archive,
> the commit id to be voted upon, the tag to be voted upon, the list of fixed
> issues, a link to the keys to verify the signature of the release artifact.
>
> Once the incubator PMC approves the release, we upload the release to the
> incubator distribution directory, podling distribution directory, and our
> own web site.
>
> Let me know if you have any questions.
>
> -Gon
>
>
> On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <bg...@gmail.com> wrote:
>>
>> Hi,
>>
>> The release management is a bit more complicated than I thought.
>> There're things I am figuring out since this is the first one.
>> Hopefully I will send out an update today. Stay tuned.
>>
>> -Gon
>>
>> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <ma...@weimo.de> wrote:
>>>
>>> Hi,
>>>
>>> any updates on the schedule or task list for the rest of us to get this
>>> done?
>>>
>>> Thanks,
>>>
>>> Markus
>>>
>>> -----Original Message-----
>>> From: "Byung-Gon Chun" <bg...@gmail.com>
>>> Sent: 12/5/2014 16:06
>>> To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
>>> Subject: Tentative schedule of the first release: week of Dec. 15
>>>
>>> Hi all,
>>>
>>> This is a quick update on the first release.
>>> The first release of Apache REEF will be done in the week of Dec. 15.
>>> I will give you more details about the release process in Apache later.
>>>
>>> Meanwhile, if you have features you want to include in the first release,
>>> please do so before Dec. 15.
>>>
>>> Cheers,
>>> Gon
>>>
>>> --
>>> Byung-Gon Chun
>>>
>>
>>
>> --
>> Byung-Gon Chun
>>
>
>
Re: Tentative schedule of the first release: week of Dec. 15
Posted by Byung-Gon Chun <bg...@gmail.com>.
Hi,
Warning. This email is long.
Release voting is done in two phases.
First, the PPMC votes on a release once it's ready by the release manager
(me this time). If that passes, we call a vote on the release; incubator
PMC members vote.
PPMC should check the following for the release.
1. checksums and PGP signatures are valid
The release candidate is signed with PGP key ....., which is included
in the repository's KEYS file: <url>
checksums: sha1 and/or md5 checksums
2. Build is successful including automated tests
3. DISCLAIMER is correct. filenames include "incubating".
The file name should be 0.10.0-incubating.
We have to add DISCLAIMER.
Podling web sites MUST include a clear disclaimer on their website and in
all documentation (including releases) stating that they are in incubation.
Podlings SHOULD use the following text for all disclaimers (replace the
underlined phrases as appropriate):
Apache *Podling-Name* is an effort undergoing incubation at The Apache
Software Foundation (ASF), sponsored by the *name of Apache TLP sponsor*.
Incubation is required of all newly accepted projects until a further
review indicates that the infrastructure, communications, and decision
making process have stabilized in a manner consistent with other successful
ASF projects. While incubation status is not necessarily a reflection of
the completeness or stability of the code, it does indicate that the
project has yet to be fully endorsed by the ASF.
Podlings wishing to use a different disclaimer message MUST have the
disclaimer approved by the Incubator PMC prior to use.
For releases, the text SHOULD be included in a separate DISCLAIMER file
stored alongside the NOTICE and LICENSE files.
4. Top-level LICENSE and NOTICE are correct for each distribution.
I believe we don't have any bundled dependencies. In that case, we can go
with an easy path.
LICENSE should contain the text of the ALv2
<http://apache.org/licenses/LICENSE-2.0.txt> -- no more, no less.
NOTICE should contain only the following text, adapted with the product's
name and copyright dates.
Apache [PRODUCT_NAME]Copyright [XXXX-20XX] The Apache Software Foundation
This product includes software developed atThe Apache Software
Foundation (http://www.apache.org/).
If not, we should add bundled dependency information.
5. All source files have license headers where appropriate.
I believe rat checks this.
6. The provenance of all source files is clear (ASF or software grants).
We took care of SGA and CLA. Right?
So, we just need to get approval from IPMC members by voting.
7. Dependency licenses are ok. We cannot have the following dependency
license.
- Binary Code License (BCL)
- Special exceptions to the GNU GPL (e.g. GNU Classpath)
- GNU GPL 1, 2, 3 <http://www.opensource.org/licenses/gpl-license.php>
- GNU LGPL 2, 2.1, 3
<http://www.opensource.org/licenses/lgpl-license.php>
- Affero GPL 3 <http://www.opensource.org/licenses/agpl-v3.html>
- NPL 1.0/NPL 1.1
- QPL <http://doc.trolltech.com/3.0/license.html>
- Sleepycat License <http://www.opensource.org/licenses/sleepycat.php>
- Microsoft Limited Public License
<http://msdn.microsoft.com/en-us/cc300389.aspx#MLPL>
- Code Project Open License (CPOL)
8. Release consists of source code only, no binaries.
We should not have any dll, so, jar.
* Some like to add notices (such as RELEASE_NOTES, READMEand CHANGES) for
each release.
CHANGES lists fixed JIRA issues.
After addressing the above, I will create a source tarball (*.tar.gz),
signature (*.asc), checksum (*.sha,*.md5) and post them in my apache
account, which is a typical way of posting RCs.
I will send out the SHA1 checksum of the archive,
the commit id to be voted upon, the tag to be voted upon, the list of fixed
issues, a link to the keys to verify the signature of the release artifact.
Once the incubator PMC approves the release, we upload the release to the
incubator distribution directory, podling distribution directory, and our
own web site.
Let me know if you have any questions.
-Gon
On Wed, Dec 17, 2014 at 7:49 AM, Byung-Gon Chun <bg...@gmail.com> wrote:
>
> Hi,
>
> The release management is a bit more complicated than I thought.
> There're things I am figuring out since this is the first one.
> Hopefully I will send out an update today. Stay tuned.
>
> -Gon
>
> On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <ma...@weimo.de> wrote:
>>
>> Hi,
>>
>> any updates on the schedule or task list for the rest of us to get this
>> done?
>>
>> Thanks,
>>
>> Markus
>>
>> -----Original Message-----
>> From: "Byung-Gon Chun" <bg...@gmail.com>
>> Sent: 12/5/2014 16:06
>> To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
>> Subject: Tentative schedule of the first release: week of Dec. 15
>>
>> Hi all,
>>
>> This is a quick update on the first release.
>> The first release of Apache REEF will be done in the week of Dec. 15.
>> I will give you more details about the release process in Apache later.
>>
>> Meanwhile, if you have features you want to include in the first release,
>> please do so before Dec. 15.
>>
>> Cheers,
>> Gon
>>
>> --
>> Byung-Gon Chun
>>
>
>
> --
> Byung-Gon Chun
>
--
Byung-Gon Chun
Re: Tentative schedule of the first release: week of Dec. 15
Posted by Byung-Gon Chun <bg...@gmail.com>.
Hi,
The release management is a bit more complicated than I thought.
There're things I am figuring out since this is the first one.
Hopefully I will send out an update today. Stay tuned.
-Gon
On Tue, Dec 16, 2014 at 10:19 AM, Markus Weimer <ma...@weimo.de> wrote:
>
> Hi,
>
> any updates on the schedule or task list for the rest of us to get this
> done?
>
> Thanks,
>
> Markus
>
> -----Original Message-----
> From: "Byung-Gon Chun" <bg...@gmail.com>
> Sent: 12/5/2014 16:06
> To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
> Subject: Tentative schedule of the first release: week of Dec. 15
>
> Hi all,
>
> This is a quick update on the first release.
> The first release of Apache REEF will be done in the week of Dec. 15.
> I will give you more details about the release process in Apache later.
>
> Meanwhile, if you have features you want to include in the first release,
> please do so before Dec. 15.
>
> Cheers,
> Gon
>
> --
> Byung-Gon Chun
>
--
Byung-Gon Chun
RE: Tentative schedule of the first release: week of Dec. 15
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
any updates on the schedule or task list for the rest of us to get this done?
Thanks,
Markus
-----Original Message-----
From: "Byung-Gon Chun" <bg...@gmail.com>
Sent: 12/5/2014 16:06
To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
Subject: Tentative schedule of the first release: week of Dec. 15
Hi all,
This is a quick update on the first release.
The first release of Apache REEF will be done in the week of Dec. 15.
I will give you more details about the release process in Apache later.
Meanwhile, if you have features you want to include in the first release,
please do so before Dec. 15.
Cheers,
Gon
--
Byung-Gon Chun