You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Saikat Kanjilal <sx...@gmail.com> on 2017/06/23 23:14:39 UTC
Does reef build on the mac
All,
I'm trying to spinup the new reef-runtime-spark project and I had some
pre-cursory questions:
1) It seems that we dont have a top level build file on the java side of
the world that builds all the maven projects, it would be really helpful to
have something like this that figures out all the dependencies and creates
an uber jar assembly containing all the other jar files
2) When I go to build some of the java projects individually , for example
reef-common I get errors like this:
Could not resolve dependencies for project
org.apache.reef:reef-common:jar:0.16.0-SNAPSHOT: Failed to collect
dependencies at org.apache.reef:reef-annotations:jar:0.16.0-SNAPSHOT:
Failed to read artifact descriptor for
org.apache.reef:reef-annotations:jar:0.16.0-SNAPSHOT: Failure to find
org.apache.reef:reef-project:pom:0.16.0-SNAPSHOT in
http://repository.apache.org/snapshots was cached in the local repository,
resolution will not be reattempted until the update interval of
apache.snapshots has elapsed or updates are forced -> [Help 1]
Now I recursively went and tried to build the annotations and that seems to
build successfully but I cant get reef-common to recognize that I have the
annotations jar file stored in my local maven repo
We really need to be more specific in our build instructions on the java
side as to what to build first and this is why I feel point number 1 is
needed.
Thoughts, I've scoured the following docs:
https://github.com/apache/reef/blob/master/lang/java/BUILD.md as well as
https://cwiki.apache.org/confluence/display/REEF/Linux and have found
neither to be particularly helpful in deciphering these build errors.
Any insight is much appreciated.
Re: Does reef build on the mac
Posted by Markus Weimer <ma...@weimo.de>.
Also, there are ways to extract the dependency graph from POMs:
https://stackoverflow.com/questions/4084669/how-to-generate-a-graph-of-the-dependency-between-all-modules-of-a-maven-project
Markus
On Wed, Jun 28, 2017 at 12:50 AM, Mariia Mykhailova
<ma...@microsoft.com.invalid> wrote:
> pom.xml in the root folder of REEF project also contains some meta-information which describes the project as a whole and not just Java part of it (info like licensing, urls, mailing lists etc.)
>
>
> I wouldn't get into the trouble of moving it (and potentially splitting it into Java-only part and project-general part) just for the sake of keeping our inner organizational nazi happy :-)
>
>
> +1 for not maintaining a separate dependencies graph; any documentation like this will tend to drift away from the code it describes over time as new dependencies are added.
>
>
> -Mariia
>
> ________________________________
> From: Markus Weimer <ma...@weimo.de>
> Sent: Tuesday, June 27, 2017 6:48:16 PM
> To: REEF Developers Mailinglist
> Subject: Re: Does reef build on the mac
>
> On Mon, Jun 26, 2017 at 3:35 PM, Saikat Kanjilal <sx...@gmail.com> wrote:
>> Not to sound like an organizational nazi :) but is it possible to move the
>> root level pom file into the lang\java directory , it seems like the .net
>> side is self contained and we should use that consistency for the build
>> mechanisms on the java side as well.
>
> That seems feasible. However, we use the pom in the root directory to
> also trigger a scan of the .NET code for compliance with our license
> header requirement. If we have a replacement for that in msbuild, we
> could move the pom.
>
>> Also there are times when one
>> wishes to only build some of the components so it'd also be good to
>> document the build dependency hierarchy in the build.md as well, again for
>> a beginner level person coming into the project it helps to understand this
>> dependency while debugging/building.
>
> That is less likely. The build dependencies are all encoded in the
> POMs, and the maintenance of an additional verbatim description in a
> text file is unlikely to succeed. Also, build systems are generally
> doing an OK job at identifying which parts of a project need to be
> rebuilt after changes.
>
> Markus
Re: Does reef build on the mac
Posted by Mariia Mykhailova <ma...@microsoft.com.INVALID>.
pom.xml in the root folder of REEF project also contains some meta-information which describes the project as a whole and not just Java part of it (info like licensing, urls, mailing lists etc.)
I wouldn't get into the trouble of moving it (and potentially splitting it into Java-only part and project-general part) just for the sake of keeping our inner organizational nazi happy :-)
+1 for not maintaining a separate dependencies graph; any documentation like this will tend to drift away from the code it describes over time as new dependencies are added.
-Mariia
________________________________
From: Markus Weimer <ma...@weimo.de>
Sent: Tuesday, June 27, 2017 6:48:16 PM
To: REEF Developers Mailinglist
Subject: Re: Does reef build on the mac
On Mon, Jun 26, 2017 at 3:35 PM, Saikat Kanjilal <sx...@gmail.com> wrote:
> Not to sound like an organizational nazi :) but is it possible to move the
> root level pom file into the lang\java directory , it seems like the .net
> side is self contained and we should use that consistency for the build
> mechanisms on the java side as well.
That seems feasible. However, we use the pom in the root directory to
also trigger a scan of the .NET code for compliance with our license
header requirement. If we have a replacement for that in msbuild, we
could move the pom.
> Also there are times when one
> wishes to only build some of the components so it'd also be good to
> document the build dependency hierarchy in the build.md as well, again for
> a beginner level person coming into the project it helps to understand this
> dependency while debugging/building.
That is less likely. The build dependencies are all encoded in the
POMs, and the maintenance of an additional verbatim description in a
text file is unlikely to succeed. Also, build systems are generally
doing an OK job at identifying which parts of a project need to be
rebuilt after changes.
Markus
Re: Does reef build on the mac
Posted by Markus Weimer <ma...@weimo.de>.
On Mon, Jun 26, 2017 at 3:35 PM, Saikat Kanjilal <sx...@gmail.com> wrote:
> Not to sound like an organizational nazi :) but is it possible to move the
> root level pom file into the lang\java directory , it seems like the .net
> side is self contained and we should use that consistency for the build
> mechanisms on the java side as well.
That seems feasible. However, we use the pom in the root directory to
also trigger a scan of the .NET code for compliance with our license
header requirement. If we have a replacement for that in msbuild, we
could move the pom.
> Also there are times when one
> wishes to only build some of the components so it'd also be good to
> document the build dependency hierarchy in the build.md as well, again for
> a beginner level person coming into the project it helps to understand this
> dependency while debugging/building.
That is less likely. The build dependencies are all encoded in the
POMs, and the maintenance of an additional verbatim description in a
text file is unlikely to succeed. Also, build systems are generally
doing an OK job at identifying which parts of a project need to be
rebuilt after changes.
Markus
Re: Does reef build on the mac
Posted by Saikat Kanjilal <sx...@gmail.com>.
Not to sound like an organizational nazi :) but is it possible to move the
root level pom file into the lang\java directory , it seems like the .net
side is self contained and we should use that consistency for the build
mechanisms on the java side as well. Also there are times when one
wishes to only build some of the components so it'd also be good to
document the build dependency hierarchy in the build.md as well, again for
a beginner level person coming into the project it helps to understand this
dependency while debugging/building.
On Mon, Jun 26, 2017 at 1:21 PM, Mariia Mykhailova <
mamykhai@microsoft.com.invalid> wrote:
> Indeed we've missed to mention that these commands should be executed from
> the root REEF folder.
> I've sent PR to fix this: https://github.com/apache/reef/pull/1318
>
> -Mariia
>
> -----Original Message-----
> From: Markus Weimer [mailto:markus@weimo.de]
> Sent: Saturday, June 24, 2017 9:31 AM
> To: REEF Developers Mailinglist <de...@reef.apache.org>
> Subject: Re: Does reef build on the mac
>
> > 1) It seems that we dont have a top level build file on the java side
> > of the world that builds all the maven projects, it would be really
> > helpful to have something like this that figures out all the
> > dependencies and creates an uber jar assembly containing all the other
> > jar files
>
> We do. the root of the repository has `pom.xml`, which is the main build
> file. Just type `mvn -DskipTests clean install` to compile REEF.
> This is documented in `lang/java/BUILD.md`. What's missing there are
> instructions that you should do this in the root directory of the project.
> Contributions welcome :)
>
> Markus
>
RE: Does reef build on the mac
Posted by Mariia Mykhailova <ma...@microsoft.com.INVALID>.
Indeed we've missed to mention that these commands should be executed from the root REEF folder.
I've sent PR to fix this: https://github.com/apache/reef/pull/1318
-Mariia
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Saturday, June 24, 2017 9:31 AM
To: REEF Developers Mailinglist <de...@reef.apache.org>
Subject: Re: Does reef build on the mac
> 1) It seems that we dont have a top level build file on the java side
> of the world that builds all the maven projects, it would be really
> helpful to have something like this that figures out all the
> dependencies and creates an uber jar assembly containing all the other
> jar files
We do. the root of the repository has `pom.xml`, which is the main build file. Just type `mvn -DskipTests clean install` to compile REEF.
This is documented in `lang/java/BUILD.md`. What's missing there are instructions that you should do this in the root directory of the project. Contributions welcome :)
Markus
Re: Does reef build on the mac
Posted by Markus Weimer <ma...@weimo.de>.
> 1) It seems that we dont have a top level build file on the java side of
> the world that builds all the maven projects, it would be really helpful to
> have something like this that figures out all the dependencies and creates
> an uber jar assembly containing all the other jar files
We do. the root of the repository has `pom.xml`, which is the main
build file. Just type `mvn -DskipTests clean install` to compile REEF.
This is documented in `lang/java/BUILD.md`. What's missing there are
instructions that you should do this in the root directory of the
project. Contributions welcome :)
Markus