You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@edgent.apache.org by "John D. Ament" <jo...@apache.org> on 2017/05/23 01:54:02 UTC
Understanding the snapshot and release process
Hey,
I'm trying to understand the current snapshot and release process for
Edgent, specifically to make it so that we can easily publish JARs into
maven central.
I'll be honest, I've never seen such a complicated gradle build. So I am
in a little over my head.
I'm not sure all of the steps being executed previously, but I'm struggling
with locally signed artifacts (for the convenience binaries). What steps
get executed to make this happen?
I was able to push up edgent snapshots to the ASF's nexus instance, which
you can find here:
https://repository.apache.org/content/repositories/snapshots/org/apache/edgent/
.
This was done by simply populating my ~/.gradle/gradle.properties and
running "./gradlew publish"
John
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
No worries,
As long as you didn't do a "deploy" nothing should be uploaded :-)
Chris
Von meinem Samsung Galaxy Smartphone gesendet.
-------- Ursprüngliche Nachricht --------
Von: Dale LaBossiere <dm...@gmail.com>
Datum: 07.06.17 19:35 (GMT+01:00)
An: dev@edgent.apache.org
Betreff: Re: Understanding the snapshot and release process
install -Papache-release is generating the (arc) signature file :-)
Chris, what needs to be done to get it to also generate the .md5 and .sha files?
Also, did I now just populate the ASF snapshot repo? :-)
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
install -Papache-release is generating the (arc) signature file :-)
Chris, what needs to be done to get it to also generate the .md5 and .sha files?
Also, did I now just populate the ASF snapshot repo? :-)
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
> On Jun 8, 2017, at 10:43 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Eventually your approach with a sub-directory inside the target would be a valid option … would probably have to adjust the maven-clean-plugin too, but that should work in getting all results inside the target directory in parallel.
maven-clean-plugin seemed to work OK. Hmm… would there be a way to get a “clean all”? Maybe that’s what you meant.
mvn clean -P tgt-java7 # cleans all target/java7
mvn clean # cleans all target/java8
That was with pom.xml changes:
<properties>
<edgent.target.kind>java8</edgent.target.kind>
...
</properties>
<profiles>
<profile>
<id>tgt-java7</id>
<activation>
<property>
<name>tgt</name>
<value>java7</value>
</property>
</activation>
<properties>
<edgent.target.kind>java7</edgent.target.kind>
</properties>
...
</profile>
<build>
<directory>target/${edgent.target.kind}</directory>
...
</build>
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Eventually your approach with a sub-directory inside the target would be a valid option … would probably have to adjust the maven-clean-plugin too, but that should work in getting all results inside the target directory in parallel.
Chris
Am 08.06.17, 16:13 schrieb "Dale LaBossiere" <dm...@gmail.com>:
> On Jun 8, 2017, at 4:19 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Regarding java8 and java7 artifacts …
Coincidentally, I started investigating this space yesterday and retrolambda-maven-plugin
and was just composing mail on the question of publishing java8 and java7 to mvn repos :-)
[ fwiw, so far I had defined a Profile (e.g., mvn install -Ptgt=java7) and
changed the “target” build directory to “target/${edgent.target.kind}”
(e.g., target/{java8,java7}). Things were working fine up to that point:
generated class files and module jars. ]
> This could be problematic, if we wanted to publish both.
> ...
> I could offer to generally build java7 versions and to test them with java7 and java8 … I think this topic should be discussed a little more before I start implementing this.
>
> Right now, I would opt for distributing java8 binaries to Maven Central and to offer downloadable java7 binary distributions.
Ugh. Publishing java8 in a mvn repo but java7/android as a binary tgz seems really bad.
It creates two different ways (and needed doc) for users building an Edgent app
and installing Edgent jars onto a “device”.
Could you elaborate on the problems using the maven coordinate “classifier”?
The maven doc [1] has this exactly as use case for it (they use “jdk15” vs “jdk14”).
At a high (ignorant) level I don’t have a problem with the classifier being nil for java8 :-)
If using a coordinate classifier is a no-go, what about (grimace) incorporating it into the
groupId: org.apache.edgent, org.apache.edgent.java7 ?
A bit related, what coordinate component is “incubating” supposed to be incorporated into?
— Dale
[1] https://maven.apache.org/pom.html <https://maven.apache.org/pom.html>
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
sorry no idea about travis and memory
On Jun 8, 2017, at 10:40 AM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...
> Today I had some progress on the java7 side.
...
> problem. The default methods in DirectTestSetup can’t seem to be backported if the interface and the implementing class are not in the same retrolambda execution. Eventually refactoring the tests to work without default methods would be a good idea.
I’ll look into that.
> The problem with classifiers is that maven automatically resolves artifacts. Depending on the scope different artifacts are pulled in. In general, we would be using the “compile” scope. So, if I add a dependency with classifier “java7” then it would pull in that jar in the java7 version, but the transitive dependencies would probably be the java8 ones, so you would have to explicitly define all the transitive dependencies with the classifier. We don’t want that ;-)
Thought I was following it but then not :-)
I may have to try a test using classifier so as to really understand how/why it won’t work.
But if encoding tgt-kind in the versionId is the way to go… (I think it would be nice if it could be avoided for tgt-java8, e.g., just 1.2.0, not 1.2.0.java8)
In travis I don’t think you’ll be able to just/only say
mvn clean deploy # compile/test/deploy java8
mvn clean deploy -Pjava7 # compile/test/deploy java7
That’s because the compile for java7, including the tests, must be done with a java8 JDK (with retrolambda then applied). There’s no java7 JDK in the mix for java7 compile.
But to really test the compiled java7-compatible tests one has to first switch the environment to a java7 JRE and then run them in that environment (they can’t be compiled in a java7 env because they use lambdas).
That’s why the travis / gradle/ant stuff was structured out the way it was:
switch to java8
gradlew release # compiles Edgent into jars for java8,7,android.
# NOTE, tests are not bundled into jars. So there are just the j8 class files
gradlew test # runs the already compiled Edgent tests (with java8)
gradlew test7Compile # recompile the tests-only to be java7 compatible
switch to java7
gradlew test7Run # runs the already java7-compatible compiled Edgent tests (with java7)
Also at a high level I think we don’t want to deploy one config unless all configs are OK.
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Just have a look at the last few builds on Travis that my pull request initiated. You will see it failing with different tests every time I ran the build even if nothing changed (I think in the last commit I simply removed a space in one of the poms).
Was successful …
After moving some tests from Unit-Test to Integration-Test phase:
Failure in module “Apache Edgent: Providers: Development”
- DevelopmentPlumbingTest>PlumbingTest.testParallelBalanced:628
After adding the explice lifecycle mapping plugin config (has no effect on the build)
Failure in module “Apache Edgent: Connectors: WSClient Websocket”
- [ERROR] WebSocketClientGlobalTest>WebSocketClientTest.testReconnect:359->TopologyAbstractTest.completeAndValidate:82->TopologyAbstractTest.completeAndValidate:98 contents:[一, 二, 四]
- [ERROR] WebSocketClientGlobalTest>WebSocketClientTest.testSslReconnect:571->TopologyAbstractTest.completeAndValidate:82->TopologyAbstractTest.completeAndValidate:98 contents:[一, 二, 四]
After removing a simple space character in a pom
Failure in module “Apache Edgent: Providers: Direct”
- [ERROR] DirectPlumbingTest>PlumbingTest.testPressureReliever:194 delay:TAI:332@380
So it seems that there is a hand full of tests that could need some hardening …
Chris
Am 15.06.17, 14:42 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Thx for the cleanup for Eclipse.
> On Jun 15, 2017, at 7:09 AM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...It seems as if some unit-tests tend to fail in random intervals. Right now, I had 3 Travis runs without any functional changes and I got failures on 3 different places. Is this “normal”? If yes … I think I should start identifiying the “random failers” and investigate the reason.
What were the failing tests?
MqttOpenTest and Http[Global]Test use external public servers and can be impacted by them. An attempt has been made to desensitize them (e.g., trying to connect and then skipping instead of failing tests) but sometimes things seem to leak through.
All of the tests have time limits imposed and will “fail" if they don’t complete in time (some have had more narrow limits than others). I’ve seen these sort of failures on occasion such as when the test system is heavily loaded.
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Thx for the cleanup for Eclipse.
> On Jun 15, 2017, at 7:09 AM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...It seems as if some unit-tests tend to fail in random intervals. Right now, I had 3 Travis runs without any functional changes and I got failures on 3 different places. Is this “normal”? If yes … I think I should start identifiying the “random failers” and investigate the reason.
What were the failing tests?
MqttOpenTest and Http[Global]Test use external public servers and can be impacted by them. An attempt has been made to desensitize them (e.g., trying to connect and then skipping instead of failing tests) but sometimes things seem to leak through.
All of the tests have time limits imposed and will “fail" if they don’t complete in time (some have had more narrow limits than others). I’ve seen these sort of failures on occasion such as when the test system is heavily loaded.
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Just another question …
It seems as if some unit-tests tend to fail in random intervals. Right now, I had 3 Travis runs without any functional changes and I got failures on 3 different places. Is this “normal”? If yes … I think I should start identifiying the “random failers” and investigate the reason.
Chris
Am 15.06.17, 12:49 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Your wish is my command ;-)
I just installed the latest Eclipse “Neon.3” in the Developer edition (which comes with Maven support).
Then I manually deleted all existing eclipse settings in the Edgent project as otherwise Eclipse would have insisted on using them instead. Also, I added them to the main gitignore file to prevent them from being tracked when re-created.
After that I imported my existing Maven project into my Eclipse workspace. It correctly detected all plugins I am using in the build and correctly setup the M2E plugins for these (glad I’m just using the usual suspects of maven plugins). In the end, I still got 3 errors complaining about the copy plugin not being able to copy because the jars not having been assembled yet. So, I moved all tests that require the loading of WAR or JAR files to the integration-test phase (where they belong anyway) and re-tried. Eclipse still had the same issues. In the end, I forced Eclipse to ignore the plugin execution. So, If you try to execute any of the “IT” suffixed tests in Eclipse, these will probably fail if you haven’t done a Maven build prior to test execution.
Right now, I have the project imported in Eclipse and am getting 0 errors. I tried running some Junit tests from Eclipse and all worked nicely :-)
So, I guess were coming close to the finishing line.
Chris
Am 14.06.17, 18:09 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Yup, just noticed it. Looking good!
> On Jun 14, 2017, at 11:36 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Ok, so I just pushed a major update.
>
> On my machine with these changes both the java7 and java8 builds seem to be working. In the next few days I’ll probably play around with some of the excluded modules and see if we can include them in the java7 build. Right now, it sort of looks as if they were excluded simply because they didn’t immediately work. So I’ll go through them and try to find out which ones we can add and write down why some cant (This information seems to be missing at the moment)
I suggest deferring the “add more to java7” effort until everything else needed to reach the state of being able to develop and make a release with the new machinery is done. Maybe deferring until after actually making a release with the new machinery - IMO we should do one asap.
> Another thing I did, was to add the RAT plugin, go through the exclusions and clean up the exclusions. Some were excluding stuff that shouldn’t be excluded and it reported quite a hand full of files without ASF headers. I added them so please have a look if this was ok.
LGTM
>
> The Eclipse support is also on my list (even if I have to admit that I always hate having to open that IDE ;-) )
:-) the sooner the better so that more folks can pull in the PR to experiment.
Thanks!
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Great news/progress!
I think it’s OK if j7 tests are not run for PRs/travis and only run from Jenkins for SNAPSHOTS/release
Shorter PR validation times… yay! :-)
Will all this work for creating our android platform release artifacts?
Two things happen when generating the android release:
- a subset of the j7 jars are included (some filtered out due to utilizing features not available on android)
- the android-{topology, hardware} jars are built on j8/j7 and included in the android release,
but they are removed from j8/j7 before releasing j8/j7.
— Dale
> On Jun 17, 2017, at 1:06 PM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Ok … reporting some progress here.
>
> I managed to get the build to create the java8 and the java7 artifacts in one run and to run the tests in both versions.
> The only thing I am currently struggling with is the Maven toolchain support.
>
> Maven provides a mechanism called “toolchains” in which you can for example define multiple JDKs and use this to do exactly what we want: run all tests in the normal modules with Java8 and the ones in the Java7 modules with Java7. Unfortunately, we would need the paths to the JDKs on Travis to use this. It would be easy on Apache Jenkins, but right now I think it would probably be better to let the tests on Travis all run with Java8 and to run the Java7 tests with Java7 on the ASF Jenkins.
>
> What do you think?
>
> Will continue porting all the individual modules … currently only ported the api modules . A lot more to go ;-)
>
> Chris
>
>
>
> Am 17.06.17, 16:29 schrieb "Christofer Dutz" <ch...@c-ware.de>:
>
> Ok … so I think I might have a possible solution, thanks to Karl Heinz Marbaise on the Maven list.
>
> We don’t use profiles at all and build all artifacts in one run. The difference would be that we create mini modules in the “java7” directory which build the java7 versions of the java8 artifacs by copying and unpacking the java8 versions to the mini projects target directory, have retrolambda do it’s magic there and package them normally. In this case, I would like to use a different groupId for the java7 artifacts instead of using different versions as this way we will have far less trouble releasing stuff.
>
> Using maven toolchains I might even be able to run the transformed tests in the mini projects with java7 … haven’t used toolchains till now, but it looks as if they would deliver exactly the functionality we need.
>
> So I’m going to give this approach a try.
>
> Chris
>
>
> Am 16.06.17, 18:43 schrieb "Dale LaBossiere" <dm...@gmail.com>:
>
>
>> On Jun 16, 2017, at 4:46 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>>
>> In case of a variable for the groupId the warning is the same except that it mentions groupId instead of version :-(
>
> Hmm… any thoughts on plausible alternatives? ...and implications to
> dev, release, and consumption of java7&android Edgent?
>
> At a high level, something just seems wrong that it’s this hard to
> create multiple variants of a product w/mvn, where the variants’
> jars need different coordinates along some dimension.
> The retrolambda plugin may be complicating things but it doesn't
> feel like it’s the cause of the main issue: variable coordinate decls.
>
> Related, without the use of variable-version decls, even ignoring
> this j8/j7 thing, there would be <version>1.2.0-SNAPSHOT</version>
> throughout every one of our (~65) poms. Even using
> variable-version decls, that static decl is present in every pom’s
> <parent> decl! Is it the really case that when we want to release
> 1.2.0 (not SNAPSHOT) or when it’s time for 1.3.0, every pom
> has to be edited?
>
> Just brainstorming… my head hurts :-\ I apologize in advance… :-)
>
> E<n> means Edgent-java<n> ...
>
> Is there any ~convenient way to synthesize/use alternate pom’s for E7?
> Might there be a way to leverage “mvn —file <pom>” and also pom.xml:<relativePath>?
>
> e.g., have the pom.xml implicitly be j8/E8 oriented with non-variable
> coordinate decl for the project.
>
> Imagine building / staging a RC for E7 would be something like:
>
> mk-j7-poms # create pom-j7.xml files adding “-java7” in versions and a “relativePath” of ../pom-j7.xml
> # **/pom-7.xml added to .gitignore; target dir is still variable
>
> mvn -f pom-j7.xml -Pjava7 clean install -DskipTests
> # TBD do j7 testing
> # …
> mvn -f pom-j7.xml -Pjava7 deploy -Papache-release -DskipTests
>
> Or if we’re going that far, then maybe needing/using a separate
> clone/workspace to build & deploy E7 might be tolerable?
>
> Though… while the gradle/ant tooling created j7 jars using *the* j8 jars,
> this mvn tooling is already regenerating the j8 classes
> and then running them through retrolambda, right? (if so, ugh)
>
> Kinda feels like E7 artifacts should instead just declare a
> dependency on their corresponding E8 jar artifact (not src)
> and just do some (retrolambda) transform to generate the E7 jar artifacts.
> Which seems to imply separate (~parallel) simpler poms for E7
> which can then also have non-variable-version decl?
> e.g., then something like
> cd java7
> mvn clean install -DskipTests
> # TBD test
> mvn deploy -Papache-release -DskipTests
>
> Also, the current "mvn install -Papache-release -Pjava7"
> generates -sources.jar, -javadoc.jar, -source-release.zip
> Do we really want to distribute those (duplicate) set of those artifacts?
> Don’t know if that might just mean more config tweaking or
> if the retrolambda plugin / “deploy” flow for java isn’t really what
> we want.
>
> Your thoughts on all of this?
>
> — Dale
>
>
>
>
>
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Ok … reporting some progress here.
I managed to get the build to create the java8 and the java7 artifacts in one run and to run the tests in both versions.
The only thing I am currently struggling with is the Maven toolchain support.
Maven provides a mechanism called “toolchains” in which you can for example define multiple JDKs and use this to do exactly what we want: run all tests in the normal modules with Java8 and the ones in the Java7 modules with Java7. Unfortunately, we would need the paths to the JDKs on Travis to use this. It would be easy on Apache Jenkins, but right now I think it would probably be better to let the tests on Travis all run with Java8 and to run the Java7 tests with Java7 on the ASF Jenkins.
What do you think?
Will continue porting all the individual modules … currently only ported the api modules . A lot more to go ;-)
Chris
Am 17.06.17, 16:29 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Ok … so I think I might have a possible solution, thanks to Karl Heinz Marbaise on the Maven list.
We don’t use profiles at all and build all artifacts in one run. The difference would be that we create mini modules in the “java7” directory which build the java7 versions of the java8 artifacs by copying and unpacking the java8 versions to the mini projects target directory, have retrolambda do it’s magic there and package them normally. In this case, I would like to use a different groupId for the java7 artifacts instead of using different versions as this way we will have far less trouble releasing stuff.
Using maven toolchains I might even be able to run the transformed tests in the mini projects with java7 … haven’t used toolchains till now, but it looks as if they would deliver exactly the functionality we need.
So I’m going to give this approach a try.
Chris
Am 16.06.17, 18:43 schrieb "Dale LaBossiere" <dm...@gmail.com>:
> On Jun 16, 2017, at 4:46 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> In case of a variable for the groupId the warning is the same except that it mentions groupId instead of version :-(
Hmm… any thoughts on plausible alternatives? ...and implications to
dev, release, and consumption of java7&android Edgent?
At a high level, something just seems wrong that it’s this hard to
create multiple variants of a product w/mvn, where the variants’
jars need different coordinates along some dimension.
The retrolambda plugin may be complicating things but it doesn't
feel like it’s the cause of the main issue: variable coordinate decls.
Related, without the use of variable-version decls, even ignoring
this j8/j7 thing, there would be <version>1.2.0-SNAPSHOT</version>
throughout every one of our (~65) poms. Even using
variable-version decls, that static decl is present in every pom’s
<parent> decl! Is it the really case that when we want to release
1.2.0 (not SNAPSHOT) or when it’s time for 1.3.0, every pom
has to be edited?
Just brainstorming… my head hurts :-\ I apologize in advance… :-)
E<n> means Edgent-java<n> ...
Is there any ~convenient way to synthesize/use alternate pom’s for E7?
Might there be a way to leverage “mvn —file <pom>” and also pom.xml:<relativePath>?
e.g., have the pom.xml implicitly be j8/E8 oriented with non-variable
coordinate decl for the project.
Imagine building / staging a RC for E7 would be something like:
mk-j7-poms # create pom-j7.xml files adding “-java7” in versions and a “relativePath” of ../pom-j7.xml
# **/pom-7.xml added to .gitignore; target dir is still variable
mvn -f pom-j7.xml -Pjava7 clean install -DskipTests
# TBD do j7 testing
# …
mvn -f pom-j7.xml -Pjava7 deploy -Papache-release -DskipTests
Or if we’re going that far, then maybe needing/using a separate
clone/workspace to build & deploy E7 might be tolerable?
Though… while the gradle/ant tooling created j7 jars using *the* j8 jars,
this mvn tooling is already regenerating the j8 classes
and then running them through retrolambda, right? (if so, ugh)
Kinda feels like E7 artifacts should instead just declare a
dependency on their corresponding E8 jar artifact (not src)
and just do some (retrolambda) transform to generate the E7 jar artifacts.
Which seems to imply separate (~parallel) simpler poms for E7
which can then also have non-variable-version decl?
e.g., then something like
cd java7
mvn clean install -DskipTests
# TBD test
mvn deploy -Papache-release -DskipTests
Also, the current "mvn install -Papache-release -Pjava7"
generates -sources.jar, -javadoc.jar, -source-release.zip
Do we really want to distribute those (duplicate) set of those artifacts?
Don’t know if that might just mean more config tweaking or
if the retrolambda plugin / “deploy” flow for java isn’t really what
we want.
Your thoughts on all of this?
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Ok … so I think I might have a possible solution, thanks to Karl Heinz Marbaise on the Maven list.
We don’t use profiles at all and build all artifacts in one run. The difference would be that we create mini modules in the “java7” directory which build the java7 versions of the java8 artifacs by copying and unpacking the java8 versions to the mini projects target directory, have retrolambda do it’s magic there and package them normally. In this case, I would like to use a different groupId for the java7 artifacts instead of using different versions as this way we will have far less trouble releasing stuff.
Using maven toolchains I might even be able to run the transformed tests in the mini projects with java7 … haven’t used toolchains till now, but it looks as if they would deliver exactly the functionality we need.
So I’m going to give this approach a try.
Chris
Am 16.06.17, 18:43 schrieb "Dale LaBossiere" <dm...@gmail.com>:
> On Jun 16, 2017, at 4:46 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> In case of a variable for the groupId the warning is the same except that it mentions groupId instead of version :-(
Hmm… any thoughts on plausible alternatives? ...and implications to
dev, release, and consumption of java7&android Edgent?
At a high level, something just seems wrong that it’s this hard to
create multiple variants of a product w/mvn, where the variants’
jars need different coordinates along some dimension.
The retrolambda plugin may be complicating things but it doesn't
feel like it’s the cause of the main issue: variable coordinate decls.
Related, without the use of variable-version decls, even ignoring
this j8/j7 thing, there would be <version>1.2.0-SNAPSHOT</version>
throughout every one of our (~65) poms. Even using
variable-version decls, that static decl is present in every pom’s
<parent> decl! Is it the really case that when we want to release
1.2.0 (not SNAPSHOT) or when it’s time for 1.3.0, every pom
has to be edited?
Just brainstorming… my head hurts :-\ I apologize in advance… :-)
E<n> means Edgent-java<n> ...
Is there any ~convenient way to synthesize/use alternate pom’s for E7?
Might there be a way to leverage “mvn —file <pom>” and also pom.xml:<relativePath>?
e.g., have the pom.xml implicitly be j8/E8 oriented with non-variable
coordinate decl for the project.
Imagine building / staging a RC for E7 would be something like:
mk-j7-poms # create pom-j7.xml files adding “-java7” in versions and a “relativePath” of ../pom-j7.xml
# **/pom-7.xml added to .gitignore; target dir is still variable
mvn -f pom-j7.xml -Pjava7 clean install -DskipTests
# TBD do j7 testing
# …
mvn -f pom-j7.xml -Pjava7 deploy -Papache-release -DskipTests
Or if we’re going that far, then maybe needing/using a separate
clone/workspace to build & deploy E7 might be tolerable?
Though… while the gradle/ant tooling created j7 jars using *the* j8 jars,
this mvn tooling is already regenerating the j8 classes
and then running them through retrolambda, right? (if so, ugh)
Kinda feels like E7 artifacts should instead just declare a
dependency on their corresponding E8 jar artifact (not src)
and just do some (retrolambda) transform to generate the E7 jar artifacts.
Which seems to imply separate (~parallel) simpler poms for E7
which can then also have non-variable-version decl?
e.g., then something like
cd java7
mvn clean install -DskipTests
# TBD test
mvn deploy -Papache-release -DskipTests
Also, the current "mvn install -Papache-release -Pjava7"
generates -sources.jar, -javadoc.jar, -source-release.zip
Do we really want to distribute those (duplicate) set of those artifacts?
Don’t know if that might just mean more config tweaking or
if the retrolambda plugin / “deploy” flow for java isn’t really what
we want.
Your thoughts on all of this?
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
> On Jun 16, 2017, at 4:46 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> In case of a variable for the groupId the warning is the same except that it mentions groupId instead of version :-(
Hmm… any thoughts on plausible alternatives? ...and implications to
dev, release, and consumption of java7&android Edgent?
At a high level, something just seems wrong that it’s this hard to
create multiple variants of a product w/mvn, where the variants’
jars need different coordinates along some dimension.
The retrolambda plugin may be complicating things but it doesn't
feel like it’s the cause of the main issue: variable coordinate decls.
Related, without the use of variable-version decls, even ignoring
this j8/j7 thing, there would be <version>1.2.0-SNAPSHOT</version>
throughout every one of our (~65) poms. Even using
variable-version decls, that static decl is present in every pom’s
<parent> decl! Is it the really case that when we want to release
1.2.0 (not SNAPSHOT) or when it’s time for 1.3.0, every pom
has to be edited?
Just brainstorming… my head hurts :-\ I apologize in advance… :-)
E<n> means Edgent-java<n> ...
Is there any ~convenient way to synthesize/use alternate pom’s for E7?
Might there be a way to leverage “mvn —file <pom>” and also pom.xml:<relativePath>?
e.g., have the pom.xml implicitly be j8/E8 oriented with non-variable
coordinate decl for the project.
Imagine building / staging a RC for E7 would be something like:
mk-j7-poms # create pom-j7.xml files adding “-java7” in versions and a “relativePath” of ../pom-j7.xml
# **/pom-7.xml added to .gitignore; target dir is still variable
mvn -f pom-j7.xml -Pjava7 clean install -DskipTests
# TBD do j7 testing
# …
mvn -f pom-j7.xml -Pjava7 deploy -Papache-release -DskipTests
Or if we’re going that far, then maybe needing/using a separate
clone/workspace to build & deploy E7 might be tolerable?
Though… while the gradle/ant tooling created j7 jars using *the* j8 jars,
this mvn tooling is already regenerating the j8 classes
and then running them through retrolambda, right? (if so, ugh)
Kinda feels like E7 artifacts should instead just declare a
dependency on their corresponding E8 jar artifact (not src)
and just do some (retrolambda) transform to generate the E7 jar artifacts.
Which seems to imply separate (~parallel) simpler poms for E7
which can then also have non-variable-version decl?
e.g., then something like
cd java7
mvn clean install -DskipTests
# TBD test
mvn deploy -Papache-release -DskipTests
Also, the current "mvn install -Papache-release -Pjava7"
generates -sources.jar, -javadoc.jar, -source-release.zip
Do we really want to distribute those (duplicate) set of those artifacts?
Don’t know if that might just mean more config tweaking or
if the retrolambda plugin / “deploy” flow for java isn’t really what
we want.
Your thoughts on all of this?
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
In case of a variable for the groupId the warning is the same except that it mentions groupId instead of version :-(
Chris
Am 15.06.17, 23:18 schrieb "Dale LaBossiere" <dm...@gmail.com>:
ugh. There are ~160 lines of this every time you run a mvn cmd… and I hadn’t noticed the:
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
Sounds pretty ominous.
Seems to me like people will quickly tire of / get annoyed by this.
Might it be a lesser of evils to put the java17 in the groupId instead?
— Dale
> On Jun 15, 2017, at 11:55 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Unfortunately we have to live with that.
> Using variables for the versions can cause problems. In our case I made sure they don't. But we need that in order to create the java 8 and java 7 packages.
>
> But Maven tends to be over careful.
>
> Chris
>
>
>
> Von meinem Samsung Galaxy Smartphone gesendet.
>
>
> -------- Ursprüngliche Nachricht --------
> Von: Dale LaBossiere <dm...@gmail.com>
> Datum: 15.06.17 17:19 (GMT+01:00)
> An: dev@edgent.apache.org
> Betreff: Re: Understanding the snapshot and release process
>
> I just pulled down the latest PR contents into a new clone. I get a bunch of warnings like the one below (but things work).
>
> mvn clean install
> [INFO] Scanning for projects...
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model for org.apache.edgent.analytics:edgent-analytics-sensors:jar:1.2.0-SNAPSHOT
> [WARNING] 'version' contains an expression but should be a constant. @ org.apache.edgent.analytics:edgent-analytics-sensors:${edgent.version}, /Users/dlaboss/git/incubator-edgen\
> t-4/analytics/sensors/pom.xml, line 31, column 12
>
> — Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
ugh. There are ~160 lines of this every time you run a mvn cmd… and I hadn’t noticed the:
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
Sounds pretty ominous.
Seems to me like people will quickly tire of / get annoyed by this.
Might it be a lesser of evils to put the java17 in the groupId instead?
— Dale
> On Jun 15, 2017, at 11:55 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Unfortunately we have to live with that.
> Using variables for the versions can cause problems. In our case I made sure they don't. But we need that in order to create the java 8 and java 7 packages.
>
> But Maven tends to be over careful.
>
> Chris
>
>
>
> Von meinem Samsung Galaxy Smartphone gesendet.
>
>
> -------- Ursprüngliche Nachricht --------
> Von: Dale LaBossiere <dm...@gmail.com>
> Datum: 15.06.17 17:19 (GMT+01:00)
> An: dev@edgent.apache.org
> Betreff: Re: Understanding the snapshot and release process
>
> I just pulled down the latest PR contents into a new clone. I get a bunch of warnings like the one below (but things work).
>
> mvn clean install
> [INFO] Scanning for projects...
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model for org.apache.edgent.analytics:edgent-analytics-sensors:jar:1.2.0-SNAPSHOT
> [WARNING] 'version' contains an expression but should be a constant. @ org.apache.edgent.analytics:edgent-analytics-sensors:${edgent.version}, /Users/dlaboss/git/incubator-edgen\
> t-4/analytics/sensors/pom.xml, line 31, column 12
>
> — Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Unfortunately we have to live with that.
Using variables for the versions can cause problems. In our case I made sure they don't. But we need that in order to create the java 8 and java 7 packages.
But Maven tends to be over careful.
Chris
Von meinem Samsung Galaxy Smartphone gesendet.
-------- Ursprüngliche Nachricht --------
Von: Dale LaBossiere <dm...@gmail.com>
Datum: 15.06.17 17:19 (GMT+01:00)
An: dev@edgent.apache.org
Betreff: Re: Understanding the snapshot and release process
I just pulled down the latest PR contents into a new clone. I get a bunch of warnings like the one below (but things work).
mvn clean install
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.edgent.analytics:edgent-analytics-sensors:jar:1.2.0-SNAPSHOT
[WARNING] 'version' contains an expression but should be a constant. @ org.apache.edgent.analytics:edgent-analytics-sensors:${edgent.version}, /Users/dlaboss/git/incubator-edgen\
t-4/analytics/sensors/pom.xml, line 31, column 12
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
I just pulled down the latest PR contents into a new clone. I get a bunch of warnings like the one below (but things work).
mvn clean install
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.edgent.analytics:edgent-analytics-sensors:jar:1.2.0-SNAPSHOT
[WARNING] 'version' contains an expression but should be a constant. @ org.apache.edgent.analytics:edgent-analytics-sensors:${edgent.version}, /Users/dlaboss/git/incubator-edgen\
t-4/analytics/sensors/pom.xml, line 31, column 12
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Your wish is my command ;-)
I just installed the latest Eclipse “Neon.3” in the Developer edition (which comes with Maven support).
Then I manually deleted all existing eclipse settings in the Edgent project as otherwise Eclipse would have insisted on using them instead. Also, I added them to the main gitignore file to prevent them from being tracked when re-created.
After that I imported my existing Maven project into my Eclipse workspace. It correctly detected all plugins I am using in the build and correctly setup the M2E plugins for these (glad I’m just using the usual suspects of maven plugins). In the end, I still got 3 errors complaining about the copy plugin not being able to copy because the jars not having been assembled yet. So, I moved all tests that require the loading of WAR or JAR files to the integration-test phase (where they belong anyway) and re-tried. Eclipse still had the same issues. In the end, I forced Eclipse to ignore the plugin execution. So, If you try to execute any of the “IT” suffixed tests in Eclipse, these will probably fail if you haven’t done a Maven build prior to test execution.
Right now, I have the project imported in Eclipse and am getting 0 errors. I tried running some Junit tests from Eclipse and all worked nicely :-)
So, I guess were coming close to the finishing line.
Chris
Am 14.06.17, 18:09 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Yup, just noticed it. Looking good!
> On Jun 14, 2017, at 11:36 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Ok, so I just pushed a major update.
>
> On my machine with these changes both the java7 and java8 builds seem to be working. In the next few days I’ll probably play around with some of the excluded modules and see if we can include them in the java7 build. Right now, it sort of looks as if they were excluded simply because they didn’t immediately work. So I’ll go through them and try to find out which ones we can add and write down why some cant (This information seems to be missing at the moment)
I suggest deferring the “add more to java7” effort until everything else needed to reach the state of being able to develop and make a release with the new machinery is done. Maybe deferring until after actually making a release with the new machinery - IMO we should do one asap.
> Another thing I did, was to add the RAT plugin, go through the exclusions and clean up the exclusions. Some were excluding stuff that shouldn’t be excluded and it reported quite a hand full of files without ASF headers. I added them so please have a look if this was ok.
LGTM
>
> The Eclipse support is also on my list (even if I have to admit that I always hate having to open that IDE ;-) )
:-) the sooner the better so that more folks can pull in the PR to experiment.
Thanks!
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Yup, just noticed it. Looking good!
> On Jun 14, 2017, at 11:36 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Ok, so I just pushed a major update.
>
> On my machine with these changes both the java7 and java8 builds seem to be working. In the next few days I’ll probably play around with some of the excluded modules and see if we can include them in the java7 build. Right now, it sort of looks as if they were excluded simply because they didn’t immediately work. So I’ll go through them and try to find out which ones we can add and write down why some cant (This information seems to be missing at the moment)
I suggest deferring the “add more to java7” effort until everything else needed to reach the state of being able to develop and make a release with the new machinery is done. Maybe deferring until after actually making a release with the new machinery - IMO we should do one asap.
> Another thing I did, was to add the RAT plugin, go through the exclusions and clean up the exclusions. Some were excluding stuff that shouldn’t be excluded and it reported quite a hand full of files without ASF headers. I added them so please have a look if this was ok.
LGTM
>
> The Eclipse support is also on my list (even if I have to admit that I always hate having to open that IDE ;-) )
:-) the sooner the better so that more folks can pull in the PR to experiment.
Thanks!
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Ok, so I just pushed a major update.
On my machine with these changes both the java7 and java8 builds seem to be working. In the next few days I’ll probably play around with some of the excluded modules and see if we can include them in the java7 build. Right now, it sort of looks as if they were excluded simply because they didn’t immediately work. So I’ll go through them and try to find out which ones we can add and write down why some cant (This information seems to be missing at the moment)
Another thing I did, was to add the RAT plugin, go through the exclusions and clean up the exclusions. Some were excluding stuff that shouldn’t be excluded and it reported quite a hand full of files without ASF headers. I added them so please have a look if this was ok.
The Eclipse support is also on my list (even if I have to admit that I always hate having to open that IDE ;-) )
Chris
Am 13.06.17, 17:45 schrieb "Dale LaBossiere" <dm...@gmail.com>:
> On Jun 12, 2017, at 4:56 PM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...
> In my opinion IDE settings shouldn’t be checked in and should rather be added to the gitignore list. Don’t know how often we had issues in Flex with checked in IDE settings :-(
Agreed, with maven projects/pom’s in place, the Eclipse IDE settings/.project/.classpath would be removed from the repo.
>
> The warnings are related to the fact that all Apache Maven configurations have the “Apache root pom” as top-most parent. A lot Some things are configured there. However I’m using different versions of some plugins and re-defining them in the edgent-parent. That’s what the first block of warnings are about.
Since the overrides are intentional, can you add the “annotations” to eliminate the warnings? (and maybe a comment as to why a different version is being selected?) https://stackoverflow.com/questions/30782453/maven-suppress-overriding-managed-version-warning-in-eclipse <https://stackoverflow.com/questions/30782453/maven-suppress-overriding-managed-version-warning-in-eclipse>
>
> The rest of the blocks are because in order to work correctly the e2m plugin has built-in configurations on what to do for which default maven plugin. This is how Eclipse knows when to copy resources, compile classes and eventually even generate code. However per default not all plugins are handled by the m2e plugin. That’s what the second block is about. There is a way to get rid of them and eventually even tell Eclipse what to do, but this would need some fine tuning. I would like to address that as soon as we’re finished. Otherwise I would probably have to re-configure that over and over again. Mabe it would be good to tell Eclipse to ignore the plugins it doesn’t know so the errors go away and then to fine tune them afterwards.
Dealing with the warnings later seems OK. Can you make the change to eliminate the errors now? You probably know what needs to be done but fwiw here’s what Eclipse quick-fix added:
@dales-mbp:1049> diff test/fvtiot/pom.xml{,.sav}
62,96d61
< <pluginManagement>
< <plugins>
< <!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
< <plugin>
< <groupId>org.eclipse.m2e</groupId>
< <artifactId>lifecycle-mapping</artifactId>
< <version>1.0.0</version>
< <configuration>
< <lifecycleMappingMetadata>
< <pluginExecutions>
< <pluginExecution>
< <pluginExecutionFilter>
< <groupId>
< org.apache.maven.plugins
< </groupId>
< <artifactId>
< maven-dependency-plugin
< </artifactId>
< <versionRange>
< [3.0.1,)
< </versionRange>
< <goals>
< <goal>copy</goal>
< </goals>
< </pluginExecutionFilter>
< <action>
< <ignore></ignore>
< </action>
< </pluginExecution>
< </pluginExecutions>
< </lifecycleMappingMetadata>
< </configuration>
< </plugin>
< </plugins>
< </pluginManagement>
@dales-mbp:1050>
Thanks,
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
> On Jun 12, 2017, at 4:56 PM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...
> In my opinion IDE settings shouldn’t be checked in and should rather be added to the gitignore list. Don’t know how often we had issues in Flex with checked in IDE settings :-(
Agreed, with maven projects/pom’s in place, the Eclipse IDE settings/.project/.classpath would be removed from the repo.
>
> The warnings are related to the fact that all Apache Maven configurations have the “Apache root pom” as top-most parent. A lot Some things are configured there. However I’m using different versions of some plugins and re-defining them in the edgent-parent. That’s what the first block of warnings are about.
Since the overrides are intentional, can you add the “annotations” to eliminate the warnings? (and maybe a comment as to why a different version is being selected?) https://stackoverflow.com/questions/30782453/maven-suppress-overriding-managed-version-warning-in-eclipse <https://stackoverflow.com/questions/30782453/maven-suppress-overriding-managed-version-warning-in-eclipse>
>
> The rest of the blocks are because in order to work correctly the e2m plugin has built-in configurations on what to do for which default maven plugin. This is how Eclipse knows when to copy resources, compile classes and eventually even generate code. However per default not all plugins are handled by the m2e plugin. That’s what the second block is about. There is a way to get rid of them and eventually even tell Eclipse what to do, but this would need some fine tuning. I would like to address that as soon as we’re finished. Otherwise I would probably have to re-configure that over and over again. Mabe it would be good to tell Eclipse to ignore the plugins it doesn’t know so the errors go away and then to fine tune them afterwards.
Dealing with the warnings later seems OK. Can you make the change to eliminate the errors now? You probably know what needs to be done but fwiw here’s what Eclipse quick-fix added:
@dales-mbp:1049> diff test/fvtiot/pom.xml{,.sav}
62,96d61
< <pluginManagement>
< <plugins>
< <!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
< <plugin>
< <groupId>org.eclipse.m2e</groupId>
< <artifactId>lifecycle-mapping</artifactId>
< <version>1.0.0</version>
< <configuration>
< <lifecycleMappingMetadata>
< <pluginExecutions>
< <pluginExecution>
< <pluginExecutionFilter>
< <groupId>
< org.apache.maven.plugins
< </groupId>
< <artifactId>
< maven-dependency-plugin
< </artifactId>
< <versionRange>
< [3.0.1,)
< </versionRange>
< <goals>
< <goal>copy</goal>
< </goals>
< </pluginExecutionFilter>
< <action>
< <ignore></ignore>
< </action>
< </pluginExecution>
< </pluginExecutions>
< </lifecycleMappingMetadata>
< </configuration>
< </plugin>
< </plugins>
< </pluginManagement>
@dales-mbp:1050>
Thanks,
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Dale,
Glad you like that … I was going to address the “checked in IDE settings” thing soon ;-)
I am working with IntelliJ without any issues in my Maven setup. Had to rename the Gradle build to make IntelliJ chose Maven though.
In my opinion IDE settings shouldn’t be checked in and should rather be added to the gitignore list. Don’t know how often we had issues in Flex with checked in IDE settings :-(
The warnings are related to the fact that all Apache Maven configurations have the “Apache root pom” as top-most parent. A lot Some things are configured there. However I’m using different versions of some plugins and re-defining them in the edgent-parent. That’s what the first block of warnings are about.
The rest of the blocks are because in order to work correctly the e2m plugin has built-in configurations on what to do for which default maven plugin. This is how Eclipse knows when to copy resources, compile classes and eventually even generate code. However per default not all plugins are handled by the m2e plugin. That’s what the second block is about. There is a way to get rid of them and eventually even tell Eclipse what to do, but this would need some fine tuning. I would like to address that as soon as we’re finished. Otherwise I would probably have to re-configure that over and over again. Mabe it would be good to tell Eclipse to ignore the plugins it doesn’t know so the errors go away and then to fine tune them afterwards.
Chris
Von: Dale LaBossiere <dm...@gmail.com>
Antworten an: "dev@edgent.apache.org" <de...@edgent.apache.org>
Datum: Montag, 12. Juni 2017 um 22:43
An: "dev@edgent.apache.org" <de...@edgent.apache.org>
Betreff: Re: Understanding the snapshot and release process
Chris, I started to try to get Eclipse working with the maven PR-309 changes. Making some progress.
At a high level, I removed all existing .project and .classpath Eclipse files from my local git clone.
Then I created a new Eclipse workspace and did an “Import existing Maven Projects”.
Eclipse then created a .project and .classpath from the poms.
Seemingly all good stuff. And presumably analogous to importing a maven project into IntelliJ.
But I have 3 build problems and 66 (mostly pom.xml related) warnings.
What do you make of these warnings in the top-level pom.xml? Googling seems to tell me that the “Overriding managed version” warnings are because the top-level pom.xml is overriding the versions for the plugins inherited from the parent pom.xml (org.apache:apache:18 ???) ???
[cid:image001.png@01D2E3CF.2BA7FA50]
There are a lot (every project/module I assume) of these:
[cid:image002.png@01D2E3CF.2BA7FA50]
Any thoughts on the 3 errors?
[cid:image003.png@01D2E3CF.2BA7FA50]
the full error msgs are:
[cid:image004.png@01D2E3CF.2BA7FA50]
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Chris, I started to try to get Eclipse working with the maven PR-309 changes. Making some progress.
At a high level, I removed all existing .project and .classpath Eclipse files from my local git clone.
Then I created a new Eclipse workspace and did an “Import existing Maven Projects”.
Eclipse then created a .project and .classpath from the poms.
Seemingly all good stuff. And presumably analogous to importing a maven project into IntelliJ.
But I have 3 build problems and 66 (mostly pom.xml related) warnings.
What do you make of these warnings in the top-level pom.xml? Googling seems to tell me that the “Overriding managed version” warnings are because the top-level pom.xml is overriding the versions for the plugins inherited from the parent pom.xml (org.apache:apache:18 ???) ???
There are a lot (every project/module I assume) of these:
Any thoughts on the 3 errors?
the full error msgs are:
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
> On Jun 13, 2017, at 10:16 AM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...
> Right now, I am still having issues running all Tests in the java7 version … math3 is sort of causing issues, I haven’t quite figured out why.
Edgent’s java7 platform support doesn’t include everything in the java8 platform. See https://github.com/apache/incubator-edgent/blob/master/JAVA_SUPPORT.md
For the most part I’m not quite sure why that is. In some cases I think it was just “we’ll confirm module/project xyz works on java7 when there’s a demonstrated demand for it.”
I see math3 is one such module, hence it hasn’t been getting included/tested in the java7 config. See the commentary in https://github.com/apache/incubator-edgent/blob/master/platform/java7/build.xml#L118 <https://github.com/apache/incubator-edgent/blob/master/platform/java7/build.xml#L118>
The same is true for the android platform support, though apparently JMX isn’t supported on android so that explains some of its exclusions. See https://github.com/apache/incubator-edgent/blob/master/platform/android/build.xml#L41 <https://github.com/apache/incubator-edgent/blob/master/platform/android/build.xml#L41>
The net is that some conditionalization of what’s included in java7 and android is needed.
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Aaahh … well that’s a different thing … that’s already possible with maven :-)
Right now, all Apache releases go to a so-called release-repository. That’s a Maven repository containing only the artifacts of the current release. The project can vote on this. If the vote doesn’t pass, the repo is simply dropped and it was as if nothing had happened. If the release passes, the artifacts are released to the Apache Maven Release Repo and this is automatically mirrored to Maven Central.
So, it seems this part is already taken care of … one more thing to check off the list ;-)
I guess Travis was chosen, because Github was selected as primary repo for the project and the integration with auto-built pull requests is great, I have to admit that. But it is problematic to setup auto deployment of SNAPSHOT versions as we would have to check in credentials for deploying to the Apache Maven repo and that’s not going to happen. As the build setup is quite trivial I think using both is the best option. Use Travis for the auto-testing of pull-requests. Use Apache Jenkins for deploying SNAPSHOTS and doing code-analysis on Sonarqube and stuff like that.
Right now, I am still having issues running all Tests in the java7 version … math3 is sort of causing issues, I haven’t quite figured out why.
Chris
Am 13.06.17, 15:59 schrieb "Dale LaBossiere" <dm...@gmail.com>:
> On Jun 12, 2017, at 5:04 PM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...
> I guess also executing the pre-built test with java7 should also be possible, but I’m struggling with one other of your requirements:
> “Only deploy everything if the java8, java7 and android builds succeed”. I hope you can see the dilemma.
It’s possible my terminology is confusing/inaccurate. Mostly I think I meant that for releases (not talking about snapshots nor release-candidate staging for voting) we should only make the final release artifacts, including convenience binaries, public only if/once all configurations have passed testing&voting.
I think it’s time for me to go back to all the ASF release doc info. I know a lot has changed and earlier I didn’t pay any attention to snapshots nor maven related things because we weren’t doing them.
> Here it might be a better option to add a Jenkinsfile to the repository and setup a Jenkins job on the ASF Jenkins. With this I am a lot more flexible than with Travis. We could continue to use Travis for the auto-testing of pull requests, but would live with running the java7 testsuite with a java8 VM, but have the Jenkins job deploy the SNAPSHOTS to the ASF repo.
>
> What do you think about that? Building on Apache Machines also gives a lot of other benefits.
I don’t know why Travis was chosen. Is the ASF Jenkins service relatively new or not good/viable for PR-auto-test use? Regardless, partially or fully switching to Jenkins tooling as appropriate is OK with me - unless there are some things you forgot to mention :-)
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
> On Jun 12, 2017, at 5:04 PM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...
> I guess also executing the pre-built test with java7 should also be possible, but I’m struggling with one other of your requirements:
> “Only deploy everything if the java8, java7 and android builds succeed”. I hope you can see the dilemma.
It’s possible my terminology is confusing/inaccurate. Mostly I think I meant that for releases (not talking about snapshots nor release-candidate staging for voting) we should only make the final release artifacts, including convenience binaries, public only if/once all configurations have passed testing&voting.
I think it’s time for me to go back to all the ASF release doc info. I know a lot has changed and earlier I didn’t pay any attention to snapshots nor maven related things because we weren’t doing them.
> Here it might be a better option to add a Jenkinsfile to the repository and setup a Jenkins job on the ASF Jenkins. With this I am a lot more flexible than with Travis. We could continue to use Travis for the auto-testing of pull requests, but would live with running the java7 testsuite with a java8 VM, but have the Jenkins job deploy the SNAPSHOTS to the ASF repo.
>
> What do you think about that? Building on Apache Machines also gives a lot of other benefits.
I don’t know why Travis was chosen. Is the ASF Jenkins service relatively new or not good/viable for PR-auto-test use? Regardless, partially or fully switching to Jenkins tooling as appropriate is OK with me - unless there are some things you forgot to mention :-)
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Well Maven can do quite a lot of things. But there’s one base principal which I lay great emphasis on in my Maven trainings.
“If it’s hard to do with Maven, you’re probably doing it wrong” :-)
I guess the main problem is that we need Java8 to compile the tests and Java7 to run them. As a maven build fires up a VM to do the build the version of java used throughout the build is sort of defined by this call to run maven. Anything else would require to fork a new process with the JavaVM VM … this is not handled. If it was possible to build and test with java7, this wouldn’t be a problem.
I guess also executing the pre-built test with java7 should also be possible, but I’m struggling with one other of your requirements:
“Only deploy everything if the java8, java7 and android builds succeed”. I hope you can see the dilemma.
Here it might be a better option to add a Jenkinsfile to the repository and setup a Jenkins job on the ASF Jenkins. With this I am a lot more flexible than with Travis. We could continue to use Travis for the auto-testing of pull requests, but would live with running the java7 testsuite with a java8 VM, but have the Jenkins job deploy the SNAPSHOTS to the ASF repo.
What do you think about that? Building on Apache Machines also gives a lot of other benefits.
Chris
Am 12.06.17, 18:55 schrieb "Dale LaBossiere" <dm...@gmail.com>:
> On Jun 12, 2017, at 10:23 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Well “mvn test” would run the maven lifecycle up to the test phase, this includes compiling etc … and compiling doesn’t work with Java7.
Maven doesn’t have the notion of build-avoidance / artifact reuse?
I can imagine that even if it did, it might record the JDK version used to generate the artifacts and then not reuse them when the current JDK version is different.
> We could probably create some solution to manually load and run the tests inside the test-jars but integrating this into one nice Maven build could be a challenging task, so say the least … but we wouldn’t have to, if you would be ok with the signature-verification option.
I’m holding out till I better understand maven’s behavior/limitations and what other strategies might exist :-)
>
> Seems not, so I’ll continue digging … but this one will probably take a little longer.
What do other projects do if for example they build/release to a least-common-demonitor, e.g., 1.7, and still want to do actual test verification against the higher compatible/supported version, e.g., 1.8? They don’t/can’t use standard maven machinery for that? They must build some parallel test execution / reporting machinery?
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
> On Jun 12, 2017, at 10:23 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Well “mvn test” would run the maven lifecycle up to the test phase, this includes compiling etc … and compiling doesn’t work with Java7.
Maven doesn’t have the notion of build-avoidance / artifact reuse?
I can imagine that even if it did, it might record the JDK version used to generate the artifacts and then not reuse them when the current JDK version is different.
> We could probably create some solution to manually load and run the tests inside the test-jars but integrating this into one nice Maven build could be a challenging task, so say the least … but we wouldn’t have to, if you would be ok with the signature-verification option.
I’m holding out till I better understand maven’s behavior/limitations and what other strategies might exist :-)
>
> Seems not, so I’ll continue digging … but this one will probably take a little longer.
What do other projects do if for example they build/release to a least-common-demonitor, e.g., 1.7, and still want to do actual test verification against the higher compatible/supported version, e.g., 1.8? They don’t/can’t use standard maven machinery for that? They must build some parallel test execution / reporting machinery?
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Well “mvn test” would run the maven lifecycle up to the test phase, this includes compiling etc … and compiling doesn’t work with Java7.
We could probably create some solution to manually load and run the tests inside the test-jars but integrating this into one nice Maven build could be a challenging task, so say the least … but we wouldn’t have to, if you would be ok with the signature-verification option.
Seems not, so I’ll continue digging … but this one will probably take a little longer.
Chris
Am 12.06.17, 15:37 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Thanks for all the progress!
It’s a little unsettling to not be able to run the regression unit/functional tests w/1.7 :-)
If the local mvn repo has the up-to-date built 1.7 compat Edgent jars and the local workspace has the up-to-date built 1.7 compat test class files, why doesn’t running “mvn test -Pjava7” w/1.7 just reuse them?
— Dale
> On Jun 12, 2017, at 7:35 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Ok … so I’m working on the open issues … I do have one question though …
> Compiling with one java version and then running a different one with the tests will be challenging, if not impossible.
> What I did a while ago was to introduce the animal-sniffer plugin. This very strictly checks the resulting class files against the signatures for a given JDK to ensure the classes would work in that particular JDK. Would this be enough for you? It’s what we settled for in the Flex project. In the past we did have to re-release stuff because even if we compiled to output 1.7 code, the code wasn’t runnable on 1.7. Since introducing the animal-sniffer plugin we never had problems like this again.
>
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Thanks for all the progress!
It’s a little unsettling to not be able to run the regression unit/functional tests w/1.7 :-)
If the local mvn repo has the up-to-date built 1.7 compat Edgent jars and the local workspace has the up-to-date built 1.7 compat test class files, why doesn’t running “mvn test -Pjava7” w/1.7 just reuse them?
— Dale
> On Jun 12, 2017, at 7:35 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Ok … so I’m working on the open issues … I do have one question though …
> Compiling with one java version and then running a different one with the tests will be challenging, if not impossible.
> What I did a while ago was to introduce the animal-sniffer plugin. This very strictly checks the resulting class files against the signatures for a given JDK to ensure the classes would work in that particular JDK. Would this be enough for you? It’s what we settled for in the Flex project. In the past we did have to re-release stuff because even if we compiled to output 1.7 code, the code wasn’t runnable on 1.7. Since introducing the animal-sniffer plugin we never had problems like this again.
>
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Ok … so I’m working on the open issues … I do have one question though …
Compiling with one java version and then running a different one with the tests will be challenging, if not impossible.
What I did a while ago was to introduce the animal-sniffer plugin. This very strictly checks the resulting class files against the signatures for a given JDK to ensure the classes would work in that particular JDK. Would this be enough for you? It’s what we settled for in the Flex project. In the past we did have to re-release stuff because even if we compiled to output 1.7 code, the code wasn’t runnable on 1.7. Since introducing the animal-sniffer plugin we never had problems like this again.
Chris
Am 10.06.17, 22:45 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Dale,
Thanks for that … I had noticed the merged pull request … today I pulled in the upstream changes and tweaked the build a little.
Now both the java8 and java7 builds seem to be working. I also made the build compile to different target sub-directories so now you can inspect the output from both runs without having them interfering.
Chris
Am 09.06.17, 22:29 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Hi Chris, in case you hadn’t noticed the following merged PR (on master) eliminates the
use of “default”. So you should be able to rebase your branch your branch and be all set.
https://github.com/apache/incubator-edgent/pull/310 <https://github.com/apache/incubator-edgent/pull/310>
> On Jun 8, 2017, at 10:40 AM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...problem. The default methods in DirectTestSetup can’t seem to be backported if the interface and the implementing class are not in the same retrolambda execution. Eventually refactoring the tests to work without default methods would be a good idea.
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Dale,
Thanks for that … I had noticed the merged pull request … today I pulled in the upstream changes and tweaked the build a little.
Now both the java8 and java7 builds seem to be working. I also made the build compile to different target sub-directories so now you can inspect the output from both runs without having them interfering.
Chris
Am 09.06.17, 22:29 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Hi Chris, in case you hadn’t noticed the following merged PR (on master) eliminates the
use of “default”. So you should be able to rebase your branch your branch and be all set.
https://github.com/apache/incubator-edgent/pull/310 <https://github.com/apache/incubator-edgent/pull/310>
> On Jun 8, 2017, at 10:40 AM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...problem. The default methods in DirectTestSetup can’t seem to be backported if the interface and the implementing class are not in the same retrolambda execution. Eventually refactoring the tests to work without default methods would be a good idea.
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Hi Chris, in case you hadn’t noticed the following merged PR (on master) eliminates the
use of “default”. So you should be able to rebase your branch your branch and be all set.
https://github.com/apache/incubator-edgent/pull/310 <https://github.com/apache/incubator-edgent/pull/310>
> On Jun 8, 2017, at 10:40 AM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...problem. The default methods in DirectTestSetup can’t seem to be backported if the interface and the implementing class are not in the same retrolambda execution. Eventually refactoring the tests to work without default methods would be a good idea.
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Dale,
Today I had some progress on the java7 side.
I successfully added the retrolambda-maven-plugin in a dedicated java7 profile to the main pom. Now the plugin automatically kicks in and converts the classes after compiling them. I also added the animal-sniffer-maven-plugin to explicitly check against the signatures of 1.7 or 1.8 JDKs. This all seems to be working. Unfortunately, the backporting of default methods in interfaces seems to be a problem. The default methods in DirectTestSetup can’t seem to be backported if the interface and the implementing class are not in the same retrolambda execution. Eventually refactoring the tests to work without default methods would be a good idea.
The problem with classifiers is that maven automatically resolves artifacts. Depending on the scope different artifacts are pulled in. In general, we would be using the “compile” scope. So, if I add a dependency with classifier “java7” then it would pull in that jar in the java7 version, but the transitive dependencies would probably be the java8 ones, so you would have to explicitly define all the transitive dependencies with the classifier. We don’t want that ;-)
I also contacted the maven guys on their mailing list. They did have the one or the other solution. In general, they would all come down to adding the type to the version. So, we would start producing “1.2.0.java7” and “1.2.0.java8” or “1.2.0.7” and “1.2.0.8” versions (I would prefer the first as it doesn’t suggest the java 8 version being slightly newer than the 7 version). Another alternative would be to prefix the version with the target version. “7.1.2.0” and “8.1.2.0” … but I don’t really like that option. In this case, I would have to refactor the build to use variables in all modules except the parent poms (you can’t use variables for parent poms). Then travis/Jenkins would run two build:
mvn clean deploy
mvn clean deploy –Pjava7
In the end, it would result in both versions being built, tested and deployed, but in the end the java7 version would remain in the workspace as the second executions “clean” removed any reference to the java8 version.
Hope that explains the dilemma a little more. At least with these version-suffixes we would have a fully working solution. I’ll get into changing this tomorrow.
By the way … anyone got an idea how I can make Travis not die on out of memory? Tried several things, but it didn’t seem to do the job.
Chris
Am 08.06.17, 16:13 schrieb "Dale LaBossiere" <dm...@gmail.com>:
> On Jun 8, 2017, at 4:19 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Regarding java8 and java7 artifacts …
Coincidentally, I started investigating this space yesterday and retrolambda-maven-plugin
and was just composing mail on the question of publishing java8 and java7 to mvn repos :-)
[ fwiw, so far I had defined a Profile (e.g., mvn install -Ptgt=java7) and
changed the “target” build directory to “target/${edgent.target.kind}”
(e.g., target/{java8,java7}). Things were working fine up to that point:
generated class files and module jars. ]
> This could be problematic, if we wanted to publish both.
> ...
> I could offer to generally build java7 versions and to test them with java7 and java8 … I think this topic should be discussed a little more before I start implementing this.
>
> Right now, I would opt for distributing java8 binaries to Maven Central and to offer downloadable java7 binary distributions.
Ugh. Publishing java8 in a mvn repo but java7/android as a binary tgz seems really bad.
It creates two different ways (and needed doc) for users building an Edgent app
and installing Edgent jars onto a “device”.
Could you elaborate on the problems using the maven coordinate “classifier”?
The maven doc [1] has this exactly as use case for it (they use “jdk15” vs “jdk14”).
At a high (ignorant) level I don’t have a problem with the classifier being nil for java8 :-)
If using a coordinate classifier is a no-go, what about (grimace) incorporating it into the
groupId: org.apache.edgent, org.apache.edgent.java7 ?
A bit related, what coordinate component is “incubating” supposed to be incorporated into?
— Dale
[1] https://maven.apache.org/pom.html <https://maven.apache.org/pom.html>
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
> On Jun 8, 2017, at 4:19 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Regarding java8 and java7 artifacts …
Coincidentally, I started investigating this space yesterday and retrolambda-maven-plugin
and was just composing mail on the question of publishing java8 and java7 to mvn repos :-)
[ fwiw, so far I had defined a Profile (e.g., mvn install -Ptgt=java7) and
changed the “target” build directory to “target/${edgent.target.kind}”
(e.g., target/{java8,java7}). Things were working fine up to that point:
generated class files and module jars. ]
> This could be problematic, if we wanted to publish both.
> ...
> I could offer to generally build java7 versions and to test them with java7 and java8 … I think this topic should be discussed a little more before I start implementing this.
>
> Right now, I would opt for distributing java8 binaries to Maven Central and to offer downloadable java7 binary distributions.
Ugh. Publishing java8 in a mvn repo but java7/android as a binary tgz seems really bad.
It creates two different ways (and needed doc) for users building an Edgent app
and installing Edgent jars onto a “device”.
Could you elaborate on the problems using the maven coordinate “classifier”?
The maven doc [1] has this exactly as use case for it (they use “jdk15” vs “jdk14”).
At a high (ignorant) level I don’t have a problem with the classifier being nil for java8 :-)
If using a coordinate classifier is a no-go, what about (grimace) incorporating it into the
groupId: org.apache.edgent, org.apache.edgent.java7 ?
A bit related, what coordinate component is “incubating” supposed to be incorporated into?
— Dale
[1] https://maven.apache.org/pom.html <https://maven.apache.org/pom.html>
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Regarding java8 and java7 artifacts …
This could be problematic, if we wanted to publish both.
Having a binary tar.gz with a java8 and one with java7 doesn’t seem to be a problem, but having two artifacts deployed to Maven-Central is no trivial task.
Exactly this type of jars for different java versions is one of the things the Maven guys are currently working on as part of the Java 9 – Jigsaw support. But it’s a nightmare. We could start creating java8 jars as default and add a second “java7” jar using the “java7” classifier, but then a lot of the mechanisms in Maven no longer work.
I could offer to generally build java7 versions and to test them with java7 and java8 … I think this topic should be discussed a little more before I start implementing this.
Right now, I would opt for distributing java8 binaries to Maven Central and to offer downloadable java7 binary distributions.
Chris
Am 07.06.17, 21:06 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi,
Right now I didn't configure anything regarding the assembly of the source artifact. It's apaches default config. I'll look into a more fine-tuned version.
And regarding the artifact name ... Justin suggested that change, but 90% of the other projects don't do it that way. I was waiting for your opinions before doing that. It does make the dependencies a lot more verbose. After all you can't use the artifacts without the groupId "org.apache.effect" which includes "apache" and "edgent".
But sure, I'll change that if you all want me to.
Chris
Von meinem Samsung Galaxy Smartphone gesendet.
-------- Ursprüngliche Nachricht --------
Von: Dale LaBossiere <dm...@gmail.com>
Datum: 07.06.17 18:45 (GMT+01:00)
An: dev@edgent.apache.org
Betreff: Re: Understanding the snapshot and release process
OK, so mvn install -Papache-release generates the source release bundle zip.
I poked around for a while but couldn’t come up with the incantation
needed to start excluding things from the zip. e.g., the zip includes
all of the residual gradle generated **/build/
Chris, can you enhance the configs so as to exclude the various
things that the gradle build was configured to exclude?
See top level build.gradle/srcReleaseTarGz task.
Also exclude src/site (that present in the zip too), right?
The generated zip’s name is:
target/edgent-parent-1.2.0-SNAPSHOT-source-release.zip
“edgent-parent” ? today we, I thought correctly, have “apache-edgent” there.
Also, the required “incubating” is missing in the release bundle name.
Thanks,
— Dale
> On Jun 7, 2017, at 10:35 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Checking our repo:
> https://repository.apache.org/#view-repositories;releases~browsestorage
>
> I can see that fewer than about 10% of all apache projects do that … But I wouldn’t mind adding that, if the others agree.
>
> Chris
>
>
>
> Am 07.06.17, 14:54 schrieb "Justin Mclean" <ju...@classsoftware.com>:
>
> Hi,
>
> While not a requirement prefixing with apache may add some legal protection and is good from a branding point of of view.
>
> Thanks,
> Justin
>
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi,
Right now I didn't configure anything regarding the assembly of the source artifact. It's apaches default config. I'll look into a more fine-tuned version.
And regarding the artifact name ... Justin suggested that change, but 90% of the other projects don't do it that way. I was waiting for your opinions before doing that. It does make the dependencies a lot more verbose. After all you can't use the artifacts without the groupId "org.apache.effect" which includes "apache" and "edgent".
But sure, I'll change that if you all want me to.
Chris
Von meinem Samsung Galaxy Smartphone gesendet.
-------- Ursprüngliche Nachricht --------
Von: Dale LaBossiere <dm...@gmail.com>
Datum: 07.06.17 18:45 (GMT+01:00)
An: dev@edgent.apache.org
Betreff: Re: Understanding the snapshot and release process
OK, so mvn install -Papache-release generates the source release bundle zip.
I poked around for a while but couldn’t come up with the incantation
needed to start excluding things from the zip. e.g., the zip includes
all of the residual gradle generated **/build/
Chris, can you enhance the configs so as to exclude the various
things that the gradle build was configured to exclude?
See top level build.gradle/srcReleaseTarGz task.
Also exclude src/site (that present in the zip too), right?
The generated zip’s name is:
target/edgent-parent-1.2.0-SNAPSHOT-source-release.zip
“edgent-parent” ? today we, I thought correctly, have “apache-edgent” there.
Also, the required “incubating” is missing in the release bundle name.
Thanks,
— Dale
> On Jun 7, 2017, at 10:35 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Checking our repo:
> https://repository.apache.org/#view-repositories;releases~browsestorage
>
> I can see that fewer than about 10% of all apache projects do that … But I wouldn’t mind adding that, if the others agree.
>
> Chris
>
>
>
> Am 07.06.17, 14:54 schrieb "Justin Mclean" <ju...@classsoftware.com>:
>
> Hi,
>
> While not a requirement prefixing with apache may add some legal protection and is good from a branding point of of view.
>
> Thanks,
> Justin
>
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
OK, so mvn install -Papache-release generates the source release bundle zip.
I poked around for a while but couldn’t come up with the incantation
needed to start excluding things from the zip. e.g., the zip includes
all of the residual gradle generated **/build/
Chris, can you enhance the configs so as to exclude the various
things that the gradle build was configured to exclude?
See top level build.gradle/srcReleaseTarGz task.
Also exclude src/site (that present in the zip too), right?
The generated zip’s name is:
target/edgent-parent-1.2.0-SNAPSHOT-source-release.zip
“edgent-parent” ? today we, I thought correctly, have “apache-edgent” there.
Also, the required “incubating” is missing in the release bundle name.
Thanks,
— Dale
> On Jun 7, 2017, at 10:35 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Checking our repo:
> https://repository.apache.org/#view-repositories;releases~browsestorage
>
> I can see that fewer than about 10% of all apache projects do that … But I wouldn’t mind adding that, if the others agree.
>
> Chris
>
>
>
> Am 07.06.17, 14:54 schrieb "Justin Mclean" <ju...@classsoftware.com>:
>
> Hi,
>
> While not a requirement prefixing with apache may add some legal protection and is good from a branding point of of view.
>
> Thanks,
> Justin
>
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Checking our repo:
https://repository.apache.org/#view-repositories;releases~browsestorage
I can see that fewer than about 10% of all apache projects do that … But I wouldn’t mind adding that, if the others agree.
Chris
Am 07.06.17, 14:54 schrieb "Justin Mclean" <ju...@classsoftware.com>:
Hi,
While not a requirement prefixing with apache may add some legal protection and is good from a branding point of of view.
Thanks,
Justin
Re: Understanding the snapshot and release process
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
While not a requirement prefixing with apache may add some legal protection and is good from a branding point of of view.
Thanks,
Justin
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Well looking through the Apache repo there is no 100% standard, but most artifacts are indeed prefixed with the project name but not apache … I’ll add the edgent to each of the artifacts.
Chris
Am 07.06.17, 13:47 schrieb "John D. Ament" <jo...@apache.org>:
Usually the maven norm is something like
com.mycompany.myproduct:myproduct-some-module.
So I would expect to see groupIds of org.apache.edgent, and artifact id's
of edgent-some-module. However, I've noticed that both Eclipse and IBM
(obviously not related in any way) seem to use completely different
artifact formats. I would recommend being an Apache project you adopt the
Apache Maven standard format.
John
On Wed, Jun 7, 2017 at 7:44 AM Christofer Dutz <ch...@c-ware.de>
wrote:
> Hi,
>
> I’ll answer this one quickly as I can do that without digging in too deep.
>
> I wouldn’t add the apache-edgent to the artifact names as the groupId
> already qualifies them as apache edgent modules.
> However when creating a release zip/tar.gz with the sources or binary
> distribution I would add the “apache-edgent-” prefix.
>
> Chris
>
>
> Am 06.06.17, 23:02 schrieb "Dale LaBossiere" <dm...@gmail.com>:
>
> Currently the generated JAR/WAR lack an “edgent” or “apache-edgent”
> prefix, e.g., "api-topology-1.2.0-SNAPSHOT.jar". How important/required is
> it to add that before we start publishing these to mvn repos?
>
> What’s the current ASF best-practice in this regard? e.g., I some
> (limited) # of “apache-XYZZY-…” in maven central. Presumably there are
> many many others that omit the “apache-" - e.g., “beam-sdks-…”
>
> — Dale
>
>
>
>
Re: Understanding the snapshot and release process
Posted by "John D. Ament" <jo...@apache.org>.
Usually the maven norm is something like
com.mycompany.myproduct:myproduct-some-module.
So I would expect to see groupIds of org.apache.edgent, and artifact id's
of edgent-some-module. However, I've noticed that both Eclipse and IBM
(obviously not related in any way) seem to use completely different
artifact formats. I would recommend being an Apache project you adopt the
Apache Maven standard format.
John
On Wed, Jun 7, 2017 at 7:44 AM Christofer Dutz <ch...@c-ware.de>
wrote:
> Hi,
>
> I’ll answer this one quickly as I can do that without digging in too deep.
>
> I wouldn’t add the apache-edgent to the artifact names as the groupId
> already qualifies them as apache edgent modules.
> However when creating a release zip/tar.gz with the sources or binary
> distribution I would add the “apache-edgent-” prefix.
>
> Chris
>
>
> Am 06.06.17, 23:02 schrieb "Dale LaBossiere" <dm...@gmail.com>:
>
> Currently the generated JAR/WAR lack an “edgent” or “apache-edgent”
> prefix, e.g., "api-topology-1.2.0-SNAPSHOT.jar". How important/required is
> it to add that before we start publishing these to mvn repos?
>
> What’s the current ASF best-practice in this regard? e.g., I some
> (limited) # of “apache-XYZZY-…” in maven central. Presumably there are
> many many others that omit the “apache-" - e.g., “beam-sdks-…”
>
> — Dale
>
>
>
>
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi,
I’ll answer this one quickly as I can do that without digging in too deep.
I wouldn’t add the apache-edgent to the artifact names as the groupId already qualifies them as apache edgent modules.
However when creating a release zip/tar.gz with the sources or binary distribution I would add the “apache-edgent-” prefix.
Chris
Am 06.06.17, 23:02 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Currently the generated JAR/WAR lack an “edgent” or “apache-edgent” prefix, e.g., "api-topology-1.2.0-SNAPSHOT.jar". How important/required is it to add that before we start publishing these to mvn repos?
What’s the current ASF best-practice in this regard? e.g., I some (limited) # of “apache-XYZZY-…” in maven central. Presumably there are many many others that omit the “apache-" - e.g., “beam-sdks-…”
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Currently the generated JAR/WAR lack an “edgent” or “apache-edgent” prefix, e.g., "api-topology-1.2.0-SNAPSHOT.jar". How important/required is it to add that before we start publishing these to mvn repos?
What’s the current ASF best-practice in this regard? e.g., I some (limited) # of “apache-XYZZY-…” in maven central. Presumably there are many many others that omit the “apache-" - e.g., “beam-sdks-…”
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Hi Chris,
I was trying the different forms of test execution noted in [1].
I’ve been mucking with my (Eclipse) .classpath files but that shouldn’t affect the following :-)
I tried running just the jdbc connector tests.
gradlew connectors:jdbc:test # does everything needed to run the “test” task for just this component
mvn -pl components-jdbc test # from [1]
[ERROR] Could not find the selected project in the reactor: connectors-jdbc @
…
that was when run in the top level directory.
also tried it after ‘cd connectors/jdbc’ and got the same results.
What am I doing wrong?
— Dale
[1] https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle
Re: Maven wiki page [was Re: Understanding the snapshot and release
process]
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Queeniw,
thanks … I added most of the things in the upper section.
For the questions in the lower one, I would like to ask you to watch the little maven introduction video I did. It should answer a lot of questions. Questions that my answers would probably sound confusing without this minimal Maven background.
Chris
Am 05.06.17, 20:39 schrieb "Queenie Ma" <qu...@apache.org>:
Hi Chris, I just granted you permissions to make changes to the wiki.
—
Queenie
> On Monday, Jun 05, 2017 at 11:37 AM, Christofer Dutz <christofer.dutz@c-ware.de (mailto:christofer.dutz@c-ware.de)> wrote:
> I’d love to, but you would need to give me the charma to do so (
>
> Chris
>
> Am 05.06.17, 20:14 schrieb "Dale LaBossiere" <dl...@apache.org>:
>
> I created a wiki page [1] to help us work through / collaborate on this.
> IMO there are too many questions, etc to effectively deal with it just on the dev@ list
>
> Chris, maybe you could fill in some of the (red) blanks on that page while continue my maven education.
>
> [1] https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle <https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle>
>
> — Dale
>
>
>
Re: Maven wiki page [was Re: Understanding the snapshot and
release process]
Posted by Queenie Ma <qu...@apache.org>.
Hi Chris, I just granted you permissions to make changes to the wiki.
—
Queenie
> On Monday, Jun 05, 2017 at 11:37 AM, Christofer Dutz <christofer.dutz@c-ware.de (mailto:christofer.dutz@c-ware.de)> wrote:
> I’d love to, but you would need to give me the charma to do so (
>
> Chris
>
> Am 05.06.17, 20:14 schrieb "Dale LaBossiere" <dl...@apache.org>:
>
> I created a wiki page [1] to help us work through / collaborate on this.
> IMO there are too many questions, etc to effectively deal with it just on the dev@ list
>
> Chris, maybe you could fill in some of the (red) blanks on that page while continue my maven education.
>
> [1] https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle <https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle>
>
> — Dale
>
>
>
Re: Maven wiki page [was Re: Understanding the snapshot and release
process]
Posted by Christofer Dutz <ch...@c-ware.de>.
I’d love to, but you would need to give me the charma to do so (
Chris
Am 05.06.17, 20:14 schrieb "Dale LaBossiere" <dl...@apache.org>:
I created a wiki page [1] to help us work through / collaborate on this.
IMO there are too many questions, etc to effectively deal with it just on the dev@ list
Chris, maybe you could fill in some of the (red) blanks on that page while continue my maven education.
[1] https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle <https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle>
— Dale
Maven wiki page [was Re: Understanding the snapshot and release
process]
Posted by Dale LaBossiere <dl...@apache.org>.
I created a wiki page [1] to help us work through / collaborate on this.
IMO there are too many questions, etc to effectively deal with it just on the dev@ list
Chris, maybe you could fill in some of the (red) blanks on that page while continue my maven education.
[1] https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle <https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle>
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi,
So, I just added some changes to get the site generation working. To have a look, just run
mvn clean install site:site site:stage
After the build just go to the target/staging directory and open the index.html in any browser. There you can see the generated site for the project.
Chris
Am 02.06.17, 20:59 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Thanks, I’ve pulled it into my workspace and started to explore.
— Dale
> On Jun 2, 2017, at 11:26 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Hi,
>
> So, I just create a PR as I think I am currently finished with everything I wanted to fix.
>
> Edgent should now be buildable with Maven command. At least I managed to get the travis build working with maven ☺
>
> Here some structural changes I did. I refactored the project and extracted the TestApplications.java together with its service descriptor into a dedicated module located in the test submodule. Wherever a jar was referenced for dynamically loading, now I don’t reference the jar where it’s created during the build, I use the maven-dependency plugin to copy it to a location local to the current module and simply load it from there. This way I didn’t have to deal with any too complicated classpath lookups. I could not only run the tests as part of the Maven build, but also directly inside my IDE (IntellIJ).
>
> Now some helpful maven commands:
>
> Simply build the project running all unit tests:
>
> mvn clean install
>
> Deploy SNAPSHOTS in Apaches Maven repository:
>
> mvn clean deploy
>
> Additionally, build and deploy source and javadoc artifacts (all signed and require correct pgp setup on the client side):
>
> mvn clean deploy -Papache-release
>
> Build without running tests:
>
> mvn clean install -DskipTests
>
> The site generation still needs some love as it would currently run all test twice.
>
> Please have a look at this.
>
> Chris
>
>
>
>
>
> Am 31.05.17, 18:41 schrieb "Dale LaBossiere" <dm...@gmail.com>:
>
> Christofer, can you create a [WIP] PR for this work? Then I’ll have a context from which I can review / ask stupid nubie questions :-)
>
> Also, what’s the executive summary for running maven to build a release (or at least all of the jars) using this work? And to run the tests? I’d like to pull the changes into my workspace and really see what’s generated, etc
>
> Thanks!
> — Dale
>
>> On May 28, 2017, at 10:03 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>>
>> had some time to finish the maven migration … I just pushed the changes to the “feature/maven” branch of my fork on github: https://github.com/chrisdutz/incubator-edgent/tree/feature/maven <https://github.com/chrisdutz/incubator-edgent/tree/feature/maven>
>
>
>
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Thanks, I’ve pulled it into my workspace and started to explore.
— Dale
> On Jun 2, 2017, at 11:26 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Hi,
>
> So, I just create a PR as I think I am currently finished with everything I wanted to fix.
>
> Edgent should now be buildable with Maven command. At least I managed to get the travis build working with maven ☺
>
> Here some structural changes I did. I refactored the project and extracted the TestApplications.java together with its service descriptor into a dedicated module located in the test submodule. Wherever a jar was referenced for dynamically loading, now I don’t reference the jar where it’s created during the build, I use the maven-dependency plugin to copy it to a location local to the current module and simply load it from there. This way I didn’t have to deal with any too complicated classpath lookups. I could not only run the tests as part of the Maven build, but also directly inside my IDE (IntellIJ).
>
> Now some helpful maven commands:
>
> Simply build the project running all unit tests:
>
> mvn clean install
>
> Deploy SNAPSHOTS in Apaches Maven repository:
>
> mvn clean deploy
>
> Additionally, build and deploy source and javadoc artifacts (all signed and require correct pgp setup on the client side):
>
> mvn clean deploy -Papache-release
>
> Build without running tests:
>
> mvn clean install -DskipTests
>
> The site generation still needs some love as it would currently run all test twice.
>
> Please have a look at this.
>
> Chris
>
>
>
>
>
> Am 31.05.17, 18:41 schrieb "Dale LaBossiere" <dm...@gmail.com>:
>
> Christofer, can you create a [WIP] PR for this work? Then I’ll have a context from which I can review / ask stupid nubie questions :-)
>
> Also, what’s the executive summary for running maven to build a release (or at least all of the jars) using this work? And to run the tests? I’d like to pull the changes into my workspace and really see what’s generated, etc
>
> Thanks!
> — Dale
>
>> On May 28, 2017, at 10:03 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>>
>> had some time to finish the maven migration … I just pushed the changes to the “feature/maven” branch of my fork on github: https://github.com/chrisdutz/incubator-edgent/tree/feature/maven <https://github.com/chrisdutz/incubator-edgent/tree/feature/maven>
>
>
>
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi,
So, I just create a PR as I think I am currently finished with everything I wanted to fix.
Edgent should now be buildable with Maven command. At least I managed to get the travis build working with maven ☺
Here some structural changes I did. I refactored the project and extracted the TestApplications.java together with its service descriptor into a dedicated module located in the test submodule. Wherever a jar was referenced for dynamically loading, now I don’t reference the jar where it’s created during the build, I use the maven-dependency plugin to copy it to a location local to the current module and simply load it from there. This way I didn’t have to deal with any too complicated classpath lookups. I could not only run the tests as part of the Maven build, but also directly inside my IDE (IntellIJ).
Now some helpful maven commands:
Simply build the project running all unit tests:
mvn clean install
Deploy SNAPSHOTS in Apaches Maven repository:
mvn clean deploy
Additionally, build and deploy source and javadoc artifacts (all signed and require correct pgp setup on the client side):
mvn clean deploy -Papache-release
Build without running tests:
mvn clean install -DskipTests
The site generation still needs some love as it would currently run all test twice.
Please have a look at this.
Chris
Am 31.05.17, 18:41 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Christofer, can you create a [WIP] PR for this work? Then I’ll have a context from which I can review / ask stupid nubie questions :-)
Also, what’s the executive summary for running maven to build a release (or at least all of the jars) using this work? And to run the tests? I’d like to pull the changes into my workspace and really see what’s generated, etc
Thanks!
— Dale
> On May 28, 2017, at 10:03 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> had some time to finish the maven migration … I just pushed the changes to the “feature/maven” branch of my fork on github: https://github.com/chrisdutz/incubator-edgent/tree/feature/maven <https://github.com/chrisdutz/incubator-edgent/tree/feature/maven>
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Christofer, can you create a [WIP] PR for this work? Then I’ll have a context from which I can review / ask stupid nubie questions :-)
Also, what’s the executive summary for running maven to build a release (or at least all of the jars) using this work? And to run the tests? I’d like to pull the changes into my workspace and really see what’s generated, etc
Thanks!
— Dale
> On May 28, 2017, at 10:03 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> had some time to finish the maven migration … I just pushed the changes to the “feature/maven” branch of my fork on github: https://github.com/chrisdutz/incubator-edgent/tree/feature/maven <https://github.com/chrisdutz/incubator-edgent/tree/feature/maven>
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Sorry for being silent the last couple of days - vacation and US holiday.
I’ll try to review all of the info / great work soon.
Thanks!
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Well the big advantage of running the jobs on ASF Jenkins is that you can auto-deploy to Nexus (ASF Maven Repo) as well as commit to Git without having to store some ones credentials on an external machine. Infra is very helpful in getting even complicated setups running (I manage the Windows based flex builds, which need Flashplayer, Browsers and Selenium). I would be more than happy to help setup everything.
Chris
Am 28.05.17, 17:26 schrieb "John D. Ament" <jo...@apache.org>:
On Sun, May 28, 2017 at 11:12 AM Christofer Dutz <ch...@c-ware.de>
wrote:
> The Maven build should work on the ASF Jenkins and hereby publish the
> snapshots. I think that in the flex project I added this to ensure its
> added to the generated "site".
>
> Which one is the hosted Jenkins you are referring to? The asf Jenkins?
>
>
Yes, the ASF jenkins.
> And what should be the problem with Travis? Why are you using two ci
> servers?
>
AFAIK, Edgent only uses travis, and doesn't currently run jobs on ASF
Jenkins. At least if they do, I was unable to find a job with a name that
was easy to identify.
> Chris
>
>
>
> Von meinem Samsung Galaxy Smartphone gesendet.
>
>
> -------- Ursprüngliche Nachricht --------
> Von: "John D. Ament" <jo...@apache.org>
> Datum: 28.05.17 16:20 (GMT+01:00)
> An: dev@edgent.apache.org
> Betreff: Re: Understanding the snapshot and release process
>
> The maven based build seems to be much more maintainable. I'm glad you
> guys came together to solve this problem :-)
>
> RE distribution of snapshots. We only presently support publishing via our
> hosted jenkins, however Edgent currently builds in Travis. So that would
> need to be changed, or some other solution.
>
> You also can drop the repositories/pluginRepositories sections of the new
> pom. Its not needed.
>
>
> John
>
> On Sun, May 28, 2017 at 10:03 AM Christofer Dutz <
> christofer.dutz@c-ware.de>
> wrote:
>
> > Hi guys,
> >
> > Strange people these Germans … if it rains, they complain about it and
> how
> > much they would like to go swimming in the sun … now that it’s 32° and
> the
> > sun is shining, no one wants to go swimming, cause it’s too fu***g hot
> ;-)
> > … well this way I had some time to finish the maven migration … I just
> > pushed the changes to the “feature/maven” branch of my fork on github:
> > https://github.com/chrisdutz/incubator-edgent/tree/feature/maven
> >
> > Here a short list on what little monsters I found siting under some
> rocks:
> >
> > 1) Usage of test-jars --> Bad practice → Move code to separate test-util
> > jar module
> > 2) AppServiceTest in provides/direct relies on system-properties to load
> > jar produced by another module → Ideally this should be refactored to
> find
> > the jar on the classpath
> > 3) HttpServerTest in console/server relies on console.war
> > 4) console/servlets module compiled to something with a name completely
> > unrelated to the project
> > 5) test/svt requires samples project → Some modules rely on samples.
> While
> > I think, this is ok for Samples themselves, nothing outside the samples
> > should rely on samples (my opinion)
> > 6) Almost all tests relying on successful SSL handshakes in
> > wsclient-javax.websocket are failing (I’ll investigate this as soon as I
> > have time, I could imagine this to be related to non US JVM encryption
> > issues we are also seeing in the Flex project)
> > 7) Some unit-tests should more be integration-tests
> >
> > I didn’t migrate the stuff in “platform” as I didn’t quite understand
> what
> > they are used for and what they should do.
> >
> > Hope you like the changes. With a build like this it should be easy to
> > setup the CI to auto distribute SNAPSHOTs and execute releases on ASF
> > machines with a simple one or two-liner.
> >
> > Chris
> >
> >
> > Am 27.05.17, 19:16 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de
> >:
> >
> > I did have some free time this weekend, so I thought I’d give the
> > Maven thing a try. And right now I haven finished, but I think I’m about
> > 80% done. So far there have been one or two little monsters under some
> > rocks … but they weren’t too nasty ;-)
> >
> > Regarding the classpath stuff in the manifest: I really dislike this
> > way of setting the classpath as it usually causes a lot of problems when
> > using it. It requires libs to be in a predefined directory structure and
> > follow a given naming convention. Try to use this in a project built with
> > Maven (just picking this as I saw this would be a problem for a Maven
> built
> > project)
> >
> > Regarding tests: I did see that there are several Unit tests I would
> > probably more call integration-tests. Especially the ones that require
> > loading of a war (console.war). So I guess you have both some classical
> > unit-tests as well as integration-tests … no need to treat them the same
> > way. Maven ususally has the unit-test and the integration-test phase for
> > exactly this.
> >
> > Regarding jacoco: So I’ll activate that in the maven build too
> >
> > Regarding automatic build: For Flex I setup the build to do a
> > multibranch pipeline build. So whenever someone checks something in
> > “develop”, “releae/*” or “feature/*” it automatically builds that
> > particular branch. However develop and “release/*” are the only ones
> > uploading SNAPSHOT versions to the ASF Nexus.
> >
> > Regarding my commitment: I think Edgent is the tool I was looking for
> > in order to make my SCADA project feature-perfect. So I have a strong
> > interest in the project maturing. I do have plans for a longer
> involvement
> > in the project. Probably I would be trying to create an open-source “plc”
> > connector (which would probably have to be developed outside of Apache
> due
> > to libraries still being GPL). But I do plan on staying around some time
> ;-)
> >
> > Regarding the usage: Maven or Gradle based application developers
> > could directly utilize SNAPSHOT and release artifacts as they are
> released.
> > Additionally, I would create a so-called assembly that produces the same
> > zip or tar-gz archives the users have been using. So there shouldn’t be
> too
> > much of a change.
> >
> > So now I’ll try to finish the rest of the connector modules maven
> > conversion (
> >
> > Chris
> >
> > Am 25.05.17, 19:37 schrieb "Dale LaBossiere" <dm...@gmail.com>:
> >
> > Awaiting others to chime in on this “switch to mvn” subject.
> > A “don’t care” (+/- 0) response is preferred over silence.
> >
> > In the mean time...
> >
> > > On May 24, 2017, at 11:41 AM, Christofer Dutz <
> > christofer.dutz@c-ware.de> wrote:
> > > ...
> > > I just checked out everything and managed to get things
> imported
> > in IntelliJ after a little struggle … this is a good job you did :-)
> > Can’t take all the credit (or blame :-)
> > >
> > > Right now, it seems as if there were Ant+Gradle+Eclipse build
> > files in there. While it might seem ...
> > gradle is the only CLI way to build edgent. Any ant build.xml
> > files are either (a) residual cruft that should be removed (my bad) or
> (b)
> > needed to leverage the ant machinery for invoking retrolambda in building
> > java7/android compatible versions of the jars (machinery that wasn’t
> > converted to pure gradle due to time/effort/value at the time).
> Attempting
> > to use ant at the top level tells you it doesn’t work :-) As for Eclipse
> > .project/.classpath, yup those are live and haven’t been a maintenance
> > issue.
> >
> > fwiw, I just removed all build.xml except the top level one and
> > those under platform and “gradle release” worked fine so I’ll create a PR
> > to clean them up as a first step.
> >
> > > ...JavaDoc is generated automatically when running a release
> > build together with the usual Maven project reports (is even configured
> in
> > the apache parent POM together with rat, deployment etc.)
> > Javadoc was complicated by creating groupings as well as
> excluding
> > it for non-API classes.
> >
> > > ...I didn’t quite understand the “Manifest” thing, but the jar
> > plugin does generate this with reasonable defaults, and can be extended
> to
> > also export the dependencies into that (even if I don’t recommend that).
> > An edgent jar’s manifest class-path includes references to its
> > immediate dependent edgent jars (not transitive) as well as references to
> > its external jar dependencies (transitively). I agree that "compiled in"
> > references to specific versions of external dependencies may not be a
> great
> > idea / is perhaps best eliminated.
> >
> > My recollection is that by default gradle did not generate any
> > manifest class-path.
> >
> > e.g., just to make this a bit more concrete, this is from
> > edgent.connectors.kafka.jar/MANIFEST.MF (see
> connectors/kafka/build.gradle
> > for more info)
> >
> > Class-Path: ../../../lib/edgent.api.topology.jar
> > ../../../ext/gson-2.2
> > .4.jar ../../../ext/slf4j-api-1.7.12.jar
> > ../../../ext/metrics-core-3.
> > 1.2.jar ../ext/kafka_2.10-0.8.2.2.jar
> > ../ext/kafka-clients-0.8.2.2.ja
> > r ../ext/log4j-1.2.16.jar ../ext/metrics-core-2.2.0.jar
> > ../ext/scala-
> > library-2.10.4.jar ../ext/zkclient-0.3.jar
> > ../ext/zookeeper-3.4.6.jar
> >
> > edgent.api.topology.jar itself has lots of edgent jar
> dependencies
> > (captured in its manifest) but that’s of no concern to the kafka
> > connector. I’m not sure maintaining this is a requirement and, at least
> > for gradle, eliminating it would have greatly simplified things.
> >
> > Test execution environment was also an issue. perhaps more a
> > result of structural issues. i.e., the gradle config treated a
> component’s
> > tests as unit tests, run against the associated component’s .class files.
> > The ant system treated them more like integration tests, run against the
> > jar files that would be bundled into a binary-release bundle; more like
> the
> > environment that Edgent-based applications would use. That was very
> > important. And due to some manual setup requirements for some connector
> > tests are excluded from execution by default.
> >
> > > …
> > > In the Flex project, I also setup the build to run SonarQube
> > analysis and automatically generate the documentation from markdown
> and/or
> > asciidoctor (which I think is very convenient) even automatically update
> > and deploy the project website.
> >
> > We include use of jacoco
> > We have travis integration for auto-PR validation.
> > It would be nice to have some periodic build/regression testing
> > run on the main (master) branch.
> >
> > > I could offer to create a fork on GitHub, create a feature
> > branch there and try to whip up a set of poms that add Maven as fourth
> > build system to the list … you could check it out and play around with
> it.
> > But I’d only do this, if there is any interest in it.
> >
> > I don’t blame you!
> >
> > One worry I have is that it ends up like the gradle effort… it
> was
> > ~trivial to get started but there was then a lot of effort required
> > identify and flesh out equivalence with the ant based build result
> > artifacts.
> > If you’re on board for the long haul in doing a conversion then
> > that lessens my concern.
> >
> > I’m on board to help but don’t want to inherit a “now finish it”
> > task :-)
> >
> > A full switch to a ~simple mvn-based build system would be the
> > goal IMO — we’d want to toss gradle. The effort ultimately also entails
> > updating a bunch of doc. Not a killer, just not to be overlooked.
> >
> > It’s unclear to me how the Edgent-based app developers will be
> > affected by all of this. OK, they build their Edgent-based app with some
> > tool that can utilize a maven repo. Then what?
> >
> > Today the story is simple: extract the binary-tgz (or a subset of
> > it) on the target (edge device), copy your app code jar/classes to the
> > target, set the CLASSPATH and go. (assume we also eliminate
> > building/distributing a binary-tgz)
> >
> > Can elaborate on that part of the story?
> >
> > Thanks!
> > — Dale
> >
> >
> >
> >
> >
> >
>
Re: Understanding the snapshot and release process
Posted by "John D. Ament" <jo...@apache.org>.
On Sun, May 28, 2017 at 11:12 AM Christofer Dutz <ch...@c-ware.de>
wrote:
> The Maven build should work on the ASF Jenkins and hereby publish the
> snapshots. I think that in the flex project I added this to ensure its
> added to the generated "site".
>
> Which one is the hosted Jenkins you are referring to? The asf Jenkins?
>
>
Yes, the ASF jenkins.
> And what should be the problem with Travis? Why are you using two ci
> servers?
>
AFAIK, Edgent only uses travis, and doesn't currently run jobs on ASF
Jenkins. At least if they do, I was unable to find a job with a name that
was easy to identify.
> Chris
>
>
>
> Von meinem Samsung Galaxy Smartphone gesendet.
>
>
> -------- Ursprüngliche Nachricht --------
> Von: "John D. Ament" <jo...@apache.org>
> Datum: 28.05.17 16:20 (GMT+01:00)
> An: dev@edgent.apache.org
> Betreff: Re: Understanding the snapshot and release process
>
> The maven based build seems to be much more maintainable. I'm glad you
> guys came together to solve this problem :-)
>
> RE distribution of snapshots. We only presently support publishing via our
> hosted jenkins, however Edgent currently builds in Travis. So that would
> need to be changed, or some other solution.
>
> You also can drop the repositories/pluginRepositories sections of the new
> pom. Its not needed.
>
>
> John
>
> On Sun, May 28, 2017 at 10:03 AM Christofer Dutz <
> christofer.dutz@c-ware.de>
> wrote:
>
> > Hi guys,
> >
> > Strange people these Germans … if it rains, they complain about it and
> how
> > much they would like to go swimming in the sun … now that it’s 32° and
> the
> > sun is shining, no one wants to go swimming, cause it’s too fu***g hot
> ;-)
> > … well this way I had some time to finish the maven migration … I just
> > pushed the changes to the “feature/maven” branch of my fork on github:
> > https://github.com/chrisdutz/incubator-edgent/tree/feature/maven
> >
> > Here a short list on what little monsters I found siting under some
> rocks:
> >
> > 1) Usage of test-jars --> Bad practice → Move code to separate test-util
> > jar module
> > 2) AppServiceTest in provides/direct relies on system-properties to load
> > jar produced by another module → Ideally this should be refactored to
> find
> > the jar on the classpath
> > 3) HttpServerTest in console/server relies on console.war
> > 4) console/servlets module compiled to something with a name completely
> > unrelated to the project
> > 5) test/svt requires samples project → Some modules rely on samples.
> While
> > I think, this is ok for Samples themselves, nothing outside the samples
> > should rely on samples (my opinion)
> > 6) Almost all tests relying on successful SSL handshakes in
> > wsclient-javax.websocket are failing (I’ll investigate this as soon as I
> > have time, I could imagine this to be related to non US JVM encryption
> > issues we are also seeing in the Flex project)
> > 7) Some unit-tests should more be integration-tests
> >
> > I didn’t migrate the stuff in “platform” as I didn’t quite understand
> what
> > they are used for and what they should do.
> >
> > Hope you like the changes. With a build like this it should be easy to
> > setup the CI to auto distribute SNAPSHOTs and execute releases on ASF
> > machines with a simple one or two-liner.
> >
> > Chris
> >
> >
> > Am 27.05.17, 19:16 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de
> >:
> >
> > I did have some free time this weekend, so I thought I’d give the
> > Maven thing a try. And right now I haven finished, but I think I’m about
> > 80% done. So far there have been one or two little monsters under some
> > rocks … but they weren’t too nasty ;-)
> >
> > Regarding the classpath stuff in the manifest: I really dislike this
> > way of setting the classpath as it usually causes a lot of problems when
> > using it. It requires libs to be in a predefined directory structure and
> > follow a given naming convention. Try to use this in a project built with
> > Maven (just picking this as I saw this would be a problem for a Maven
> built
> > project)
> >
> > Regarding tests: I did see that there are several Unit tests I would
> > probably more call integration-tests. Especially the ones that require
> > loading of a war (console.war). So I guess you have both some classical
> > unit-tests as well as integration-tests … no need to treat them the same
> > way. Maven ususally has the unit-test and the integration-test phase for
> > exactly this.
> >
> > Regarding jacoco: So I’ll activate that in the maven build too
> >
> > Regarding automatic build: For Flex I setup the build to do a
> > multibranch pipeline build. So whenever someone checks something in
> > “develop”, “releae/*” or “feature/*” it automatically builds that
> > particular branch. However develop and “release/*” are the only ones
> > uploading SNAPSHOT versions to the ASF Nexus.
> >
> > Regarding my commitment: I think Edgent is the tool I was looking for
> > in order to make my SCADA project feature-perfect. So I have a strong
> > interest in the project maturing. I do have plans for a longer
> involvement
> > in the project. Probably I would be trying to create an open-source “plc”
> > connector (which would probably have to be developed outside of Apache
> due
> > to libraries still being GPL). But I do plan on staying around some time
> ;-)
> >
> > Regarding the usage: Maven or Gradle based application developers
> > could directly utilize SNAPSHOT and release artifacts as they are
> released.
> > Additionally, I would create a so-called assembly that produces the same
> > zip or tar-gz archives the users have been using. So there shouldn’t be
> too
> > much of a change.
> >
> > So now I’ll try to finish the rest of the connector modules maven
> > conversion (
> >
> > Chris
> >
> > Am 25.05.17, 19:37 schrieb "Dale LaBossiere" <dm...@gmail.com>:
> >
> > Awaiting others to chime in on this “switch to mvn” subject.
> > A “don’t care” (+/- 0) response is preferred over silence.
> >
> > In the mean time...
> >
> > > On May 24, 2017, at 11:41 AM, Christofer Dutz <
> > christofer.dutz@c-ware.de> wrote:
> > > ...
> > > I just checked out everything and managed to get things
> imported
> > in IntelliJ after a little struggle … this is a good job you did :-)
> > Can’t take all the credit (or blame :-)
> > >
> > > Right now, it seems as if there were Ant+Gradle+Eclipse build
> > files in there. While it might seem ...
> > gradle is the only CLI way to build edgent. Any ant build.xml
> > files are either (a) residual cruft that should be removed (my bad) or
> (b)
> > needed to leverage the ant machinery for invoking retrolambda in building
> > java7/android compatible versions of the jars (machinery that wasn’t
> > converted to pure gradle due to time/effort/value at the time).
> Attempting
> > to use ant at the top level tells you it doesn’t work :-) As for Eclipse
> > .project/.classpath, yup those are live and haven’t been a maintenance
> > issue.
> >
> > fwiw, I just removed all build.xml except the top level one and
> > those under platform and “gradle release” worked fine so I’ll create a PR
> > to clean them up as a first step.
> >
> > > ...JavaDoc is generated automatically when running a release
> > build together with the usual Maven project reports (is even configured
> in
> > the apache parent POM together with rat, deployment etc.)
> > Javadoc was complicated by creating groupings as well as
> excluding
> > it for non-API classes.
> >
> > > ...I didn’t quite understand the “Manifest” thing, but the jar
> > plugin does generate this with reasonable defaults, and can be extended
> to
> > also export the dependencies into that (even if I don’t recommend that).
> > An edgent jar’s manifest class-path includes references to its
> > immediate dependent edgent jars (not transitive) as well as references to
> > its external jar dependencies (transitively). I agree that "compiled in"
> > references to specific versions of external dependencies may not be a
> great
> > idea / is perhaps best eliminated.
> >
> > My recollection is that by default gradle did not generate any
> > manifest class-path.
> >
> > e.g., just to make this a bit more concrete, this is from
> > edgent.connectors.kafka.jar/MANIFEST.MF (see
> connectors/kafka/build.gradle
> > for more info)
> >
> > Class-Path: ../../../lib/edgent.api.topology.jar
> > ../../../ext/gson-2.2
> > .4.jar ../../../ext/slf4j-api-1.7.12.jar
> > ../../../ext/metrics-core-3.
> > 1.2.jar ../ext/kafka_2.10-0.8.2.2.jar
> > ../ext/kafka-clients-0.8.2.2.ja
> > r ../ext/log4j-1.2.16.jar ../ext/metrics-core-2.2.0.jar
> > ../ext/scala-
> > library-2.10.4.jar ../ext/zkclient-0.3.jar
> > ../ext/zookeeper-3.4.6.jar
> >
> > edgent.api.topology.jar itself has lots of edgent jar
> dependencies
> > (captured in its manifest) but that’s of no concern to the kafka
> > connector. I’m not sure maintaining this is a requirement and, at least
> > for gradle, eliminating it would have greatly simplified things.
> >
> > Test execution environment was also an issue. perhaps more a
> > result of structural issues. i.e., the gradle config treated a
> component’s
> > tests as unit tests, run against the associated component’s .class files.
> > The ant system treated them more like integration tests, run against the
> > jar files that would be bundled into a binary-release bundle; more like
> the
> > environment that Edgent-based applications would use. That was very
> > important. And due to some manual setup requirements for some connector
> > tests are excluded from execution by default.
> >
> > > …
> > > In the Flex project, I also setup the build to run SonarQube
> > analysis and automatically generate the documentation from markdown
> and/or
> > asciidoctor (which I think is very convenient) even automatically update
> > and deploy the project website.
> >
> > We include use of jacoco
> > We have travis integration for auto-PR validation.
> > It would be nice to have some periodic build/regression testing
> > run on the main (master) branch.
> >
> > > I could offer to create a fork on GitHub, create a feature
> > branch there and try to whip up a set of poms that add Maven as fourth
> > build system to the list … you could check it out and play around with
> it.
> > But I’d only do this, if there is any interest in it.
> >
> > I don’t blame you!
> >
> > One worry I have is that it ends up like the gradle effort… it
> was
> > ~trivial to get started but there was then a lot of effort required
> > identify and flesh out equivalence with the ant based build result
> > artifacts.
> > If you’re on board for the long haul in doing a conversion then
> > that lessens my concern.
> >
> > I’m on board to help but don’t want to inherit a “now finish it”
> > task :-)
> >
> > A full switch to a ~simple mvn-based build system would be the
> > goal IMO — we’d want to toss gradle. The effort ultimately also entails
> > updating a bunch of doc. Not a killer, just not to be overlooked.
> >
> > It’s unclear to me how the Edgent-based app developers will be
> > affected by all of this. OK, they build their Edgent-based app with some
> > tool that can utilize a maven repo. Then what?
> >
> > Today the story is simple: extract the binary-tgz (or a subset of
> > it) on the target (edge device), copy your app code jar/classes to the
> > target, set the CLASSPATH and go. (assume we also eliminate
> > building/distributing a binary-tgz)
> >
> > Can elaborate on that part of the story?
> >
> > Thanks!
> > — Dale
> >
> >
> >
> >
> >
> >
>
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
The Maven build should work on the ASF Jenkins and hereby publish the snapshots. I think that in the flex project I added this to ensure its added to the generated "site".
Which one is the hosted Jenkins you are referring to? The asf Jenkins?
And what should be the problem with Travis? Why are you using two ci servers?
Chris
Von meinem Samsung Galaxy Smartphone gesendet.
-------- Ursprüngliche Nachricht --------
Von: "John D. Ament" <jo...@apache.org>
Datum: 28.05.17 16:20 (GMT+01:00)
An: dev@edgent.apache.org
Betreff: Re: Understanding the snapshot and release process
The maven based build seems to be much more maintainable. I'm glad you
guys came together to solve this problem :-)
RE distribution of snapshots. We only presently support publishing via our
hosted jenkins, however Edgent currently builds in Travis. So that would
need to be changed, or some other solution.
You also can drop the repositories/pluginRepositories sections of the new
pom. Its not needed.
John
On Sun, May 28, 2017 at 10:03 AM Christofer Dutz <ch...@c-ware.de>
wrote:
> Hi guys,
>
> Strange people these Germans … if it rains, they complain about it and how
> much they would like to go swimming in the sun … now that it’s 32° and the
> sun is shining, no one wants to go swimming, cause it’s too fu***g hot ;-)
> … well this way I had some time to finish the maven migration … I just
> pushed the changes to the “feature/maven” branch of my fork on github:
> https://github.com/chrisdutz/incubator-edgent/tree/feature/maven
>
> Here a short list on what little monsters I found siting under some rocks:
>
> 1) Usage of test-jars --> Bad practice → Move code to separate test-util
> jar module
> 2) AppServiceTest in provides/direct relies on system-properties to load
> jar produced by another module → Ideally this should be refactored to find
> the jar on the classpath
> 3) HttpServerTest in console/server relies on console.war
> 4) console/servlets module compiled to something with a name completely
> unrelated to the project
> 5) test/svt requires samples project → Some modules rely on samples. While
> I think, this is ok for Samples themselves, nothing outside the samples
> should rely on samples (my opinion)
> 6) Almost all tests relying on successful SSL handshakes in
> wsclient-javax.websocket are failing (I’ll investigate this as soon as I
> have time, I could imagine this to be related to non US JVM encryption
> issues we are also seeing in the Flex project)
> 7) Some unit-tests should more be integration-tests
>
> I didn’t migrate the stuff in “platform” as I didn’t quite understand what
> they are used for and what they should do.
>
> Hope you like the changes. With a build like this it should be easy to
> setup the CI to auto distribute SNAPSHOTs and execute releases on ASF
> machines with a simple one or two-liner.
>
> Chris
>
>
> Am 27.05.17, 19:16 schrieb "Christofer Dutz" <ch...@c-ware.de>:
>
> I did have some free time this weekend, so I thought I’d give the
> Maven thing a try. And right now I haven finished, but I think I’m about
> 80% done. So far there have been one or two little monsters under some
> rocks … but they weren’t too nasty ;-)
>
> Regarding the classpath stuff in the manifest: I really dislike this
> way of setting the classpath as it usually causes a lot of problems when
> using it. It requires libs to be in a predefined directory structure and
> follow a given naming convention. Try to use this in a project built with
> Maven (just picking this as I saw this would be a problem for a Maven built
> project)
>
> Regarding tests: I did see that there are several Unit tests I would
> probably more call integration-tests. Especially the ones that require
> loading of a war (console.war). So I guess you have both some classical
> unit-tests as well as integration-tests … no need to treat them the same
> way. Maven ususally has the unit-test and the integration-test phase for
> exactly this.
>
> Regarding jacoco: So I’ll activate that in the maven build too
>
> Regarding automatic build: For Flex I setup the build to do a
> multibranch pipeline build. So whenever someone checks something in
> “develop”, “releae/*” or “feature/*” it automatically builds that
> particular branch. However develop and “release/*” are the only ones
> uploading SNAPSHOT versions to the ASF Nexus.
>
> Regarding my commitment: I think Edgent is the tool I was looking for
> in order to make my SCADA project feature-perfect. So I have a strong
> interest in the project maturing. I do have plans for a longer involvement
> in the project. Probably I would be trying to create an open-source “plc”
> connector (which would probably have to be developed outside of Apache due
> to libraries still being GPL). But I do plan on staying around some time ;-)
>
> Regarding the usage: Maven or Gradle based application developers
> could directly utilize SNAPSHOT and release artifacts as they are released.
> Additionally, I would create a so-called assembly that produces the same
> zip or tar-gz archives the users have been using. So there shouldn’t be too
> much of a change.
>
> So now I’ll try to finish the rest of the connector modules maven
> conversion (
>
> Chris
>
> Am 25.05.17, 19:37 schrieb "Dale LaBossiere" <dm...@gmail.com>:
>
> Awaiting others to chime in on this “switch to mvn” subject.
> A “don’t care” (+/- 0) response is preferred over silence.
>
> In the mean time...
>
> > On May 24, 2017, at 11:41 AM, Christofer Dutz <
> christofer.dutz@c-ware.de> wrote:
> > ...
> > I just checked out everything and managed to get things imported
> in IntelliJ after a little struggle … this is a good job you did :-)
> Can’t take all the credit (or blame :-)
> >
> > Right now, it seems as if there were Ant+Gradle+Eclipse build
> files in there. While it might seem ...
> gradle is the only CLI way to build edgent. Any ant build.xml
> files are either (a) residual cruft that should be removed (my bad) or (b)
> needed to leverage the ant machinery for invoking retrolambda in building
> java7/android compatible versions of the jars (machinery that wasn’t
> converted to pure gradle due to time/effort/value at the time). Attempting
> to use ant at the top level tells you it doesn’t work :-) As for Eclipse
> .project/.classpath, yup those are live and haven’t been a maintenance
> issue.
>
> fwiw, I just removed all build.xml except the top level one and
> those under platform and “gradle release” worked fine so I’ll create a PR
> to clean them up as a first step.
>
> > ...JavaDoc is generated automatically when running a release
> build together with the usual Maven project reports (is even configured in
> the apache parent POM together with rat, deployment etc.)
> Javadoc was complicated by creating groupings as well as excluding
> it for non-API classes.
>
> > ...I didn’t quite understand the “Manifest” thing, but the jar
> plugin does generate this with reasonable defaults, and can be extended to
> also export the dependencies into that (even if I don’t recommend that).
> An edgent jar’s manifest class-path includes references to its
> immediate dependent edgent jars (not transitive) as well as references to
> its external jar dependencies (transitively). I agree that "compiled in"
> references to specific versions of external dependencies may not be a great
> idea / is perhaps best eliminated.
>
> My recollection is that by default gradle did not generate any
> manifest class-path.
>
> e.g., just to make this a bit more concrete, this is from
> edgent.connectors.kafka.jar/MANIFEST.MF (see connectors/kafka/build.gradle
> for more info)
>
> Class-Path: ../../../lib/edgent.api.topology.jar
> ../../../ext/gson-2.2
> .4.jar ../../../ext/slf4j-api-1.7.12.jar
> ../../../ext/metrics-core-3.
> 1.2.jar ../ext/kafka_2.10-0.8.2.2.jar
> ../ext/kafka-clients-0.8.2.2.ja
> r ../ext/log4j-1.2.16.jar ../ext/metrics-core-2.2.0.jar
> ../ext/scala-
> library-2.10.4.jar ../ext/zkclient-0.3.jar
> ../ext/zookeeper-3.4.6.jar
>
> edgent.api.topology.jar itself has lots of edgent jar dependencies
> (captured in its manifest) but that’s of no concern to the kafka
> connector. I’m not sure maintaining this is a requirement and, at least
> for gradle, eliminating it would have greatly simplified things.
>
> Test execution environment was also an issue. perhaps more a
> result of structural issues. i.e., the gradle config treated a component’s
> tests as unit tests, run against the associated component’s .class files.
> The ant system treated them more like integration tests, run against the
> jar files that would be bundled into a binary-release bundle; more like the
> environment that Edgent-based applications would use. That was very
> important. And due to some manual setup requirements for some connector
> tests are excluded from execution by default.
>
> > …
> > In the Flex project, I also setup the build to run SonarQube
> analysis and automatically generate the documentation from markdown and/or
> asciidoctor (which I think is very convenient) even automatically update
> and deploy the project website.
>
> We include use of jacoco
> We have travis integration for auto-PR validation.
> It would be nice to have some periodic build/regression testing
> run on the main (master) branch.
>
> > I could offer to create a fork on GitHub, create a feature
> branch there and try to whip up a set of poms that add Maven as fourth
> build system to the list … you could check it out and play around with it.
> But I’d only do this, if there is any interest in it.
>
> I don’t blame you!
>
> One worry I have is that it ends up like the gradle effort… it was
> ~trivial to get started but there was then a lot of effort required
> identify and flesh out equivalence with the ant based build result
> artifacts.
> If you’re on board for the long haul in doing a conversion then
> that lessens my concern.
>
> I’m on board to help but don’t want to inherit a “now finish it”
> task :-)
>
> A full switch to a ~simple mvn-based build system would be the
> goal IMO ― we’d want to toss gradle. The effort ultimately also entails
> updating a bunch of doc. Not a killer, just not to be overlooked.
>
> It’s unclear to me how the Edgent-based app developers will be
> affected by all of this. OK, they build their Edgent-based app with some
> tool that can utilize a maven repo. Then what?
>
> Today the story is simple: extract the binary-tgz (or a subset of
> it) on the target (edge device), copy your app code jar/classes to the
> target, set the CLASSPATH and go. (assume we also eliminate
> building/distributing a binary-tgz)
>
> Can elaborate on that part of the story?
>
> Thanks!
> ― Dale
>
>
>
>
>
>
Re: Understanding the snapshot and release process
Posted by "John D. Ament" <jo...@apache.org>.
The maven based build seems to be much more maintainable. I'm glad you
guys came together to solve this problem :-)
RE distribution of snapshots. We only presently support publishing via our
hosted jenkins, however Edgent currently builds in Travis. So that would
need to be changed, or some other solution.
You also can drop the repositories/pluginRepositories sections of the new
pom. Its not needed.
John
On Sun, May 28, 2017 at 10:03 AM Christofer Dutz <ch...@c-ware.de>
wrote:
> Hi guys,
>
> Strange people these Germans … if it rains, they complain about it and how
> much they would like to go swimming in the sun … now that it’s 32° and the
> sun is shining, no one wants to go swimming, cause it’s too fu***g hot ;-)
> … well this way I had some time to finish the maven migration … I just
> pushed the changes to the “feature/maven” branch of my fork on github:
> https://github.com/chrisdutz/incubator-edgent/tree/feature/maven
>
> Here a short list on what little monsters I found siting under some rocks:
>
> 1) Usage of test-jars --> Bad practice → Move code to separate test-util
> jar module
> 2) AppServiceTest in provides/direct relies on system-properties to load
> jar produced by another module → Ideally this should be refactored to find
> the jar on the classpath
> 3) HttpServerTest in console/server relies on console.war
> 4) console/servlets module compiled to something with a name completely
> unrelated to the project
> 5) test/svt requires samples project → Some modules rely on samples. While
> I think, this is ok for Samples themselves, nothing outside the samples
> should rely on samples (my opinion)
> 6) Almost all tests relying on successful SSL handshakes in
> wsclient-javax.websocket are failing (I’ll investigate this as soon as I
> have time, I could imagine this to be related to non US JVM encryption
> issues we are also seeing in the Flex project)
> 7) Some unit-tests should more be integration-tests
>
> I didn’t migrate the stuff in “platform” as I didn’t quite understand what
> they are used for and what they should do.
>
> Hope you like the changes. With a build like this it should be easy to
> setup the CI to auto distribute SNAPSHOTs and execute releases on ASF
> machines with a simple one or two-liner.
>
> Chris
>
>
> Am 27.05.17, 19:16 schrieb "Christofer Dutz" <ch...@c-ware.de>:
>
> I did have some free time this weekend, so I thought I’d give the
> Maven thing a try. And right now I haven finished, but I think I’m about
> 80% done. So far there have been one or two little monsters under some
> rocks … but they weren’t too nasty ;-)
>
> Regarding the classpath stuff in the manifest: I really dislike this
> way of setting the classpath as it usually causes a lot of problems when
> using it. It requires libs to be in a predefined directory structure and
> follow a given naming convention. Try to use this in a project built with
> Maven (just picking this as I saw this would be a problem for a Maven built
> project)
>
> Regarding tests: I did see that there are several Unit tests I would
> probably more call integration-tests. Especially the ones that require
> loading of a war (console.war). So I guess you have both some classical
> unit-tests as well as integration-tests … no need to treat them the same
> way. Maven ususally has the unit-test and the integration-test phase for
> exactly this.
>
> Regarding jacoco: So I’ll activate that in the maven build too
>
> Regarding automatic build: For Flex I setup the build to do a
> multibranch pipeline build. So whenever someone checks something in
> “develop”, “releae/*” or “feature/*” it automatically builds that
> particular branch. However develop and “release/*” are the only ones
> uploading SNAPSHOT versions to the ASF Nexus.
>
> Regarding my commitment: I think Edgent is the tool I was looking for
> in order to make my SCADA project feature-perfect. So I have a strong
> interest in the project maturing. I do have plans for a longer involvement
> in the project. Probably I would be trying to create an open-source “plc”
> connector (which would probably have to be developed outside of Apache due
> to libraries still being GPL). But I do plan on staying around some time ;-)
>
> Regarding the usage: Maven or Gradle based application developers
> could directly utilize SNAPSHOT and release artifacts as they are released.
> Additionally, I would create a so-called assembly that produces the same
> zip or tar-gz archives the users have been using. So there shouldn’t be too
> much of a change.
>
> So now I’ll try to finish the rest of the connector modules maven
> conversion (
>
> Chris
>
> Am 25.05.17, 19:37 schrieb "Dale LaBossiere" <dm...@gmail.com>:
>
> Awaiting others to chime in on this “switch to mvn” subject.
> A “don’t care” (+/- 0) response is preferred over silence.
>
> In the mean time...
>
> > On May 24, 2017, at 11:41 AM, Christofer Dutz <
> christofer.dutz@c-ware.de> wrote:
> > ...
> > I just checked out everything and managed to get things imported
> in IntelliJ after a little struggle … this is a good job you did :-)
> Can’t take all the credit (or blame :-)
> >
> > Right now, it seems as if there were Ant+Gradle+Eclipse build
> files in there. While it might seem ...
> gradle is the only CLI way to build edgent. Any ant build.xml
> files are either (a) residual cruft that should be removed (my bad) or (b)
> needed to leverage the ant machinery for invoking retrolambda in building
> java7/android compatible versions of the jars (machinery that wasn’t
> converted to pure gradle due to time/effort/value at the time). Attempting
> to use ant at the top level tells you it doesn’t work :-) As for Eclipse
> .project/.classpath, yup those are live and haven’t been a maintenance
> issue.
>
> fwiw, I just removed all build.xml except the top level one and
> those under platform and “gradle release” worked fine so I’ll create a PR
> to clean them up as a first step.
>
> > ...JavaDoc is generated automatically when running a release
> build together with the usual Maven project reports (is even configured in
> the apache parent POM together with rat, deployment etc.)
> Javadoc was complicated by creating groupings as well as excluding
> it for non-API classes.
>
> > ...I didn’t quite understand the “Manifest” thing, but the jar
> plugin does generate this with reasonable defaults, and can be extended to
> also export the dependencies into that (even if I don’t recommend that).
> An edgent jar’s manifest class-path includes references to its
> immediate dependent edgent jars (not transitive) as well as references to
> its external jar dependencies (transitively). I agree that "compiled in"
> references to specific versions of external dependencies may not be a great
> idea / is perhaps best eliminated.
>
> My recollection is that by default gradle did not generate any
> manifest class-path.
>
> e.g., just to make this a bit more concrete, this is from
> edgent.connectors.kafka.jar/MANIFEST.MF (see connectors/kafka/build.gradle
> for more info)
>
> Class-Path: ../../../lib/edgent.api.topology.jar
> ../../../ext/gson-2.2
> .4.jar ../../../ext/slf4j-api-1.7.12.jar
> ../../../ext/metrics-core-3.
> 1.2.jar ../ext/kafka_2.10-0.8.2.2.jar
> ../ext/kafka-clients-0.8.2.2.ja
> r ../ext/log4j-1.2.16.jar ../ext/metrics-core-2.2.0.jar
> ../ext/scala-
> library-2.10.4.jar ../ext/zkclient-0.3.jar
> ../ext/zookeeper-3.4.6.jar
>
> edgent.api.topology.jar itself has lots of edgent jar dependencies
> (captured in its manifest) but that’s of no concern to the kafka
> connector. I’m not sure maintaining this is a requirement and, at least
> for gradle, eliminating it would have greatly simplified things.
>
> Test execution environment was also an issue. perhaps more a
> result of structural issues. i.e., the gradle config treated a component’s
> tests as unit tests, run against the associated component’s .class files.
> The ant system treated them more like integration tests, run against the
> jar files that would be bundled into a binary-release bundle; more like the
> environment that Edgent-based applications would use. That was very
> important. And due to some manual setup requirements for some connector
> tests are excluded from execution by default.
>
> > …
> > In the Flex project, I also setup the build to run SonarQube
> analysis and automatically generate the documentation from markdown and/or
> asciidoctor (which I think is very convenient) even automatically update
> and deploy the project website.
>
> We include use of jacoco
> We have travis integration for auto-PR validation.
> It would be nice to have some periodic build/regression testing
> run on the main (master) branch.
>
> > I could offer to create a fork on GitHub, create a feature
> branch there and try to whip up a set of poms that add Maven as fourth
> build system to the list … you could check it out and play around with it.
> But I’d only do this, if there is any interest in it.
>
> I don’t blame you!
>
> One worry I have is that it ends up like the gradle effort… it was
> ~trivial to get started but there was then a lot of effort required
> identify and flesh out equivalence with the ant based build result
> artifacts.
> If you’re on board for the long haul in doing a conversion then
> that lessens my concern.
>
> I’m on board to help but don’t want to inherit a “now finish it”
> task :-)
>
> A full switch to a ~simple mvn-based build system would be the
> goal IMO — we’d want to toss gradle. The effort ultimately also entails
> updating a bunch of doc. Not a killer, just not to be overlooked.
>
> It’s unclear to me how the Edgent-based app developers will be
> affected by all of this. OK, they build their Edgent-based app with some
> tool that can utilize a maven repo. Then what?
>
> Today the story is simple: extract the binary-tgz (or a subset of
> it) on the target (edge device), copy your app code jar/classes to the
> target, set the CLASSPATH and go. (assume we also eliminate
> building/distributing a binary-tgz)
>
> Can elaborate on that part of the story?
>
> Thanks!
> — Dale
>
>
>
>
>
>
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi guys,
Strange people these Germans … if it rains, they complain about it and how much they would like to go swimming in the sun … now that it’s 32° and the sun is shining, no one wants to go swimming, cause it’s too fu***g hot ;-) … well this way I had some time to finish the maven migration … I just pushed the changes to the “feature/maven” branch of my fork on github: https://github.com/chrisdutz/incubator-edgent/tree/feature/maven
Here a short list on what little monsters I found siting under some rocks:
1) Usage of test-jars --> Bad practice → Move code to separate test-util jar module
2) AppServiceTest in provides/direct relies on system-properties to load jar produced by another module → Ideally this should be refactored to find the jar on the classpath
3) HttpServerTest in console/server relies on console.war
4) console/servlets module compiled to something with a name completely unrelated to the project
5) test/svt requires samples project → Some modules rely on samples. While I think, this is ok for Samples themselves, nothing outside the samples should rely on samples (my opinion)
6) Almost all tests relying on successful SSL handshakes in wsclient-javax.websocket are failing (I’ll investigate this as soon as I have time, I could imagine this to be related to non US JVM encryption issues we are also seeing in the Flex project)
7) Some unit-tests should more be integration-tests
I didn’t migrate the stuff in “platform” as I didn’t quite understand what they are used for and what they should do.
Hope you like the changes. With a build like this it should be easy to setup the CI to auto distribute SNAPSHOTs and execute releases on ASF machines with a simple one or two-liner.
Chris
Am 27.05.17, 19:16 schrieb "Christofer Dutz" <ch...@c-ware.de>:
I did have some free time this weekend, so I thought I’d give the Maven thing a try. And right now I haven finished, but I think I’m about 80% done. So far there have been one or two little monsters under some rocks … but they weren’t too nasty ;-)
Regarding the classpath stuff in the manifest: I really dislike this way of setting the classpath as it usually causes a lot of problems when using it. It requires libs to be in a predefined directory structure and follow a given naming convention. Try to use this in a project built with Maven (just picking this as I saw this would be a problem for a Maven built project)
Regarding tests: I did see that there are several Unit tests I would probably more call integration-tests. Especially the ones that require loading of a war (console.war). So I guess you have both some classical unit-tests as well as integration-tests … no need to treat them the same way. Maven ususally has the unit-test and the integration-test phase for exactly this.
Regarding jacoco: So I’ll activate that in the maven build too
Regarding automatic build: For Flex I setup the build to do a multibranch pipeline build. So whenever someone checks something in “develop”, “releae/*” or “feature/*” it automatically builds that particular branch. However develop and “release/*” are the only ones uploading SNAPSHOT versions to the ASF Nexus.
Regarding my commitment: I think Edgent is the tool I was looking for in order to make my SCADA project feature-perfect. So I have a strong interest in the project maturing. I do have plans for a longer involvement in the project. Probably I would be trying to create an open-source “plc” connector (which would probably have to be developed outside of Apache due to libraries still being GPL). But I do plan on staying around some time ;-)
Regarding the usage: Maven or Gradle based application developers could directly utilize SNAPSHOT and release artifacts as they are released. Additionally, I would create a so-called assembly that produces the same zip or tar-gz archives the users have been using. So there shouldn’t be too much of a change.
So now I’ll try to finish the rest of the connector modules maven conversion (
Chris
Am 25.05.17, 19:37 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Awaiting others to chime in on this “switch to mvn” subject.
A “don’t care” (+/- 0) response is preferred over silence.
In the mean time...
> On May 24, 2017, at 11:41 AM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...
> I just checked out everything and managed to get things imported in IntelliJ after a little struggle … this is a good job you did :-)
Can’t take all the credit (or blame :-)
>
> Right now, it seems as if there were Ant+Gradle+Eclipse build files in there. While it might seem ...
gradle is the only CLI way to build edgent. Any ant build.xml files are either (a) residual cruft that should be removed (my bad) or (b) needed to leverage the ant machinery for invoking retrolambda in building java7/android compatible versions of the jars (machinery that wasn’t converted to pure gradle due to time/effort/value at the time). Attempting to use ant at the top level tells you it doesn’t work :-) As for Eclipse .project/.classpath, yup those are live and haven’t been a maintenance issue.
fwiw, I just removed all build.xml except the top level one and those under platform and “gradle release” worked fine so I’ll create a PR to clean them up as a first step.
> ...JavaDoc is generated automatically when running a release build together with the usual Maven project reports (is even configured in the apache parent POM together with rat, deployment etc.)
Javadoc was complicated by creating groupings as well as excluding it for non-API classes.
> ...I didn’t quite understand the “Manifest” thing, but the jar plugin does generate this with reasonable defaults, and can be extended to also export the dependencies into that (even if I don’t recommend that).
An edgent jar’s manifest class-path includes references to its immediate dependent edgent jars (not transitive) as well as references to its external jar dependencies (transitively). I agree that "compiled in" references to specific versions of external dependencies may not be a great idea / is perhaps best eliminated.
My recollection is that by default gradle did not generate any manifest class-path.
e.g., just to make this a bit more concrete, this is from edgent.connectors.kafka.jar/MANIFEST.MF (see connectors/kafka/build.gradle for more info)
Class-Path: ../../../lib/edgent.api.topology.jar ../../../ext/gson-2.2
.4.jar ../../../ext/slf4j-api-1.7.12.jar ../../../ext/metrics-core-3.
1.2.jar ../ext/kafka_2.10-0.8.2.2.jar ../ext/kafka-clients-0.8.2.2.ja
r ../ext/log4j-1.2.16.jar ../ext/metrics-core-2.2.0.jar ../ext/scala-
library-2.10.4.jar ../ext/zkclient-0.3.jar ../ext/zookeeper-3.4.6.jar
edgent.api.topology.jar itself has lots of edgent jar dependencies (captured in its manifest) but that’s of no concern to the kafka connector. I’m not sure maintaining this is a requirement and, at least for gradle, eliminating it would have greatly simplified things.
Test execution environment was also an issue. perhaps more a result of structural issues. i.e., the gradle config treated a component’s tests as unit tests, run against the associated component’s .class files. The ant system treated them more like integration tests, run against the jar files that would be bundled into a binary-release bundle; more like the environment that Edgent-based applications would use. That was very important. And due to some manual setup requirements for some connector tests are excluded from execution by default.
> …
> In the Flex project, I also setup the build to run SonarQube analysis and automatically generate the documentation from markdown and/or asciidoctor (which I think is very convenient) even automatically update and deploy the project website.
We include use of jacoco
We have travis integration for auto-PR validation.
It would be nice to have some periodic build/regression testing run on the main (master) branch.
> I could offer to create a fork on GitHub, create a feature branch there and try to whip up a set of poms that add Maven as fourth build system to the list … you could check it out and play around with it. But I’d only do this, if there is any interest in it.
I don’t blame you!
One worry I have is that it ends up like the gradle effort… it was ~trivial to get started but there was then a lot of effort required identify and flesh out equivalence with the ant based build result artifacts.
If you’re on board for the long haul in doing a conversion then that lessens my concern.
I’m on board to help but don’t want to inherit a “now finish it” task :-)
A full switch to a ~simple mvn-based build system would be the goal IMO — we’d want to toss gradle. The effort ultimately also entails updating a bunch of doc. Not a killer, just not to be overlooked.
It’s unclear to me how the Edgent-based app developers will be affected by all of this. OK, they build their Edgent-based app with some tool that can utilize a maven repo. Then what?
Today the story is simple: extract the binary-tgz (or a subset of it) on the target (edge device), copy your app code jar/classes to the target, set the CLASSPATH and go. (assume we also eliminate building/distributing a binary-tgz)
Can elaborate on that part of the story?
Thanks!
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
I did have some free time this weekend, so I thought I’d give the Maven thing a try. And right now I haven finished, but I think I’m about 80% done. So far there have been one or two little monsters under some rocks … but they weren’t too nasty ;-)
Regarding the classpath stuff in the manifest: I really dislike this way of setting the classpath as it usually causes a lot of problems when using it. It requires libs to be in a predefined directory structure and follow a given naming convention. Try to use this in a project built with Maven (just picking this as I saw this would be a problem for a Maven built project)
Regarding tests: I did see that there are several Unit tests I would probably more call integration-tests. Especially the ones that require loading of a war (console.war). So I guess you have both some classical unit-tests as well as integration-tests … no need to treat them the same way. Maven ususally has the unit-test and the integration-test phase for exactly this.
Regarding jacoco: So I’ll activate that in the maven build too
Regarding automatic build: For Flex I setup the build to do a multibranch pipeline build. So whenever someone checks something in “develop”, “releae/*” or “feature/*” it automatically builds that particular branch. However develop and “release/*” are the only ones uploading SNAPSHOT versions to the ASF Nexus.
Regarding my commitment: I think Edgent is the tool I was looking for in order to make my SCADA project feature-perfect. So I have a strong interest in the project maturing. I do have plans for a longer involvement in the project. Probably I would be trying to create an open-source “plc” connector (which would probably have to be developed outside of Apache due to libraries still being GPL). But I do plan on staying around some time ;-)
Regarding the usage: Maven or Gradle based application developers could directly utilize SNAPSHOT and release artifacts as they are released. Additionally, I would create a so-called assembly that produces the same zip or tar-gz archives the users have been using. So there shouldn’t be too much of a change.
So now I’ll try to finish the rest of the connector modules maven conversion (
Chris
Am 25.05.17, 19:37 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Awaiting others to chime in on this “switch to mvn” subject.
A “don’t care” (+/- 0) response is preferred over silence.
In the mean time...
> On May 24, 2017, at 11:41 AM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...
> I just checked out everything and managed to get things imported in IntelliJ after a little struggle … this is a good job you did :-)
Can’t take all the credit (or blame :-)
>
> Right now, it seems as if there were Ant+Gradle+Eclipse build files in there. While it might seem ...
gradle is the only CLI way to build edgent. Any ant build.xml files are either (a) residual cruft that should be removed (my bad) or (b) needed to leverage the ant machinery for invoking retrolambda in building java7/android compatible versions of the jars (machinery that wasn’t converted to pure gradle due to time/effort/value at the time). Attempting to use ant at the top level tells you it doesn’t work :-) As for Eclipse .project/.classpath, yup those are live and haven’t been a maintenance issue.
fwiw, I just removed all build.xml except the top level one and those under platform and “gradle release” worked fine so I’ll create a PR to clean them up as a first step.
> ...JavaDoc is generated automatically when running a release build together with the usual Maven project reports (is even configured in the apache parent POM together with rat, deployment etc.)
Javadoc was complicated by creating groupings as well as excluding it for non-API classes.
> ...I didn’t quite understand the “Manifest” thing, but the jar plugin does generate this with reasonable defaults, and can be extended to also export the dependencies into that (even if I don’t recommend that).
An edgent jar’s manifest class-path includes references to its immediate dependent edgent jars (not transitive) as well as references to its external jar dependencies (transitively). I agree that "compiled in" references to specific versions of external dependencies may not be a great idea / is perhaps best eliminated.
My recollection is that by default gradle did not generate any manifest class-path.
e.g., just to make this a bit more concrete, this is from edgent.connectors.kafka.jar/MANIFEST.MF (see connectors/kafka/build.gradle for more info)
Class-Path: ../../../lib/edgent.api.topology.jar ../../../ext/gson-2.2
.4.jar ../../../ext/slf4j-api-1.7.12.jar ../../../ext/metrics-core-3.
1.2.jar ../ext/kafka_2.10-0.8.2.2.jar ../ext/kafka-clients-0.8.2.2.ja
r ../ext/log4j-1.2.16.jar ../ext/metrics-core-2.2.0.jar ../ext/scala-
library-2.10.4.jar ../ext/zkclient-0.3.jar ../ext/zookeeper-3.4.6.jar
edgent.api.topology.jar itself has lots of edgent jar dependencies (captured in its manifest) but that’s of no concern to the kafka connector. I’m not sure maintaining this is a requirement and, at least for gradle, eliminating it would have greatly simplified things.
Test execution environment was also an issue. perhaps more a result of structural issues. i.e., the gradle config treated a component’s tests as unit tests, run against the associated component’s .class files. The ant system treated them more like integration tests, run against the jar files that would be bundled into a binary-release bundle; more like the environment that Edgent-based applications would use. That was very important. And due to some manual setup requirements for some connector tests are excluded from execution by default.
> …
> In the Flex project, I also setup the build to run SonarQube analysis and automatically generate the documentation from markdown and/or asciidoctor (which I think is very convenient) even automatically update and deploy the project website.
We include use of jacoco
We have travis integration for auto-PR validation.
It would be nice to have some periodic build/regression testing run on the main (master) branch.
> I could offer to create a fork on GitHub, create a feature branch there and try to whip up a set of poms that add Maven as fourth build system to the list … you could check it out and play around with it. But I’d only do this, if there is any interest in it.
I don’t blame you!
One worry I have is that it ends up like the gradle effort… it was ~trivial to get started but there was then a lot of effort required identify and flesh out equivalence with the ant based build result artifacts.
If you’re on board for the long haul in doing a conversion then that lessens my concern.
I’m on board to help but don’t want to inherit a “now finish it” task :-)
A full switch to a ~simple mvn-based build system would be the goal IMO — we’d want to toss gradle. The effort ultimately also entails updating a bunch of doc. Not a killer, just not to be overlooked.
It’s unclear to me how the Edgent-based app developers will be affected by all of this. OK, they build their Edgent-based app with some tool that can utilize a maven repo. Then what?
Today the story is simple: extract the binary-tgz (or a subset of it) on the target (edge device), copy your app code jar/classes to the target, set the CLASSPATH and go. (assume we also eliminate building/distributing a binary-tgz)
Can elaborate on that part of the story?
Thanks!
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Awaiting others to chime in on this “switch to mvn” subject.
A “don’t care” (+/- 0) response is preferred over silence.
In the mean time...
> On May 24, 2017, at 11:41 AM, Christofer Dutz <ch...@c-ware.de> wrote:
> ...
> I just checked out everything and managed to get things imported in IntelliJ after a little struggle … this is a good job you did :-)
Can’t take all the credit (or blame :-)
>
> Right now, it seems as if there were Ant+Gradle+Eclipse build files in there. While it might seem ...
gradle is the only CLI way to build edgent. Any ant build.xml files are either (a) residual cruft that should be removed (my bad) or (b) needed to leverage the ant machinery for invoking retrolambda in building java7/android compatible versions of the jars (machinery that wasn’t converted to pure gradle due to time/effort/value at the time). Attempting to use ant at the top level tells you it doesn’t work :-) As for Eclipse .project/.classpath, yup those are live and haven’t been a maintenance issue.
fwiw, I just removed all build.xml except the top level one and those under platform and “gradle release” worked fine so I’ll create a PR to clean them up as a first step.
> ...JavaDoc is generated automatically when running a release build together with the usual Maven project reports (is even configured in the apache parent POM together with rat, deployment etc.)
Javadoc was complicated by creating groupings as well as excluding it for non-API classes.
> ...I didn’t quite understand the “Manifest” thing, but the jar plugin does generate this with reasonable defaults, and can be extended to also export the dependencies into that (even if I don’t recommend that).
An edgent jar’s manifest class-path includes references to its immediate dependent edgent jars (not transitive) as well as references to its external jar dependencies (transitively). I agree that "compiled in" references to specific versions of external dependencies may not be a great idea / is perhaps best eliminated.
My recollection is that by default gradle did not generate any manifest class-path.
e.g., just to make this a bit more concrete, this is from edgent.connectors.kafka.jar/MANIFEST.MF (see connectors/kafka/build.gradle for more info)
Class-Path: ../../../lib/edgent.api.topology.jar ../../../ext/gson-2.2
.4.jar ../../../ext/slf4j-api-1.7.12.jar ../../../ext/metrics-core-3.
1.2.jar ../ext/kafka_2.10-0.8.2.2.jar ../ext/kafka-clients-0.8.2.2.ja
r ../ext/log4j-1.2.16.jar ../ext/metrics-core-2.2.0.jar ../ext/scala-
library-2.10.4.jar ../ext/zkclient-0.3.jar ../ext/zookeeper-3.4.6.jar
edgent.api.topology.jar itself has lots of edgent jar dependencies (captured in its manifest) but that’s of no concern to the kafka connector. I’m not sure maintaining this is a requirement and, at least for gradle, eliminating it would have greatly simplified things.
Test execution environment was also an issue. perhaps more a result of structural issues. i.e., the gradle config treated a component’s tests as unit tests, run against the associated component’s .class files. The ant system treated them more like integration tests, run against the jar files that would be bundled into a binary-release bundle; more like the environment that Edgent-based applications would use. That was very important. And due to some manual setup requirements for some connector tests are excluded from execution by default.
> …
> In the Flex project, I also setup the build to run SonarQube analysis and automatically generate the documentation from markdown and/or asciidoctor (which I think is very convenient) even automatically update and deploy the project website.
We include use of jacoco
We have travis integration for auto-PR validation.
It would be nice to have some periodic build/regression testing run on the main (master) branch.
> I could offer to create a fork on GitHub, create a feature branch there and try to whip up a set of poms that add Maven as fourth build system to the list … you could check it out and play around with it. But I’d only do this, if there is any interest in it.
I don’t blame you!
One worry I have is that it ends up like the gradle effort… it was ~trivial to get started but there was then a lot of effort required identify and flesh out equivalence with the ant based build result artifacts.
If you’re on board for the long haul in doing a conversion then that lessens my concern.
I’m on board to help but don’t want to inherit a “now finish it” task :-)
A full switch to a ~simple mvn-based build system would be the goal IMO — we’d want to toss gradle. The effort ultimately also entails updating a bunch of doc. Not a killer, just not to be overlooked.
It’s unclear to me how the Edgent-based app developers will be affected by all of this. OK, they build their Edgent-based app with some tool that can utilize a maven repo. Then what?
Today the story is simple: extract the binary-tgz (or a subset of it) on the target (edge device), copy your app code jar/classes to the target, set the CLASSPATH and go. (assume we also eliminate building/distributing a binary-tgz)
Can elaborate on that part of the story?
Thanks!
— Dale
Re: Understanding the snapshot and release process
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Dale,
I just checked out everything and managed to get things imported in IntelliJ after a little struggle … this is a good job you did :-)
Right now, it seems as if there were Ant+Gradle+Eclipse build files in there. While it might seem convenient to be able to build with the build system of choice, the Flex project currently has big trouble with this as it’s a nightmare to keep all the different builds in sync. So, if someone is reporting problems we now usually ask “what build system are you using?” before asking any other questions or being able to help. So, I would suggest deciding which one it’s going to be and to stick to that.
From a first glance at the build, it does seem structurally quite simple. But probably there will be the one or other “rock” with a little “monster” underneath, if you look closer (there always is …)
The things you mentioned, that you seem to be missing with Gradle (even if I can’t quite understand why):
- Javadoc
- release bundles
- manifests
etc. are handled quite nicely by default Maven setups.
JavaDoc is generated automatically when running a release build together with the usual Maven project reports (is even configured in the apache parent POM together with rat, deployment etc.)
I didn’t quite understand the “Manifest” thing, but the jar plugin does generate this with reasonable defaults, and can be extended to also export the dependencies into that (even if I don’t recommend that).
Finally, the assembly-plugin is perfect for generating the binary (and source) release packages.
In the Flex project, I also setup the build to run SonarQube analysis and automatically generate the documentation from markdown and/or asciidoctor (which I think is very convenient) even automatically update and deploy the project website.
And as I said before, with Gradle you will be able to create an equally good build … but you will have to do a lot of the configuration manually and it does always offer the cheap way out. Unfortunately, every now and then someone will try and use this cheap way out, may the reason be just not knowing better or needing to get “this little feature out the door before the next release”.
It took me about every free minute for about 2 years to untangle and refactor the FlexJS Ant build converting it to a clean Maven build. If we had started working cleanly from the beginning, this wouldn’t have been a problem. Now the build times went down from 8 hours to 20 minutes, we have Jenkins automatically build and test all feature branches using the multi-branch pipeline plugin. I’m really satisfied with the results … well except the fact that I must re-sync the Ant and Maven build all the time ;-)
I could offer to create a fork on GitHub, create a feature branch there and try to whip up a set of poms that add Maven as fourth build system to the list … you could check it out and play around with it. But I’d only do this, if there is any interest in it.
Chris
Am 24.05.17, 17:08 schrieb "Dale LaBossiere" <dm...@gmail.com>:
On May 24, 2017, at 9:38 AM, Christofer Dutz <ch...@c-ware.de> wrote:
...
Regarding the build system … what is the projects general opinion on a Maven build? I know that Ant and Gradle are a lot more flexible as Maven, but I think it is a good thing that Maven is this strict, as it forces the developers to address structural issues instead of using workarounds. Especially in Open-Source projects with a lot of different levels of build system expertise I usually always had problems with non Maven builds. But this is just a general question without having seen the current state
----------------
I think we headed down the gradle path because someone stepped up and
contributed an initial set of gradle build files. And it was simple. But it omitted
building javadoc, release bundles, the jar manifest class-path that I’ve
mentioned, and maybe some additional stuff (oh, like building java7 based
jars using retrolambda, as well as building for an android tgt).
I don’t know that the project has an overall position on gradle vs maven. Folks?
I don’t have a strong opinion (though I was happy to see the elimination of an
xml based build spec :-)
Once “forced to address structural issues instead of using gradle workarounds",
might one then end up with an equivalently simple/clean gradle based system?
Though the path to get to that point may be less clear :-)
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
On May 24, 2017, at 9:38 AM, Christofer Dutz <ch...@c-ware.de> wrote:
...
Regarding the build system … what is the projects general opinion on a Maven build? I know that Ant and Gradle are a lot more flexible as Maven, but I think it is a good thing that Maven is this strict, as it forces the developers to address structural issues instead of using workarounds. Especially in Open-Source projects with a lot of different levels of build system expertise I usually always had problems with non Maven builds. But this is just a general question without having seen the current state
----------------
I think we headed down the gradle path because someone stepped up and
contributed an initial set of gradle build files. And it was simple. But it omitted
building javadoc, release bundles, the jar manifest class-path that I’ve
mentioned, and maybe some additional stuff (oh, like building java7 based
jars using retrolambda, as well as building for an android tgt).
I don’t know that the project has an overall position on gradle vs maven. Folks?
I don’t have a strong opinion (though I was happy to see the elimination of an
xml based build spec :-)
Once “forced to address structural issues instead of using gradle workarounds",
might one then end up with an equivalently simple/clean gradle based system?
Though the path to get to that point may be less clear :-)
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
Looks like we might also get some help from christofer.dutz@c-ware.de in cleaning up the build system!
I hope my previous message clarifies why things are the way they are.
I don’t know if there are simpler ways to achieve the same effects or
if it would be best to abandon some of them.
e.g., instead of having the “accurate” auto-generated MANIFEST.MF/Class-Path
in the Edgent jars, we could either have a general, include everything,
Class-Path or omit Class-Path and require applications to set CLASSPATH themselves.
In either case we could provide a manually created & maintained file identifying
all of these dependencies.
I believe John would like to get to a state where 3rd party dependencies
are not included in the convenience binary tgz. And/or eliminate the binary tgz?
Ultimately there’s a need to be to generate some bundle that includes
the Edgent jars and their external dependencies for download / install
onto an edge device. And the app won’t be built on the device.
It should be ~easy for an Edgent user to identify and exclude optional
Edgent components (the component’s Edgent jar and its deps) from
that bundle.
— Dale
Re: Understanding the snapshot and release process
Posted by Dale LaBossiere <dm...@gmail.com>.
> On May 22, 2017, at 9:54 PM, John D. Ament <jo...@apache.org> wrote:
> ...
> I'm trying to understand the current snapshot and release process for
> Edgent, specifically to make it so that we can easily publish JARs into
> maven central.
> ...
Yeah, the gradle stuff ended up that complicated as a result of preserving certain behaviors of the original ant-based build, and being a gradle/maven newbie. It would be great to be able to simplify it! [I wasn’t the author of the ant builds.]
The binary build (current and previous) results in a layout of Edgent and 3rd party jars:
java8/<component>/<edgent-jars>
[/ext/<component-dependent-3rd-party-jars>]
The generated Edgent jars include MANIFEST.MF/Class-Path for all immediate edgent-jar dependencies and the component’s transitive external dependencies. All are specified as relative
paths given the above layout.
Edgent applications need to only include an Edgent component that they directly use on the their classpath. Internal and external transitive dependencies are automatically handled by virtue of the manifests' classpath. In practice they need only include the path to the Edgent provider jar (which
takes care of all of the core/lib Edgent jars), and then Edgent jars for individual connectors or other
optional components. See [1].
The release mgmt guide [2] steps related to source-release and binary-release bundle creation and signing are:
./gradlew -Dedgent.snapshotId=“” release
./gradlew signAll —no-daemon
See [2] for staging and publishing steps too.
Oh, and there was some additional complication as a result of preserving the ability of Edgent-runtime developers to continue to use Eclipse without switching to gradle and/or maven [3]
[1] “Building Applications” at the end of https://edgent.apache.org/javadoc/latest/index.html <https://edgent.apache.org/javadoc/latest/index.html>
[2] https://cwiki.apache.org/confluence/display/EDGENT/Release+Manager%27s+Guide <https://cwiki.apache.org/confluence/display/EDGENT/Release+Manager's+Guide>
[3] https://github.com/apache/incubator-edgent/blob/master/DEVELOPMENT.md#using-eclipse
Hope that helps clarify things.
— Dale