You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Vladimir Sitnikov <si...@gmail.com> on 2019/07/01 13:11:41 UTC
CALCITE-3158, Avatica: migrate build system to Gradle
Hi,
Sorry for the long mail.
It has three basic parts: "URLs for Gradle PR", "Motivation/new features",
"build time comparison".
>> Urls for Gradle PR
I've started a PR for migrating Avatica build system from Maven to Gradle:
https://github.com/apache/calcite-avatica/pull/104
I've put a list of "todo items" at the top of PR104. Feel free to suggest
new items (or remove items).
Comments/reviews are welcome.
Some of the Gradle benefits can be found in
https://issues.apache.org/jira/browse/CALCITE-2905
As of now PR is good enough to try how Gradle build feels.
It builds the project, loads it to IDE, enables to execute tests, it can
push Avatica jars to local Maven repository.
It can push jars to the Nexus repository (e.g. for release purposes).
If you are brave enough, you could start right from "./gradlew prepareVote".
It would build all the things, and publish the artifacts to SVN/Git/Nexus
services (you could start stub SVN/Nexus instances via
https://github.com/vlsi/asflike-release-environment ),
and it would print a draft "release vote" mail with all the urls filled in
(see
https://github.com/vlsi/calcite-avatica/blob/gradle/release/build.gradle.kts#L23
)
Just a side note: current release procedure has no way to smoke test the
process. With my PR one can divert publish URLs to the test servers, so
even non-committers can do a release and compare if their results match
"the official" ones. My idea is to keep builds reproducible, so
source/binary artifacts produced by different people should have the same
SHA512.
>> Motivation/new features
If you are new to Gradle, you might find the following set of commands
useful (not everything works in Avatica's build, yet lots of basic commands
do): https://github.com/vlsi/jmeter/blob/gradle/gradle.md
Frankly speaking, I find Gradle commands are simpler to discover and
simpler to use than their Maven counterparts.
For instance, `gw tasks` (gw comes from https://github.com/dougborg/gdub)
provides a list of "documented" tasks.
then `gw <taskname>` invokes the task.
gw jar # builds the jar
gw checkstyleMain # executes Checkstyle
gw spotbugsMain # executes Spotbugs
gw rat # executes RAT, and so on.
Note: it does not mean you have to use 100 different commands, but you can
run individual tasks rather than Maven's "all-in-one install"
>> Build time comparison
Absolute values do not matter much, but those are taken from Macbook, Late
2013, 2,6 GHz Intel Core i7.
Note: major time is consumed by checkstyle, compilation, and tests. TL;DR
is Gradle does not make things slower, and sometimes it makes build faster.
1) "Repeated build of avatica-core without modifications"
mvn package -DskipTests: 18sec, 24sec, 21sec, 18sec
gw build -x test: 14sec, 1sec, 1sec, 1sec
As you see, Gradle recognizes that nothing should be compiled.
Maven seems to perform checkstyle and full compilation even for no-op cases.
2) "Alter string literal in AvaticaStatement#checkOpen and build with tests"
mvn package: 20sec, 19sec
gw build -x test: 15sec, 15sec, 15ec
3) "Alter string literal in server/...HttpServer#stop and build with
tests". "... clean" is performed once before the first measurement.
Two points here:
3.1) "clean" build is comparable (even though `clean` should be extremely
rare)
3.2) Gradle figured out that change in server/...HttpServer#stop does not
impact core, thus it avoids "core" compilation, tests, etc.
mvn package: 56sec, 48sec, 48sec
gw build: 50sec, 22sec, 22sec
Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Danny Chan <yu...@gmail.com>.
Thanks for the effort Vladimir !
Best,
Danny Chan
在 2019年9月12日 +0800 PM8:38，Vladimir Sitnikov <si...@gmail.com>，写道：
> Thanks everybody for helpful comments.
>
> At this point I'm confident the change is good, so I'm inclined to just
> commit it and cleanup glitches later if any.
> I'm inclined to remove pom.xml files from Git control as well.
>
> PS. There was Gradle-produced release candidate. You can find it under the
> subject of "[VOTE] Apache Calcite Avatica: Gradle-based release artifacts
> ready for review".
>
> Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Vladimir Sitnikov <si...@gmail.com>.
Thanks everybody for helpful comments.
At this point I'm confident the change is good, so I'm inclined to just
commit it and cleanup glitches later if any.
I'm inclined to remove pom.xml files from Git control as well.
PS. There was Gradle-produced release candidate. You can find it under the
subject of "[VOTE] Apache Calcite Avatica: Gradle-based release artifacts
ready for review".
Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Michael Mior <mm...@apache.org>.
Thanks Vladimir! Sorry for the delayed response. I haven't looked over
the artifacts myself in detail, but the diffs you mention don't raise
any red flags for me. I'm not sure what you mean by the "docker
module" though. As far as cleaning up the artifacts, I assume we
should implement a similar policy to that of releases and clear out
old RCs. These aren't maintained in the Apache Archives, but I also
don't see an obvious need to keep them around.
--
Michael Mior
mmior@apache.org
Le lun. 22 juil. 2019 à 04:54, Vladimir Sitnikov
<si...@gmail.com> a écrit :
>
> Julian>Before we accept this, I would like to make sure that the source
> Julian>artifacts are identical
>
> That is interesting.
> I thought "developer experience" was more important than a mythical "identity of the artifacts".
>
> Unfortunately, "Identical artifacts" sounds more like an excuse to avoid looking into
> the suggested build script improvements.
> The build is almost complete to fully replace Maven, and the only feedback I get is like "make the artifacts identical before we even try it".
>
> I would make a RC via Gradle soon.
>
> Note: I still have no idea on what "docker" module is for, however I do not think it is vital.
>
> I see there are lots of old files at https://dist.apache.org/repos/dist/dev/calcite/
> Do we need to keep them? I'm inclined to cleanup /dist/dev/calcite folder.
>
> ----
>
> Nevertheless, I've implemented an automated comparator that builds artifacts by both build systems
> and commits the artifacts to a temporary Git repository. Then it calls `git format-patch` to compare the results.
>
> Of course there are differences, however I would claim Gradle-generated artifacts are strictly better.
>
> Note: you can generate artifacts and inspect them via ./gradlew clean buildAllArtifacts commitGradleArtifacts compare
> The extracted artifacts are placed to $rootDir/exploded and $rootDir/git
>
> I compare the following artifacts: *.jars (except javadoc), -src.tar.gz, and *.xml.
> Gradle-based build additionally generates -src.zip (with CRLF).
>
> Differences are as follows (the patchfile that compares Maven vs Gradle artifacts is attached):
> 1) Maven build misses *-test.jar for certain projects.
> Frankly speaking, it would be nice to completely remove release of the test artifacts. However test jars are used across projects, so we can't just remove the test jars yet.
> For now I've made Gradle to generate test jars in a consistent manner.
>
> 2) Maven's naming of "-shaded.jar" files is inconsistent.
> For instance: avatica-tck-1.16.0-SNAPSHOT-shaded.jar, however shaded/core produces avatica-1.16.0-SNAPSHOT.jar (which is shaded)
>
> 3) Maven artifacts include "resources" to -sources.jar.
> Technically speaking, I do not see reasons to include non-java files to the -sources.jar
> Releasing extra files increase the surface of potential license violations, and
> I don't think releasing files like avatica-core-1.16.0-SNAPSHOT-test-sources.jar/log4j.properties help something
>
> I'm not sure what sense does it make to include *.proto files to avatica-core-1.16.0-SNAPSHOT-sources.jar.
>
> 4) Gradle-generated MANIFEST.MF do not include Build-Jdk, Created-By, Built-By.
> This is done for making the build reproducible.
>
> 5) Gradle-generated pom.xml do not contain <build> information. It does not contain <scope>test</scope> dependencies as well.
>
> 6) License files are included in a different way. Gradle build includes both LICENSE and NOTICE files from third-party dependencies.
> Those are added to /licenses/ folder (or META-INF/licenses/ for jars).
>
> 7) Maven-generated artifacts will probably be inconsistent with regards to "end-of-line" and "executable" across operation systems.
>
> 8) Maven-generated MANIFEST.MF contain non-existing URL's like:
> Implementation-URL: https://calcite.apache.org/avatica/avatica-metrics
>
> I've no idea where that link comes from, and I guess Maven generates that automatically by appending "<url>" from the parent pom with project names for subprojects.
> If we really need Implementation-URL, we should probably include valid urls.
> For now I just skip that attribute in Gradle.
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hi Vladimir,
First of all thanks a lot for the great work so far!
I never used Gradle before but I am rather convinced with all the positive
points that you raised through the various discussion so far.
Although Maven seems to be the most popular build tool, I followed various
discussions highlighting the advantages of using Gradle and it seems that
people are rather happy with it.
I think Julian raised the point about the artifacts because, as everybody
else, he is concerned that we may break something during the
migration process.
In the end what it matters is doing our best to predict every corner-case;
peoples' feedback as well as your analysis help a lot towards achieving
this goal.
At the moment, I do not have much free time to properly review the
artifacts; I will try to find some time next week if the case remains open.
Best,
Stamatis
On Mon, Jul 22, 2019 at 10:54 AM Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:
> Julian>Before we accept this, I would like to make sure that the source
> Julian>artifacts are identical
>
> That is interesting.
> I thought "developer experience" was more important than a mythical
> "identity of the artifacts".
>
> Unfortunately, "Identical artifacts" sounds more like an excuse to avoid
> looking into
> the suggested build script improvements.
> The build is almost complete to fully replace Maven, and the only feedback
> I get is like "make the artifacts identical before we even try it".
>
> I would make a RC via Gradle soon.
>
> Note: I still have no idea on what "docker" module is for, however I do
> not think it is vital.
>
> I see there are lots of old files at
> https://dist.apache.org/repos/dist/dev/calcite/
> Do we need to keep them? I'm inclined to cleanup /dist/dev/calcite folder.
>
> ----
>
> Nevertheless, I've implemented an automated comparator that builds
> artifacts by both build systems
> and commits the artifacts to a temporary Git repository. Then it calls
> `git format-patch` to compare the results.
>
> Of course there are differences, however I would claim Gradle-generated
> artifacts are strictly better.
>
> Note: you can generate artifacts and inspect them via ./gradlew clean
> buildAllArtifacts commitGradleArtifacts compare
> The extracted artifacts are placed to $rootDir/exploded and $rootDir/git
>
> I compare the following artifacts: *.jars (except javadoc), -src.tar.gz,
> and *.xml.
> Gradle-based build additionally generates -src.zip (with CRLF).
>
> Differences are as follows (the patchfile that compares Maven vs Gradle
> artifacts is attached):
> 1) Maven build misses *-test.jar for certain projects.
> Frankly speaking, it would be nice to completely remove release of the
> test artifacts. However test jars are used across projects, so we can't
> just remove the test jars yet.
> For now I've made Gradle to generate test jars in a consistent manner.
>
> 2) Maven's naming of "-shaded.jar" files is inconsistent.
> For instance: avatica-tck-1.16.0-SNAPSHOT-shaded.jar, however shaded/core
> produces avatica-1.16.0-SNAPSHOT.jar (which is shaded)
>
> 3) Maven artifacts include "resources" to -sources.jar.
> Technically speaking, I do not see reasons to include non-java files to
> the -sources.jar
> Releasing extra files increase the surface of potential license
> violations, and
> I don't think releasing files
> like avatica-core-1.16.0-SNAPSHOT-test-sources.jar/log4j.properties help
> something
>
> I'm not sure what sense does it make to include *.proto files
> to avatica-core-1.16.0-SNAPSHOT-sources.jar.
>
> 4) Gradle-generated MANIFEST.MF do not include
> Build-Jdk, Created-By, Built-By.
> This is done for making the build reproducible.
>
> 5) Gradle-generated pom.xml do not contain <build> information. It does
> not contain <scope>test</scope> dependencies as well.
>
> 6) License files are included in a different way. Gradle build includes
> both LICENSE and NOTICE files from third-party dependencies.
> Those are added to /licenses/ folder (or META-INF/licenses/ for jars).
>
> 7) Maven-generated artifacts will probably be inconsistent with regards
> to "end-of-line" and "executable" across operation systems.
>
> 8) Maven-generated MANIFEST.MF contain non-existing URL's like:
> Implementation-URL: https://calcite.apache.org/avatica/avatica-metrics
>
> I've no idea where that link comes from, and I guess Maven generates that
> automatically by appending "<url>" from the parent pom with project names
> for subprojects.
> If we really need Implementation-URL, we should probably include valid
> urls.
> For now I just skip that attribute in Gradle.
>
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Vladimir Sitnikov <si...@gmail.com>.
Julian>Before we accept this, I would like to make sure that the source
Julian>artifacts are identical
That is interesting.
I thought "developer experience" was more important than a mythical
"identity of the artifacts".
Unfortunately, "Identical artifacts" sounds more like an excuse to avoid
looking into
the suggested build script improvements.
The build is almost complete to fully replace Maven, and the only feedback
I get is like "make the artifacts identical before we even try it".
I would make a RC via Gradle soon.
Note: I still have no idea on what "docker" module is for, however I do not
think it is vital.
I see there are lots of old files at
https://dist.apache.org/repos/dist/dev/calcite/
Do we need to keep them? I'm inclined to cleanup /dist/dev/calcite folder.
----
Nevertheless, I've implemented an automated comparator that builds
artifacts by both build systems
and commits the artifacts to a temporary Git repository. Then it calls `git
format-patch` to compare the results.
Of course there are differences, however I would claim Gradle-generated
artifacts are strictly better.
Note: you can generate artifacts and inspect them via ./gradlew clean
buildAllArtifacts commitGradleArtifacts compare
The extracted artifacts are placed to $rootDir/exploded and $rootDir/git
I compare the following artifacts: *.jars (except javadoc), -src.tar.gz,
and *.xml.
Gradle-based build additionally generates -src.zip (with CRLF).
Differences are as follows (the patchfile that compares Maven vs Gradle
artifacts is attached):
1) Maven build misses *-test.jar for certain projects.
Frankly speaking, it would be nice to completely remove release of the test
artifacts. However test jars are used across projects, so we can't just
remove the test jars yet.
For now I've made Gradle to generate test jars in a consistent manner.
2) Maven's naming of "-shaded.jar" files is inconsistent.
For instance: avatica-tck-1.16.0-SNAPSHOT-shaded.jar, however shaded/core
produces avatica-1.16.0-SNAPSHOT.jar (which is shaded)
3) Maven artifacts include "resources" to -sources.jar.
Technically speaking, I do not see reasons to include non-java files to the
-sources.jar
Releasing extra files increase the surface of potential license violations,
and
I don't think releasing files
like avatica-core-1.16.0-SNAPSHOT-test-sources.jar/log4j.properties help
something
I'm not sure what sense does it make to include *.proto files
to avatica-core-1.16.0-SNAPSHOT-sources.jar.
4) Gradle-generated MANIFEST.MF do not include
Build-Jdk, Created-By, Built-By.
This is done for making the build reproducible.
5) Gradle-generated pom.xml do not contain <build> information. It does not
contain <scope>test</scope> dependencies as well.
6) License files are included in a different way. Gradle build includes
both LICENSE and NOTICE files from third-party dependencies.
Those are added to /licenses/ folder (or META-INF/licenses/ for jars).
7) Maven-generated artifacts will probably be inconsistent with regards to
"end-of-line" and "executable" across operation systems.
8) Maven-generated MANIFEST.MF contain non-existing URL's like:
Implementation-URL: https://calcite.apache.org/avatica/avatica-metrics
I've no idea where that link comes from, and I guess Maven generates that
automatically by appending "<url>" from the parent pom with project names
for subprojects.
If we really need Implementation-URL, we should probably include valid urls.
For now I just skip that attribute in Gradle.
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Julian Hyde <jh...@apache.org>.
Before we accept this, I would like to make sure that the source
artifacts are identical (in terms of name, contents, internal
directory structure) to the artifacts we produce today, and the same
goes for the contents of the JAR files (manifests,
META-INF/service/java.sql.Driver files) and generated pom.xml files.
In fact I think we should actually produce a release candidate
(including signing artifacts and uploading to nexus) as part of the
testing.
"Identical artifacts" may sound like an unreasonably high bar, but it
took a lot of effort & iterations to get them into the current shape.
On Fri, Jul 19, 2019 at 1:34 PM Vladimir Sitnikov
<si...@gmail.com> wrote:
>
> Here's an update: Gradle-based build seems to be in a quite a good shape.
>
> Just in case: it builds source artifacts (in zip and tgz formats), it
> assembles licenses, it runs tests.
> The missing bit is /docker/pom.xml -- I have no idea what it does.
>
> Any feedback?
>
> Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Vladimir Sitnikov <si...@gmail.com>.
Here's an update: Gradle-based build seems to be in a quite a good shape.
Just in case: it builds source artifacts (in zip and tgz formats), it
assembles licenses, it runs tests.
The missing bit is /docker/pom.xml -- I have no idea what it does.
Any feedback?
Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Vladimir Sitnikov <si...@gmail.com>.
Julian>Do you propose to remove pom.xml files from git
Of course I do.
I just thought "migrate Maven to Gradle" to mean "we fully replace Maven
with Gradle",
not like "we add Gradle build scripts and support two build systems at the
same time".
Julian>generate them upon release
Gradle can generate the files automatically.
It works with current PR104 state it is generated before the jars are
pushed to Nexus,
and one can easily to generate pom file (e.g.
gw generatePomFileForCorePublication or gw publishToMavenLocal)
Here's how a basic pom information is declared (e.g. name, description,
licenses, etc, etc):
https://github.com/vlsi/calcite-avatica/blob/c7bfea2e903e4e23e09f290c298f61fa71e49882/build.gradle.kts#L295-L334
The dependencies are taken from Gradle dependencies which look like
https://github.com/vlsi/calcite-avatica/blob/c7bfea2e903e4e23e09f290c298f61fa71e49882/core/build.gradle.kts#L27-L39
Here's an example how "shaded" dependencies are declared:
https://github.com/vlsi/calcite-avatica/blob/c7bfea2e903e4e23e09f290c298f61fa71e49882/tck/build.gradle.kts#L39-L49
Note how it is immediately visible which ones are shaded and which are not.
Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Julian Hyde <jh...@apache.org>.
I will support changes, such as you describe, that add a language and remove another.
I am wary of changes that add a new language (or framework, for that matter) to the mix. Hence my pushback against your use of .kts files.
We can’t entirely remove Maven and pom.xml files from the process. pom.xml files will remain artifacts that we deploy to maven central. I don’t know how this works under Gradle. Do you propose to remove pom.xml files from git and generate them upon release?
Julian
> On Jul 2, 2019, at 12:11 PM, Vladimir Sitnikov <si...@gmail.com> wrote:
>
> Michael>One of the concerns was adding another language
>
> It is not adding, but it is replacing a language.
> I replace "pom logic written in XML" with "Gradle logic written in Kotlin".
>
> Then it would probably replace /bin/bash-driven logic with the same
> "Gradle+Kotlin",
> which would remove yet another language.
>
> Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Vladimir Sitnikov <si...@gmail.com>.
Michael>One of the concerns was adding another language
It is not adding, but it is replacing a language.
I replace "pom logic written in XML" with "Gradle logic written in Kotlin".
Then it would probably replace /bin/bash-driven logic with the same
"Gradle+Kotlin",
which would remove yet another language.
Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Michael Mior <mm...@apache.org>.
> As you say, you "haven't encountered that".
My maven does work the same as yours in that I do have to take the
steps you outlined. I just haven't found it to be burdensome.
> I know of no Java-based build system. Why do you ask for that?
One of the concerns was adding another language. If this was in Java
instead of Kotlin, that concern would go away.
In any case, I'm not completely opposed to the change and I appreciate
the effort you've put forth. Personally I prefer Gradle over Maven but
I understand Julian's concerns.
--
Michael Mior
mmior@apache.org
Le mar. 2 juil. 2019 à 18:01, Vladimir Sitnikov
<si...@gmail.com> a écrit :
>
> Michael> I agree with Julian's point about adding maintenance
> Michael> overhead.
>
> Frankly speaking, it is (still) extremely hard for me to decipher Julian's
> mail.
> It is so dense that I can't really understand the meaning.
>
> For instance:
>
> Julian>Proof that the amount of knowledge required to maintain Avatica and
> Calcite is straightforward
>
> I could read "Proof" as both noun and verb.
> Both variations produce sense to me:
> a) (Please) Proof that the amount ...
> b) (the) Proof that ... is straightforward (as follows)
>
> Julian>There are more lines of code, the code is getting older, and the
> number of technologies is increasing
>
> I can't understand if that is a generic statement regarding all the changes
> in Avatica thought the recent 5-10years or if that statement classifies
> Gradle PR only.
>
> Julian>I don’t know how you can refute that claim
>
> I know neither because, well, I don't understand the preceding sentences :(
>
>
> Michael> Is there any reason we need to use Kotlin for the build script?
> Michael> Couldn't this just be more Java?
>
> Gradle has two options: Groovy or Kotlin.
>
> I know of no Java-based build system. Why do you ask for that? Do you have
> ideas?
>
> I'm sure Kotlin wins when compared with Groovy.
>
> My experience shows that:
> a) StackOverflow, Google, and Gradle documentation are good enough to
> develop Gradle scripts. Of course the maintenance is simpler than writing
> from scratch.
> b) Gradle provides explicit APIs for customization which is much simpler
> than extending Maven
> c) Gradle scripts can be debugged as usual code. That is I can put a
> breakpoint right in the middle of Gradle build script, I can watch
> variables and play with them. Can you do that in Maven? I doubt so.
>
> Michael>I agree with Julian's point about adding maintenance
> Michael>overhead.
>
> Even though you both are right that "a new technology must have some
> costs", I strongly
> disagree that Gradle for Calcite/Avatica would make maintenance more
> complicated.
>
> I have shown numerous examples, and there are multiple JIRA tickets
> regarding surprising mvn outcomes which Calcite committers can't fix.
>
> We could have a Google Hangouts talk if you will.
> It might be a nice point to spark an online conversation. I can share the
> screen and walk though basic steps.
>
> However, (even though I try to be extremely inclusive) I really fail to
> believe that you "haven't really encountered the same problems with Maven".
> I guess it means that I just failed to explain the problems in a right way.
>
> For instance. Can I pick the very first example of
> https://issues.apache.org/jira/browse/CALCITE-2905 ?
>
> 1) If you change code in "core" and or "linq4j" modules and you want to
> check "cassandra", you need to "mvn install" those modules first otherwise
> "cassandra" would use "core" from the local Maven repository
>
> As you say, you "haven't encountered that".
> Does that mean you always build and run all the tests and Calcite modules?
> I doubt so.
> Does that mean you can make changes to "linq4j", and immediately run core
> tests WITHOUT "mvn install" on linq4j? I doubt so.
> Does that mean you take "for granted that one must manually perform mvn
> install in linq4j; and only then run tests in core"? It does not have to be
> like that.
>
>
> That is especially true for cross-project changes. For instance, if you
> make a change in Avatica, you can't easily run Calcite test via Maven.
> In the same way, you can't really run a Drill test right after you modify a
> class in Calcite.
>
> Gradle makes that possible with virtually zero configuration.
>
> Once upon a time I was investigating Calcite-Drill planning issue (e.g.
> https://issues.apache.org/jira/browse/CALCITE-2223 ), and it was pity how
> developer experience of "Calcite consumers" was. One have to invent custom
> versions in poms, rebuild the projects in the right order. Sometimes I
> failed to rebuild the important ones and so on.
> It would be so much easier I could just launch a Drill test I want.
>
>
> I'm sure that feature alone dramatically reduces maintenance costs as it
> greatly simplifies cross-project testing (e.g. Avatica-Calcite, or modules
> within Calcite, or Calcite-Drill, or Avatica-Calcite-Drill or whatever
> else).
>
> Vladimir
Re: Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Haisheng Yuan <h....@alibaba-inc.com>.
> Gradle makes that possible with virtually zero configuration.
Can't agree more. In the long term, Gradle has lower maitainence cost than Maven (look at the gradle scripts and Maven pom file), let alone the performance improvement.
- Haisheng
------------------------------------------------------------------
发件人：Vladimir Sitnikov<si...@gmail.com>
日 期：2019年07月03日 00:01:31
收件人：Apache Calcite dev list<de...@calcite.apache.org>
主 题：Re: CALCITE-3158, Avatica: migrate build system to Gradle
Michael> I agree with Julian's point about adding maintenance
Michael> overhead.
Frankly speaking, it is (still) extremely hard for me to decipher Julian's
mail.
It is so dense that I can't really understand the meaning.
For instance:
Julian>Proof that the amount of knowledge required to maintain Avatica and
Calcite is straightforward
I could read "Proof" as both noun and verb.
Both variations produce sense to me:
a) (Please) Proof that the amount ...
b) (the) Proof that ... is straightforward (as follows)
Julian>There are more lines of code, the code is getting older, and the
number of technologies is increasing
I can't understand if that is a generic statement regarding all the changes
in Avatica thought the recent 5-10years or if that statement classifies
Gradle PR only.
Julian>I don’t know how you can refute that claim
I know neither because, well, I don't understand the preceding sentences :(
Michael> Is there any reason we need to use Kotlin for the build script?
Michael> Couldn't this just be more Java?
Gradle has two options: Groovy or Kotlin.
I know of no Java-based build system. Why do you ask for that? Do you have
ideas?
I'm sure Kotlin wins when compared with Groovy.
My experience shows that:
a) StackOverflow, Google, and Gradle documentation are good enough to
develop Gradle scripts. Of course the maintenance is simpler than writing
from scratch.
b) Gradle provides explicit APIs for customization which is much simpler
than extending Maven
c) Gradle scripts can be debugged as usual code. That is I can put a
breakpoint right in the middle of Gradle build script, I can watch
variables and play with them. Can you do that in Maven? I doubt so.
Michael>I agree with Julian's point about adding maintenance
Michael>overhead.
Even though you both are right that "a new technology must have some
costs", I strongly
disagree that Gradle for Calcite/Avatica would make maintenance more
complicated.
I have shown numerous examples, and there are multiple JIRA tickets
regarding surprising mvn outcomes which Calcite committers can't fix.
We could have a Google Hangouts talk if you will.
It might be a nice point to spark an online conversation. I can share the
screen and walk though basic steps.
However, (even though I try to be extremely inclusive) I really fail to
believe that you "haven't really encountered the same problems with Maven".
I guess it means that I just failed to explain the problems in a right way.
For instance. Can I pick the very first example of
https://issues.apache.org/jira/browse/CALCITE-2905 ?
1) If you change code in "core" and or "linq4j" modules and you want to
check "cassandra", you need to "mvn install" those modules first otherwise
"cassandra" would use "core" from the local Maven repository
As you say, you "haven't encountered that".
Does that mean you always build and run all the tests and Calcite modules?
I doubt so.
Does that mean you can make changes to "linq4j", and immediately run core
tests WITHOUT "mvn install" on linq4j? I doubt so.
Does that mean you take "for granted that one must manually perform mvn
install in linq4j; and only then run tests in core"? It does not have to be
like that.
That is especially true for cross-project changes. For instance, if you
make a change in Avatica, you can't easily run Calcite test via Maven.
In the same way, you can't really run a Drill test right after you modify a
class in Calcite.
Gradle makes that possible with virtually zero configuration.
Once upon a time I was investigating Calcite-Drill planning issue (e.g.
https://issues.apache.org/jira/browse/CALCITE-2223 ), and it was pity how
developer experience of "Calcite consumers" was. One have to invent custom
versions in poms, rebuild the projects in the right order. Sometimes I
failed to rebuild the important ones and so on.
It would be so much easier I could just launch a Drill test I want.
I'm sure that feature alone dramatically reduces maintenance costs as it
greatly simplifies cross-project testing (e.g. Avatica-Calcite, or modules
within Calcite, or Calcite-Drill, or Avatica-Calcite-Drill or whatever
else).
Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Julian Hyde <jh...@apache.org>.
Let me paraphrase.
You asked for proof. No proof is needed. It is self-evident.
Julian
> On Jul 2, 2019, at 9:01 AM, Vladimir Sitnikov <si...@gmail.com> wrote:
>
> Michael> I agree with Julian's point about adding maintenance
> Michael> overhead.
>
> Frankly speaking, it is (still) extremely hard for me to decipher Julian's
> mail.
> It is so dense that I can't really understand the meaning.
>
> For instance:
>
> Julian>Proof that the amount of knowledge required to maintain Avatica and
> Calcite is straightforward
>
> I could read "Proof" as both noun and verb.
> Both variations produce sense to me:
> a) (Please) Proof that the amount ...
> b) (the) Proof that ... is straightforward (as follows)
>
> Julian>There are more lines of code, the code is getting older, and the
> number of technologies is increasing
>
> I can't understand if that is a generic statement regarding all the changes
> in Avatica thought the recent 5-10years or if that statement classifies
> Gradle PR only.
>
> Julian>I don’t know how you can refute that claim
>
> I know neither because, well, I don't understand the preceding sentences :(
>
>
> Michael> Is there any reason we need to use Kotlin for the build script?
> Michael> Couldn't this just be more Java?
>
> Gradle has two options: Groovy or Kotlin.
>
> I know of no Java-based build system. Why do you ask for that? Do you have
> ideas?
>
> I'm sure Kotlin wins when compared with Groovy.
>
> My experience shows that:
> a) StackOverflow, Google, and Gradle documentation are good enough to
> develop Gradle scripts. Of course the maintenance is simpler than writing
> from scratch.
> b) Gradle provides explicit APIs for customization which is much simpler
> than extending Maven
> c) Gradle scripts can be debugged as usual code. That is I can put a
> breakpoint right in the middle of Gradle build script, I can watch
> variables and play with them. Can you do that in Maven? I doubt so.
>
> Michael>I agree with Julian's point about adding maintenance
> Michael>overhead.
>
> Even though you both are right that "a new technology must have some
> costs", I strongly
> disagree that Gradle for Calcite/Avatica would make maintenance more
> complicated.
>
> I have shown numerous examples, and there are multiple JIRA tickets
> regarding surprising mvn outcomes which Calcite committers can't fix.
>
> We could have a Google Hangouts talk if you will.
> It might be a nice point to spark an online conversation. I can share the
> screen and walk though basic steps.
>
> However, (even though I try to be extremely inclusive) I really fail to
> believe that you "haven't really encountered the same problems with Maven".
> I guess it means that I just failed to explain the problems in a right way.
>
> For instance. Can I pick the very first example of
> https://issues.apache.org/jira/browse/CALCITE-2905 ?
>
> 1) If you change code in "core" and or "linq4j" modules and you want to
> check "cassandra", you need to "mvn install" those modules first otherwise
> "cassandra" would use "core" from the local Maven repository
>
> As you say, you "haven't encountered that".
> Does that mean you always build and run all the tests and Calcite modules?
> I doubt so.
> Does that mean you can make changes to "linq4j", and immediately run core
> tests WITHOUT "mvn install" on linq4j? I doubt so.
> Does that mean you take "for granted that one must manually perform mvn
> install in linq4j; and only then run tests in core"? It does not have to be
> like that.
>
>
> That is especially true for cross-project changes. For instance, if you
> make a change in Avatica, you can't easily run Calcite test via Maven.
> In the same way, you can't really run a Drill test right after you modify a
> class in Calcite.
>
> Gradle makes that possible with virtually zero configuration.
>
> Once upon a time I was investigating Calcite-Drill planning issue (e.g.
> https://issues.apache.org/jira/browse/CALCITE-2223 ), and it was pity how
> developer experience of "Calcite consumers" was. One have to invent custom
> versions in poms, rebuild the projects in the right order. Sometimes I
> failed to rebuild the important ones and so on.
> It would be so much easier I could just launch a Drill test I want.
>
>
> I'm sure that feature alone dramatically reduces maintenance costs as it
> greatly simplifies cross-project testing (e.g. Avatica-Calcite, or modules
> within Calcite, or Calcite-Drill, or Avatica-Calcite-Drill or whatever
> else).
>
> Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Vladimir Sitnikov <si...@gmail.com>.
Michael> I agree with Julian's point about adding maintenance
Michael> overhead.
Frankly speaking, it is (still) extremely hard for me to decipher Julian's
mail.
It is so dense that I can't really understand the meaning.
For instance:
Julian>Proof that the amount of knowledge required to maintain Avatica and
Calcite is straightforward
I could read "Proof" as both noun and verb.
Both variations produce sense to me:
a) (Please) Proof that the amount ...
b) (the) Proof that ... is straightforward (as follows)
Julian>There are more lines of code, the code is getting older, and the
number of technologies is increasing
I can't understand if that is a generic statement regarding all the changes
in Avatica thought the recent 5-10years or if that statement classifies
Gradle PR only.
Julian>I don’t know how you can refute that claim
I know neither because, well, I don't understand the preceding sentences :(
Michael> Is there any reason we need to use Kotlin for the build script?
Michael> Couldn't this just be more Java?
Gradle has two options: Groovy or Kotlin.
I know of no Java-based build system. Why do you ask for that? Do you have
ideas?
I'm sure Kotlin wins when compared with Groovy.
My experience shows that:
a) StackOverflow, Google, and Gradle documentation are good enough to
develop Gradle scripts. Of course the maintenance is simpler than writing
from scratch.
b) Gradle provides explicit APIs for customization which is much simpler
than extending Maven
c) Gradle scripts can be debugged as usual code. That is I can put a
breakpoint right in the middle of Gradle build script, I can watch
variables and play with them. Can you do that in Maven? I doubt so.
Michael>I agree with Julian's point about adding maintenance
Michael>overhead.
Even though you both are right that "a new technology must have some
costs", I strongly
disagree that Gradle for Calcite/Avatica would make maintenance more
complicated.
I have shown numerous examples, and there are multiple JIRA tickets
regarding surprising mvn outcomes which Calcite committers can't fix.
We could have a Google Hangouts talk if you will.
It might be a nice point to spark an online conversation. I can share the
screen and walk though basic steps.
However, (even though I try to be extremely inclusive) I really fail to
believe that you "haven't really encountered the same problems with Maven".
I guess it means that I just failed to explain the problems in a right way.
For instance. Can I pick the very first example of
https://issues.apache.org/jira/browse/CALCITE-2905 ?
1) If you change code in "core" and or "linq4j" modules and you want to
check "cassandra", you need to "mvn install" those modules first otherwise
"cassandra" would use "core" from the local Maven repository
As you say, you "haven't encountered that".
Does that mean you always build and run all the tests and Calcite modules?
I doubt so.
Does that mean you can make changes to "linq4j", and immediately run core
tests WITHOUT "mvn install" on linq4j? I doubt so.
Does that mean you take "for granted that one must manually perform mvn
install in linq4j; and only then run tests in core"? It does not have to be
like that.
That is especially true for cross-project changes. For instance, if you
make a change in Avatica, you can't easily run Calcite test via Maven.
In the same way, you can't really run a Drill test right after you modify a
class in Calcite.
Gradle makes that possible with virtually zero configuration.
Once upon a time I was investigating Calcite-Drill planning issue (e.g.
https://issues.apache.org/jira/browse/CALCITE-2223 ), and it was pity how
developer experience of "Calcite consumers" was. One have to invent custom
versions in poms, rebuild the projects in the right order. Sometimes I
failed to rebuild the important ones and so on.
It would be so much easier I could just launch a Drill test I want.
I'm sure that feature alone dramatically reduces maintenance costs as it
greatly simplifies cross-project testing (e.g. Avatica-Calcite, or modules
within Calcite, or Calcite-Drill, or Avatica-Calcite-Drill or whatever
else).
Vladimir
Re: CALCITE-3158, Avatica: migrate build system to Gradle
Posted by Haisheng Yuan <h....@alibaba-inc.com>.
The performance improvement with Gradle is appealing. I would like to give it a try.
- Haisheng
------------------------------------------------------------------
发件人：Vladimir Sitnikov<si...@gmail.com>
日 期：2019年07月01日 21:11:41
收件人：Apache Calcite dev list<de...@calcite.apache.org>
主 题：CALCITE-3158, Avatica: migrate build system to Gradle
Hi,
Sorry for the long mail.
It has three basic parts: "URLs for Gradle PR", "Motivation/new features",
"build time comparison".
>> Urls for Gradle PR
I've started a PR for migrating Avatica build system from Maven to Gradle:
https://github.com/apache/calcite-avatica/pull/104
I've put a list of "todo items" at the top of PR104. Feel free to suggest
new items (or remove items).
Comments/reviews are welcome.
Some of the Gradle benefits can be found in
https://issues.apache.org/jira/browse/CALCITE-2905
As of now PR is good enough to try how Gradle build feels.
It builds the project, loads it to IDE, enables to execute tests, it can
push Avatica jars to local Maven repository.
It can push jars to the Nexus repository (e.g. for release purposes).
If you are brave enough, you could start right from "./gradlew prepareVote".
It would build all the things, and publish the artifacts to SVN/Git/Nexus
services (you could start stub SVN/Nexus instances via
https://github.com/vlsi/asflike-release-environment ),
and it would print a draft "release vote" mail with all the urls filled in
(see
https://github.com/vlsi/calcite-avatica/blob/gradle/release/build.gradle.kts#L23
)
Just a side note: current release procedure has no way to smoke test the
process. With my PR one can divert publish URLs to the test servers, so
even non-committers can do a release and compare if their results match
"the official" ones. My idea is to keep builds reproducible, so
source/binary artifacts produced by different people should have the same
SHA512.
>> Motivation/new features
If you are new to Gradle, you might find the following set of commands
useful (not everything works in Avatica's build, yet lots of basic commands
do): https://github.com/vlsi/jmeter/blob/gradle/gradle.md
Frankly speaking, I find Gradle commands are simpler to discover and
simpler to use than their Maven counterparts.
For instance, `gw tasks` (gw comes from https://github.com/dougborg/gdub)
provides a list of "documented" tasks.
then `gw <taskname>` invokes the task.
gw jar # builds the jar
gw checkstyleMain # executes Checkstyle
gw spotbugsMain # executes Spotbugs
gw rat # executes RAT, and so on.
Note: it does not mean you have to use 100 different commands, but you can
run individual tasks rather than Maven's "all-in-one install"
>> Build time comparison
Absolute values do not matter much, but those are taken from Macbook, Late
2013, 2,6 GHz Intel Core i7.
Note: major time is consumed by checkstyle, compilation, and tests. TL;DR
is Gradle does not make things slower, and sometimes it makes build faster.
1) "Repeated build of avatica-core without modifications"
mvn package -DskipTests: 18sec, 24sec, 21sec, 18sec
gw build -x test: 14sec, 1sec, 1sec, 1sec
As you see, Gradle recognizes that nothing should be compiled.
Maven seems to perform checkstyle and full compilation even for no-op cases.
2) "Alter string literal in AvaticaStatement#checkOpen and build with tests"
mvn package: 20sec, 19sec
gw build -x test: 15sec, 15sec, 15ec
3) "Alter string literal in server/...HttpServer#stop and build with
tests". "... clean" is performed once before the first measurement.
Two points here:
3.1) "clean" build is comparable (even though `clean` should be extremely
rare)
3.2) Gradle figured out that change in server/...HttpServer#stop does not
impact core, thus it avoids "core" compilation, tests, etc.
mvn package: 56sec, 48sec, 48sec
gw build: 50sec, 22sec, 22sec
Vladimir