You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by Christofer Dutz <ch...@c-ware.de> on 2018/12/11 20:42:30 UTC
Things I noticed
Hi all,
So I was curious, checked out the code and ran a build. I had to disable one test (more details down the document), but in the end I got a successful build. Congrats on that … some of us – especially one other of your mentors - know at least one project where it takes a little more setting up to get it to build ;-)
Here comes a list of things we will need changing for Apache (And or in order to avoid a lot of problems in the future):
* The parent pom in the root has to be set to “org.apache:apache:21” (or the latest version of that)
* As far as I know the ASF doesn’t like the “developers” section in the pom files
* All non-binary files need the ASF header (Yes … all of them … I know this will not be the most glorious job, but it has to be done)
* Module Grafana
* The parent of this module is not within the project scope. This quite often causes problems with the maven tooling and when releasing, the module it will not inherit the settings of the apache parent. This has to be changed (Try adding the iotdb-parent as parent and adding the missing configuration options from the spring-boot-starter-parent. This way you don’t have to redefine all the settings you can no longer inherit from the iotdb-parent.
* Module service-rpc and tsfile:
* You are defining a “release” profile which is obviously intended for releasing. This will not apply as apache releases are performed with the “apache-release” profile. So if important stuff is to be done here (it actually shouldn’t)
* From a quick look all defined there is also done by the profile defined in the “apache-release” profile in the “apache” parent pom. So all except the exclusion for package names including “thrift” could simply be removed. (Maybe it’s worth configuring the maven-javadoc-plugin in a pluginManagement section of the root to exclude them in general and get rid of the “release” profiles in general)
* Especially the staging part has to be removed as you will be staging to repository.apache.org instead of the oss sonatype repo.
Things I personally would suggest to change:
* Don’t use a property to define the version of the artifacts that are part of the build (project.version) (The release plugin updates this for you and believe me I have encountered a lot of problems with this approach in my last 10 years of managing Maven projects as a consultant)
* Why are you generating code (thrift) and checking that in? I would strongly suggest to always generate the code, move the thrift files to “src/main/thrift” and have the build generate things ot its usual location “target/generated-sources/thrift” (or something like that … the last part of the path may differ)
* General observations:
* Replace the antrun plugin with the dependencies plugin for copying the libs to the iotdb/lib directory (antrun is highly deprecated)
* No need to re-define the license, scm, distribution management in sub-modules
* Try to unify the plugin definitions in the root pom.
* The “release” profile could be causing problems in a real release (will not be executed as with apache it’s called “apache-release”)
* You are relying on Java 8 so the Jodatime library is included in general there is no need to use the external library and the included classes could be used)
* It might we worth defining an “example” module which contains real maven buildable examples (some projects even move them to a separate examples repository) right now there is no way to automatically detect that the code has changed and the example no longer matches that API.
* I wouldn’t produce fat jars (using the assembly plugin) as this bundles classes in to a library that might be used in another application that has dependencies to the same libraries but in a different version … trust me … these are the worst problems to track down.
* Module iotdb:
* Have the MANIFEST.MF generated by maven instead.
* You have a dependency on the antlr3 maven plugin. Instead you should depend on the antlr-runtime artifact that matches your antlr3 version.
* What is the “kvmatch-iotdb” artifact referring to? Seems it is not part of this project.
* Why are you manually configuring an additional compilation in the pre-integration-test phase? There is no code in “src/it/java”
* One test is failing because I’m in a different time zone: Failed tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest): expected:<…02-01T11:12:35.000+0[1]:00> but was:<…02-01T11:12:35.000+0[8]:00> (cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references ending with “+08:00” which makes the tests only run in one time-zone)
* cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses the deprecated internal SUN API which is not available in Java 11 anymore
* Module jdbc:
* You are adding a “interface/thrift” directory to the sources, but that doesn’t exist.
* Module service-rpc:
* Move the interfaces thirft files to “src/main/thrift” and have the code generated on every build to “target/generated-sources” (default)
* Don’t check in generated code.
* Module tsfile_
* In general the same remarks apply as to the service-rpc module
* Module spark
* You have included *.tsfile in your “.gitglobal” but seem to have forced in a “tsfile”-file in the test resources (Things can get quite messy in such situations if the file is ever changed, as the IDE will never detect changes and never commit any changes to that file)
* Module Hadoop
* Looks a little unfinished … so I won’t go into details here.
So much for a quick run through the project … If you want, I could be of assistance with the actual transformation … I know I could probably address most in half a day and that most people would probably take a lot longer. After all we want you to learn the Apache way … not to learn setting up projects ;-) (Other mentors please correct me if I’m wrong)
So … I’ll call it a night or my girlfriend will get angry ;-)
Chris
AW: Things I noticed
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi all,
ok so I just pushed another update to my PR and now the Spring-Boot application builds fine and I am also able to run it.
So from my side, I'm done for today ... if you have any questions ... don't hesitate to ask ... dropping changes like that without education is not the way we roll ;-)
Chris
Am 12.12.18, 13:43 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Ah yeah ...
I forgot to mention the pull request is open ...
https://github.com/thulab/iotdb/pull/510
Chris
Am 12.12.18, 13:41 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi all,
ok ... so I'm working on it ... ok ... I'm mostly finished.
I tested things on Mac, Windows and Linux (Ubuntu 18.04)
The root pom now contains several profiles which should automatically activate themselves as soon as the conditions are met.
As soon as a maven project contains a directory "src/main/thrift" it will download and use the executable matching the OS type and generate the sources to the maven default of "target/generated-sources/thrift"
This automatically excludes those sources from several checks (Such as the RAT tool).
Using the Maven Wrapper there the only prerequisite for building is Java 8 or above. Instead of using "mvn" you can now simply use "mvnw" and the script will auto-download and configure the right Maven version.
Some things I noticed:
- When running on Windows in the iotdb module there are systematic test failures as there seem to be some file locking problems in the teardown methods of the tests
- In the iotdb there are multiple issues with the JavaDoc generation.
- the class referenced in the MANIFEST.MF of the iotdb module doesn't exist
- The vulnerable dependency check does list several problematic library versions (Lib versions for which public CVEs have been filed)
- With the fix in the Time test the build on my Mac and Ubuntu ran flawlessly
Chris
Am 12.12.18, 09:16 schrieb "Willem Jiang" <wi...@gmail.com>:
Hi Christofer,
Please go ahead. It could be great if you can give a hand on it.
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Wed, Dec 12, 2018 at 3:12 PM Christofer Dutz
<ch...@c-ware.de> wrote:
>
> Hi Willem,
>
> If you don't mind, I would like to give it a try to automate this. I think the should be options for it.
>
> Chris
>
> Outlook for Android<https://aka.ms/ghei36> herunterladen
>
> ________________________________
> From: Willem Jiang <wi...@gmail.com>
> Sent: Wednesday, December 12, 2018 3:49:54 AM
> To: dev@iotdb.apache.org
> Subject: Re: Things I noticed
>
> I just did a quick search of thrift generated tool. It looks like
> there is no across system platform tools to do the job.
> We still need to install the thrift binary before generate the thrift
> related code. In this case I think it's fine to check in the generated
> code there. But we still need to update the rat[1] tool to skip the
> License header checking for those codes.
>
> [1]https://creadur.apache.org/rat/
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
> On Wed, Dec 12, 2018 at 8:58 AM Xiangdong Huang <sa...@gmail.com> wrote:
> >
> > Hi,
> >
> > Christofer, thank you so much for your detailed suggestion. It is a good
> > chance to let us know how a clear and canonical maven project is.
> >
> > I have organized all of these issues and open 3 issues at Github: #502,
> > #503, #504
> >
> > * https://github.com/thulab/iotdb/issues/502
> > * https://github.com/thulab/iotdb/issues/503
> > * https://github.com/thulab/iotdb/issues/504
> >
> > And, all guys (current committers), it is time to fight.
> >
> > As for the release-profile, I have no experience about that. We will try
> > our best to fix it, and may ask for more help. :D
> >
> > As for the generated codes, especially for Thrfit, I have some question.
> > Generating Thrift code on Windows is easy, you just need a thrfit.exe file.
> > But for Linux and Mac OS, you have to install the thrift yourself. And you
> > must make sure that the version of the thrift is the same with us (v0.9).
> > If someone does not install thrift, Maven will says that something like
> > `/usr/local/bin/thrift does not exist`, and the maven build will failed.
> > Besides, we can find some other projects also check in the generated thrift
> > code, like Apache Cassandra v2.0[1].
> >
> > Reference:
> > [1] https://github.com/apache/cassandra/tree/cassandra-2.0/interface
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
> >
> >
> > Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三 上午4:49写道：
> > >
> > > Hi all,
> > >
> > > So I was curious, checked out the code and ran a build. I had to disable
> > one test (more details down the document), but in the end I got a
> > successful build. Congrats on that … some of us – especially one other of
> > your mentors - know at least one project where it takes a little more
> > setting up to get it to build ;-)
> > >
> > > Here comes a list of things we will need changing for Apache (And or in
> > order to avoid a lot of problems in the future):
> > >
> > >
> > > * The parent pom in the root has to be set to “org.apache:apache:21”
> > (or the latest version of that)
> > > * As far as I know the ASF doesn’t like the “developers” section in
> > the pom files
> > > * All non-binary files need the ASF header (Yes … all of them … I
> > know this will not be the most glorious job, but it has to be done)
> > > * Module Grafana
> > > * The parent of this module is not within the project scope. This
> > quite often causes problems with the maven tooling and when releasing, the
> > module it will not inherit the settings of the apache parent. This has to
> > be changed (Try adding the iotdb-parent as parent and adding the missing
> > configuration options from the spring-boot-starter-parent. This way you
> > don’t have to redefine all the settings you can no longer inherit from the
> > iotdb-parent.
> > > * Module service-rpc and tsfile:
> > > * You are defining a “release” profile which is obviously intended
> > for releasing. This will not apply as apache releases are performed with
> > the “apache-release” profile. So if important stuff is to be done here (it
> > actually shouldn’t)
> > > * From a quick look all defined there is also done by the
> > profile defined in the “apache-release” profile in the “apache” parent pom.
> > So all except the exclusion for package names including “thrift” could
> > simply be removed. (Maybe it’s worth configuring the maven-javadoc-plugin
> > in a pluginManagement section of the root to exclude them in general and
> > get rid of the “release” profiles in general)
> > > * Especially the staging part has to be removed as you will be
> > staging to repository.apache.org instead of the oss sonatype repo.
> > >
> > > Things I personally would suggest to change:
> > >
> > >
> > > * Don’t use a property to define the version of the artifacts that
> > are part of the build (project.version) (The release plugin updates this
> > for you and believe me I have encountered a lot of problems with this
> > approach in my last 10 years of managing Maven projects as a consultant)
> > > * Why are you generating code (thrift) and checking that in? I would
> > strongly suggest to always generate the code, move the thrift files to
> > “src/main/thrift” and have the build generate things ot its usual location
> > “target/generated-sources/thrift” (or something like that … the last part
> > of the path may differ)
> > > * General observations:
> > > * Replace the antrun plugin with the dependencies plugin for
> > copying the libs to the iotdb/lib directory (antrun is highly deprecated)
> > > * No need to re-define the license, scm, distribution management
> > in sub-modules
> > > * Try to unify the plugin definitions in the root pom.
> > > * The “release” profile could be causing problems in a real
> > release (will not be executed as with apache it’s called “apache-release”)
> > > * You are relying on Java 8 so the Jodatime library is included in
> > general there is no need to use the external library and the included
> > classes could be used)
> > > * It might we worth defining an “example” module which contains
> > real maven buildable examples (some projects even move them to a separate
> > examples repository) right now there is no way to automatically detect that
> > the code has changed and the example no longer matches that API.
> > > * I wouldn’t produce fat jars (using the assembly plugin) as this
> > bundles classes in to a library that might be used in another application
> > that has dependencies to the same libraries but in a different version …
> > trust me … these are the worst problems to track down.
> > > * Module iotdb:
> > > * Have the MANIFEST.MF generated by maven instead.
> > > * You have a dependency on the antlr3 maven plugin. Instead you
> > should depend on the antlr-runtime artifact that matches your antlr3
> > version.
> > > * What is the “kvmatch-iotdb” artifact referring to? Seems it is
> > not part of this project.
> > > * Why are you manually configuring an additional compilation in
> > the pre-integration-test phase? There is no code in “src/it/java”
> > > * One test is failing because I’m in a different time zone: Failed
> > tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest):
> > expected:<…02-01T11:12:35.000+0[1]:00> but
> > was:<…02-01T11:12:35.000+0[8]:00>
> > (cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references ending
> > with “+08:00” which makes the tests only run in one time-zone)
> > > *
> > cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses
> > the deprecated internal SUN API which is not available in Java 11 anymore
> > > * Module jdbc:
> > > * You are adding a “interface/thrift” directory to the sources,
> > but that doesn’t exist.
> > > * Module service-rpc:
> > > * Move the interfaces thirft files to “src/main/thrift” and have
> > the code generated on every build to “target/generated-sources” (default)
> > > * Don’t check in generated code.
> > > * Module tsfile_
> > > * In general the same remarks apply as to the service-rpc module
> > > * Module spark
> > > * You have included *.tsfile in your “.gitglobal” but seem to have
> > forced in a “tsfile”-file in the test resources (Things can get quite messy
> > in such situations if the file is ever changed, as the IDE will never
> > detect changes and never commit any changes to that file)
> > > * Module Hadoop
> > > * Looks a little unfinished … so I won’t go into details here.
> > >
> > > So much for a quick run through the project … If you want, I could be of
> > assistance with the actual transformation … I know I could probably address
> > most in half a day and that most people would probably take a lot longer.
> > After all we want you to learn the Apache way … not to learn setting up
> > projects ;-) (Other mentors please correct me if I’m wrong)
> > >
> > > So … I’ll call it a night or my girlfriend will get angry ;-)
> > >
> > > Chris
> > >
> > >
AW: Things I noticed
Posted by Christofer Dutz <ch...@c-ware.de>.
Ah yeah ...
I forgot to mention the pull request is open ...
https://github.com/thulab/iotdb/pull/510
Chris
Am 12.12.18, 13:41 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi all,
ok ... so I'm working on it ... ok ... I'm mostly finished.
I tested things on Mac, Windows and Linux (Ubuntu 18.04)
The root pom now contains several profiles which should automatically activate themselves as soon as the conditions are met.
As soon as a maven project contains a directory "src/main/thrift" it will download and use the executable matching the OS type and generate the sources to the maven default of "target/generated-sources/thrift"
This automatically excludes those sources from several checks (Such as the RAT tool).
Using the Maven Wrapper there the only prerequisite for building is Java 8 or above. Instead of using "mvn" you can now simply use "mvnw" and the script will auto-download and configure the right Maven version.
Some things I noticed:
- When running on Windows in the iotdb module there are systematic test failures as there seem to be some file locking problems in the teardown methods of the tests
- In the iotdb there are multiple issues with the JavaDoc generation.
- the class referenced in the MANIFEST.MF of the iotdb module doesn't exist
- The vulnerable dependency check does list several problematic library versions (Lib versions for which public CVEs have been filed)
- With the fix in the Time test the build on my Mac and Ubuntu ran flawlessly
Chris
Am 12.12.18, 09:16 schrieb "Willem Jiang" <wi...@gmail.com>:
Hi Christofer,
Please go ahead. It could be great if you can give a hand on it.
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Wed, Dec 12, 2018 at 3:12 PM Christofer Dutz
<ch...@c-ware.de> wrote:
>
> Hi Willem,
>
> If you don't mind, I would like to give it a try to automate this. I think the should be options for it.
>
> Chris
>
> Outlook for Android<https://aka.ms/ghei36> herunterladen
>
> ________________________________
> From: Willem Jiang <wi...@gmail.com>
> Sent: Wednesday, December 12, 2018 3:49:54 AM
> To: dev@iotdb.apache.org
> Subject: Re: Things I noticed
>
> I just did a quick search of thrift generated tool. It looks like
> there is no across system platform tools to do the job.
> We still need to install the thrift binary before generate the thrift
> related code. In this case I think it's fine to check in the generated
> code there. But we still need to update the rat[1] tool to skip the
> License header checking for those codes.
>
> [1]https://creadur.apache.org/rat/
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
> On Wed, Dec 12, 2018 at 8:58 AM Xiangdong Huang <sa...@gmail.com> wrote:
> >
> > Hi,
> >
> > Christofer, thank you so much for your detailed suggestion. It is a good
> > chance to let us know how a clear and canonical maven project is.
> >
> > I have organized all of these issues and open 3 issues at Github: #502,
> > #503, #504
> >
> > * https://github.com/thulab/iotdb/issues/502
> > * https://github.com/thulab/iotdb/issues/503
> > * https://github.com/thulab/iotdb/issues/504
> >
> > And, all guys (current committers), it is time to fight.
> >
> > As for the release-profile, I have no experience about that. We will try
> > our best to fix it, and may ask for more help. :D
> >
> > As for the generated codes, especially for Thrfit, I have some question.
> > Generating Thrift code on Windows is easy, you just need a thrfit.exe file.
> > But for Linux and Mac OS, you have to install the thrift yourself. And you
> > must make sure that the version of the thrift is the same with us (v0.9).
> > If someone does not install thrift, Maven will says that something like
> > `/usr/local/bin/thrift does not exist`, and the maven build will failed.
> > Besides, we can find some other projects also check in the generated thrift
> > code, like Apache Cassandra v2.0[1].
> >
> > Reference:
> > [1] https://github.com/apache/cassandra/tree/cassandra-2.0/interface
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
> >
> >
> > Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三 上午4:49写道：
> > >
> > > Hi all,
> > >
> > > So I was curious, checked out the code and ran a build. I had to disable
> > one test (more details down the document), but in the end I got a
> > successful build. Congrats on that … some of us – especially one other of
> > your mentors - know at least one project where it takes a little more
> > setting up to get it to build ;-)
> > >
> > > Here comes a list of things we will need changing for Apache (And or in
> > order to avoid a lot of problems in the future):
> > >
> > >
> > > * The parent pom in the root has to be set to “org.apache:apache:21”
> > (or the latest version of that)
> > > * As far as I know the ASF doesn’t like the “developers” section in
> > the pom files
> > > * All non-binary files need the ASF header (Yes … all of them … I
> > know this will not be the most glorious job, but it has to be done)
> > > * Module Grafana
> > > * The parent of this module is not within the project scope. This
> > quite often causes problems with the maven tooling and when releasing, the
> > module it will not inherit the settings of the apache parent. This has to
> > be changed (Try adding the iotdb-parent as parent and adding the missing
> > configuration options from the spring-boot-starter-parent. This way you
> > don’t have to redefine all the settings you can no longer inherit from the
> > iotdb-parent.
> > > * Module service-rpc and tsfile:
> > > * You are defining a “release” profile which is obviously intended
> > for releasing. This will not apply as apache releases are performed with
> > the “apache-release” profile. So if important stuff is to be done here (it
> > actually shouldn’t)
> > > * From a quick look all defined there is also done by the
> > profile defined in the “apache-release” profile in the “apache” parent pom.
> > So all except the exclusion for package names including “thrift” could
> > simply be removed. (Maybe it’s worth configuring the maven-javadoc-plugin
> > in a pluginManagement section of the root to exclude them in general and
> > get rid of the “release” profiles in general)
> > > * Especially the staging part has to be removed as you will be
> > staging to repository.apache.org instead of the oss sonatype repo.
> > >
> > > Things I personally would suggest to change:
> > >
> > >
> > > * Don’t use a property to define the version of the artifacts that
> > are part of the build (project.version) (The release plugin updates this
> > for you and believe me I have encountered a lot of problems with this
> > approach in my last 10 years of managing Maven projects as a consultant)
> > > * Why are you generating code (thrift) and checking that in? I would
> > strongly suggest to always generate the code, move the thrift files to
> > “src/main/thrift” and have the build generate things ot its usual location
> > “target/generated-sources/thrift” (or something like that … the last part
> > of the path may differ)
> > > * General observations:
> > > * Replace the antrun plugin with the dependencies plugin for
> > copying the libs to the iotdb/lib directory (antrun is highly deprecated)
> > > * No need to re-define the license, scm, distribution management
> > in sub-modules
> > > * Try to unify the plugin definitions in the root pom.
> > > * The “release” profile could be causing problems in a real
> > release (will not be executed as with apache it’s called “apache-release”)
> > > * You are relying on Java 8 so the Jodatime library is included in
> > general there is no need to use the external library and the included
> > classes could be used)
> > > * It might we worth defining an “example” module which contains
> > real maven buildable examples (some projects even move them to a separate
> > examples repository) right now there is no way to automatically detect that
> > the code has changed and the example no longer matches that API.
> > > * I wouldn’t produce fat jars (using the assembly plugin) as this
> > bundles classes in to a library that might be used in another application
> > that has dependencies to the same libraries but in a different version …
> > trust me … these are the worst problems to track down.
> > > * Module iotdb:
> > > * Have the MANIFEST.MF generated by maven instead.
> > > * You have a dependency on the antlr3 maven plugin. Instead you
> > should depend on the antlr-runtime artifact that matches your antlr3
> > version.
> > > * What is the “kvmatch-iotdb” artifact referring to? Seems it is
> > not part of this project.
> > > * Why are you manually configuring an additional compilation in
> > the pre-integration-test phase? There is no code in “src/it/java”
> > > * One test is failing because I’m in a different time zone: Failed
> > tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest):
> > expected:<…02-01T11:12:35.000+0[1]:00> but
> > was:<…02-01T11:12:35.000+0[8]:00>
> > (cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references ending
> > with “+08:00” which makes the tests only run in one time-zone)
> > > *
> > cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses
> > the deprecated internal SUN API which is not available in Java 11 anymore
> > > * Module jdbc:
> > > * You are adding a “interface/thrift” directory to the sources,
> > but that doesn’t exist.
> > > * Module service-rpc:
> > > * Move the interfaces thirft files to “src/main/thrift” and have
> > the code generated on every build to “target/generated-sources” (default)
> > > * Don’t check in generated code.
> > > * Module tsfile_
> > > * In general the same remarks apply as to the service-rpc module
> > > * Module spark
> > > * You have included *.tsfile in your “.gitglobal” but seem to have
> > forced in a “tsfile”-file in the test resources (Things can get quite messy
> > in such situations if the file is ever changed, as the IDE will never
> > detect changes and never commit any changes to that file)
> > > * Module Hadoop
> > > * Looks a little unfinished … so I won’t go into details here.
> > >
> > > So much for a quick run through the project … If you want, I could be of
> > assistance with the actual transformation … I know I could probably address
> > most in half a day and that most people would probably take a lot longer.
> > After all we want you to learn the Apache way … not to learn setting up
> > projects ;-) (Other mentors please correct me if I’m wrong)
> > >
> > > So … I’ll call it a night or my girlfriend will get angry ;-)
> > >
> > > Chris
> > >
> > >
Re: Things I noticed
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Xiangdong,
A cve is a Common Vulnerability Enumeration.
As soon as a Vulnerability for an Apache project is found, for example, there is a process in which we claim a new cve number. This number is globally unique. As soon as the vulnerability is fixed we file a report with that cve as reference.
Now everyone can see what was wrong, what it's effects were and how to resolve the issue.
The plugin I added checks for every used library, if such a report was filled for that particular library. And it could several libs die which vulnerabilities are known
Chris
Outlook for Android<https://aka.ms/ghei36> herunterladen
________________________________
From: Xiangdong Huang <sa...@gmail.com>
Sent: Thursday, December 13, 2018 2:40:29 AM
To: dev@iotdb.apache.org
Subject: Re: Things I noticed
Hi,
Thank Chris for the wonderful maven settings and test. That is great!
> - When running on Windows in the iotdb module there are systematic test
failures as there seem to be some file locking problems in the teardown
methods of the tests
@gaofei and @liukun, can you reproduce the problem, record it on issues and
fix it?
> In the iotdb there are multiple issues with the JavaDoc generation.
@ all other committers, can we supply JavaDoc ASAP?
> the class referenced in the MANIFEST.MF of the iotdb module doesn't exist
I will remove the manifest.mf. The class was for performance testing and
has been discarded.
> The vulnerable dependency check does list several problematic library
versions (Lib versions for which public CVEs have been filed)
I am not unfamiliar with CVE, can anyone give me a reference？ or tell me
what it is short for? Then I can fix the problem.
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三 下午8:41写道：
> Hi all,
>
> ok ... so I'm working on it ... ok ... I'm mostly finished.
>
> I tested things on Mac, Windows and Linux (Ubuntu 18.04)
>
> The root pom now contains several profiles which should automatically
> activate themselves as soon as the conditions are met.
> As soon as a maven project contains a directory "src/main/thrift" it will
> download and use the executable matching the OS type and generate the
> sources to the maven default of "target/generated-sources/thrift"
> This automatically excludes those sources from several checks (Such as the
> RAT tool).
>
> Using the Maven Wrapper there the only prerequisite for building is Java 8
> or above. Instead of using "mvn" you can now simply use "mvnw" and the
> script will auto-download and configure the right Maven version.
>
> Some things I noticed:
> - When running on Windows in the iotdb module there are systematic test
> failures as there seem to be some file locking problems in the teardown
> methods of the tests
> - In the iotdb there are multiple issues with the JavaDoc generation.
> - the class referenced in the MANIFEST.MF of the iotdb module doesn't exist
> - The vulnerable dependency check does list several problematic library
> versions (Lib versions for which public CVEs have been filed)
> - With the fix in the Time test the build on my Mac and Ubuntu ran
> flawlessly
>
>
> Chris
>
>
>
>
>
>
>
> Am 12.12.18, 09:16 schrieb "Willem Jiang" <wi...@gmail.com>:
>
> Hi Christofer,
>
> Please go ahead. It could be great if you can give a hand on it.
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Wed, Dec 12, 2018 at 3:12 PM Christofer Dutz
> <ch...@c-ware.de> wrote:
> >
> > Hi Willem,
> >
> > If you don't mind, I would like to give it a try to automate this. I
> think the should be options for it.
> >
> > Chris
> >
> > Outlook for Android<https://aka.ms/ghei36> herunterladen
> >
> > ________________________________
> > From: Willem Jiang <wi...@gmail.com>
> > Sent: Wednesday, December 12, 2018 3:49:54 AM
> > To: dev@iotdb.apache.org
> > Subject: Re: Things I noticed
> >
> > I just did a quick search of thrift generated tool. It looks like
> > there is no across system platform tools to do the job.
> > We still need to install the thrift binary before generate the thrift
> > related code. In this case I think it's fine to check in the
> generated
> > code there. But we still need to update the rat[1] tool to skip the
> > License header checking for those codes.
> >
> > [1]https://creadur.apache.org/rat/
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> > On Wed, Dec 12, 2018 at 8:58 AM Xiangdong Huang <sa...@gmail.com>
> wrote:
> > >
> > > Hi,
> > >
> > > Christofer, thank you so much for your detailed suggestion. It is
> a good
> > > chance to let us know how a clear and canonical maven project is.
> > >
> > > I have organized all of these issues and open 3 issues at Github:
> #502,
> > > #503, #504
> > >
> > > * https://github.com/thulab/iotdb/issues/502
> > > * https://github.com/thulab/iotdb/issues/503
> > > * https://github.com/thulab/iotdb/issues/504
> > >
> > > And, all guys (current committers), it is time to fight.
> > >
> > > As for the release-profile, I have no experience about that. We
> will try
> > > our best to fix it, and may ask for more help. :D
> > >
> > > As for the generated codes, especially for Thrfit, I have some
> question.
> > > Generating Thrift code on Windows is easy, you just need a
> thrfit.exe file.
> > > But for Linux and Mac OS, you have to install the thrift yourself.
> And you
> > > must make sure that the version of the thrift is the same with us
> (v0.9).
> > > If someone does not install thrift, Maven will says that something
> like
> > > `/usr/local/bin/thrift does not exist`, and the maven build will
> failed.
> > > Besides, we can find some other projects also check in the
> generated thrift
> > > code, like Apache Cassandra v2.0[1].
> > >
> > > Reference:
> > > [1]
> https://github.com/apache/cassandra/tree/cassandra-2.0/interface
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > > 黄向东
> > > 清华大学 软件学院
> > >
> > >
> > > Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三
> 上午4:49写道：
> > > >
> > > > Hi all,
> > > >
> > > > So I was curious, checked out the code and ran a build. I had to
> disable
> > > one test (more details down the document), but in the end I got a
> > > successful build. Congrats on that … some of us – especially one
> other of
> > > your mentors - know at least one project where it takes a little
> more
> > > setting up to get it to build ;-)
> > > >
> > > > Here comes a list of things we will need changing for Apache
> (And or in
> > > order to avoid a lot of problems in the future):
> > > >
> > > >
> > > > * The parent pom in the root has to be set to
> “org.apache:apache:21”
> > > (or the latest version of that)
> > > > * As far as I know the ASF doesn’t like the “developers”
> section in
> > > the pom files
> > > > * All non-binary files need the ASF header (Yes … all of
> them … I
> > > know this will not be the most glorious job, but it has to be done)
> > > > * Module Grafana
> > > > * The parent of this module is not within the project
> scope. This
> > > quite often causes problems with the maven tooling and when
> releasing, the
> > > module it will not inherit the settings of the apache parent. This
> has to
> > > be changed (Try adding the iotdb-parent as parent and adding the
> missing
> > > configuration options from the spring-boot-starter-parent. This
> way you
> > > don’t have to redefine all the settings you can no longer inherit
> from the
> > > iotdb-parent.
> > > > * Module service-rpc and tsfile:
> > > > * You are defining a “release” profile which is obviously
> intended
> > > for releasing. This will not apply as apache releases are
> performed with
> > > the “apache-release” profile. So if important stuff is to be done
> here (it
> > > actually shouldn’t)
> > > > * From a quick look all defined there is also done by
> the
> > > profile defined in the “apache-release” profile in the “apache”
> parent pom.
> > > So all except the exclusion for package names including “thrift”
> could
> > > simply be removed. (Maybe it’s worth configuring the
> maven-javadoc-plugin
> > > in a pluginManagement section of the root to exclude them in
> general and
> > > get rid of the “release” profiles in general)
> > > > * Especially the staging part has to be removed as you
> will be
> > > staging to repository.apache.org instead of the oss sonatype repo.
> > > >
> > > > Things I personally would suggest to change:
> > > >
> > > >
> > > > * Don’t use a property to define the version of the
> artifacts that
> > > are part of the build (project.version) (The release plugin
> updates this
> > > for you and believe me I have encountered a lot of problems with
> this
> > > approach in my last 10 years of managing Maven projects as a
> consultant)
> > > > * Why are you generating code (thrift) and checking that in?
> I would
> > > strongly suggest to always generate the code, move the thrift
> files to
> > > “src/main/thrift” and have the build generate things ot its usual
> location
> > > “target/generated-sources/thrift” (or something like that … the
> last part
> > > of the path may differ)
> > > > * General observations:
> > > > * Replace the antrun plugin with the dependencies plugin
> for
> > > copying the libs to the iotdb/lib directory (antrun is highly
> deprecated)
> > > > * No need to re-define the license, scm, distribution
> management
> > > in sub-modules
> > > > * Try to unify the plugin definitions in the root pom.
> > > > * The “release” profile could be causing problems in a
> real
> > > release (will not be executed as with apache it’s called
> “apache-release”)
> > > > * You are relying on Java 8 so the Jodatime library is
> included in
> > > general there is no need to use the external library and the
> included
> > > classes could be used)
> > > > * It might we worth defining an “example” module which
> contains
> > > real maven buildable examples (some projects even move them to a
> separate
> > > examples repository) right now there is no way to automatically
> detect that
> > > the code has changed and the example no longer matches that API.
> > > > * I wouldn’t produce fat jars (using the assembly plugin)
> as this
> > > bundles classes in to a library that might be used in another
> application
> > > that has dependencies to the same libraries but in a different
> version …
> > > trust me … these are the worst problems to track down.
> > > > * Module iotdb:
> > > > * Have the MANIFEST.MF generated by maven instead.
> > > > * You have a dependency on the antlr3 maven plugin.
> Instead you
> > > should depend on the antlr-runtime artifact that matches your
> antlr3
> > > version.
> > > > * What is the “kvmatch-iotdb” artifact referring to?
> Seems it is
> > > not part of this project.
> > > > * Why are you manually configuring an additional
> compilation in
> > > the pre-integration-test phase? There is no code in “src/it/java”
> > > > * One test is failing because I’m in a different time
> zone: Failed
> > > tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest):
> > > expected:<…02-01T11:12:35.000+0[1]:00> but
> > > was:<…02-01T11:12:35.000+0[8]:00>
> > > (cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references
> ending
> > > with “+08:00” which makes the tests only run in one time-zone)
> > > > *
> > >
> cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses
> > > the deprecated internal SUN API which is not available in Java 11
> anymore
> > > > * Module jdbc:
> > > > * You are adding a “interface/thrift” directory to the
> sources,
> > > but that doesn’t exist.
> > > > * Module service-rpc:
> > > > * Move the interfaces thirft files to “src/main/thrift”
> and have
> > > the code generated on every build to “target/generated-sources”
> (default)
> > > > * Don’t check in generated code.
> > > > * Module tsfile_
> > > > * In general the same remarks apply as to the service-rpc
> module
> > > > * Module spark
> > > > * You have included *.tsfile in your “.gitglobal” but
> seem to have
> > > forced in a “tsfile”-file in the test resources (Things can get
> quite messy
> > > in such situations if the file is ever changed, as the IDE will
> never
> > > detect changes and never commit any changes to that file)
> > > > * Module Hadoop
> > > > * Looks a little unfinished … so I won’t go into details
> here.
> > > >
> > > > So much for a quick run through the project … If you want, I
> could be of
> > > assistance with the actual transformation … I know I could
> probably address
> > > most in half a day and that most people would probably take a lot
> longer.
> > > After all we want you to learn the Apache way … not to learn
> setting up
> > > projects ;-) (Other mentors please correct me if I’m wrong)
> > > >
> > > > So … I’ll call it a night or my girlfriend will get angry ;-)
> > > >
> > > > Chris
> > > >
> > > >
>
>
>
Re: Things I noticed
Posted by Xiangdong Huang <sa...@gmail.com>.
Hi,
Thank Chris for the wonderful maven settings and test. That is great!
> - When running on Windows in the iotdb module there are systematic test
failures as there seem to be some file locking problems in the teardown
methods of the tests
@gaofei and @liukun, can you reproduce the problem, record it on issues and
fix it?
> In the iotdb there are multiple issues with the JavaDoc generation.
@ all other committers, can we supply JavaDoc ASAP?
> the class referenced in the MANIFEST.MF of the iotdb module doesn't exist
I will remove the manifest.mf. The class was for performance testing and
has been discarded.
> The vulnerable dependency check does list several problematic library
versions (Lib versions for which public CVEs have been filed)
I am not unfamiliar with CVE, can anyone give me a reference？ or tell me
what it is short for? Then I can fix the problem.
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三 下午8:41写道：
> Hi all,
>
> ok ... so I'm working on it ... ok ... I'm mostly finished.
>
> I tested things on Mac, Windows and Linux (Ubuntu 18.04)
>
> The root pom now contains several profiles which should automatically
> activate themselves as soon as the conditions are met.
> As soon as a maven project contains a directory "src/main/thrift" it will
> download and use the executable matching the OS type and generate the
> sources to the maven default of "target/generated-sources/thrift"
> This automatically excludes those sources from several checks (Such as the
> RAT tool).
>
> Using the Maven Wrapper there the only prerequisite for building is Java 8
> or above. Instead of using "mvn" you can now simply use "mvnw" and the
> script will auto-download and configure the right Maven version.
>
> Some things I noticed:
> - When running on Windows in the iotdb module there are systematic test
> failures as there seem to be some file locking problems in the teardown
> methods of the tests
> - In the iotdb there are multiple issues with the JavaDoc generation.
> - the class referenced in the MANIFEST.MF of the iotdb module doesn't exist
> - The vulnerable dependency check does list several problematic library
> versions (Lib versions for which public CVEs have been filed)
> - With the fix in the Time test the build on my Mac and Ubuntu ran
> flawlessly
>
>
> Chris
>
>
>
>
>
>
>
> Am 12.12.18, 09:16 schrieb "Willem Jiang" <wi...@gmail.com>:
>
> Hi Christofer,
>
> Please go ahead. It could be great if you can give a hand on it.
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Wed, Dec 12, 2018 at 3:12 PM Christofer Dutz
> <ch...@c-ware.de> wrote:
> >
> > Hi Willem,
> >
> > If you don't mind, I would like to give it a try to automate this. I
> think the should be options for it.
> >
> > Chris
> >
> > Outlook for Android<https://aka.ms/ghei36> herunterladen
> >
> > ________________________________
> > From: Willem Jiang <wi...@gmail.com>
> > Sent: Wednesday, December 12, 2018 3:49:54 AM
> > To: dev@iotdb.apache.org
> > Subject: Re: Things I noticed
> >
> > I just did a quick search of thrift generated tool. It looks like
> > there is no across system platform tools to do the job.
> > We still need to install the thrift binary before generate the thrift
> > related code. In this case I think it's fine to check in the
> generated
> > code there. But we still need to update the rat[1] tool to skip the
> > License header checking for those codes.
> >
> > [1]https://creadur.apache.org/rat/
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> > On Wed, Dec 12, 2018 at 8:58 AM Xiangdong Huang <sa...@gmail.com>
> wrote:
> > >
> > > Hi,
> > >
> > > Christofer, thank you so much for your detailed suggestion. It is
> a good
> > > chance to let us know how a clear and canonical maven project is.
> > >
> > > I have organized all of these issues and open 3 issues at Github:
> #502,
> > > #503, #504
> > >
> > > * https://github.com/thulab/iotdb/issues/502
> > > * https://github.com/thulab/iotdb/issues/503
> > > * https://github.com/thulab/iotdb/issues/504
> > >
> > > And, all guys (current committers), it is time to fight.
> > >
> > > As for the release-profile, I have no experience about that. We
> will try
> > > our best to fix it, and may ask for more help. :D
> > >
> > > As for the generated codes, especially for Thrfit, I have some
> question.
> > > Generating Thrift code on Windows is easy, you just need a
> thrfit.exe file.
> > > But for Linux and Mac OS, you have to install the thrift yourself.
> And you
> > > must make sure that the version of the thrift is the same with us
> (v0.9).
> > > If someone does not install thrift, Maven will says that something
> like
> > > `/usr/local/bin/thrift does not exist`, and the maven build will
> failed.
> > > Besides, we can find some other projects also check in the
> generated thrift
> > > code, like Apache Cassandra v2.0[1].
> > >
> > > Reference:
> > > [1]
> https://github.com/apache/cassandra/tree/cassandra-2.0/interface
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > > 黄向东
> > > 清华大学 软件学院
> > >
> > >
> > > Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三
> 上午4:49写道：
> > > >
> > > > Hi all,
> > > >
> > > > So I was curious, checked out the code and ran a build. I had to
> disable
> > > one test (more details down the document), but in the end I got a
> > > successful build. Congrats on that … some of us – especially one
> other of
> > > your mentors - know at least one project where it takes a little
> more
> > > setting up to get it to build ;-)
> > > >
> > > > Here comes a list of things we will need changing for Apache
> (And or in
> > > order to avoid a lot of problems in the future):
> > > >
> > > >
> > > > * The parent pom in the root has to be set to
> “org.apache:apache:21”
> > > (or the latest version of that)
> > > > * As far as I know the ASF doesn’t like the “developers”
> section in
> > > the pom files
> > > > * All non-binary files need the ASF header (Yes … all of
> them … I
> > > know this will not be the most glorious job, but it has to be done)
> > > > * Module Grafana
> > > > * The parent of this module is not within the project
> scope. This
> > > quite often causes problems with the maven tooling and when
> releasing, the
> > > module it will not inherit the settings of the apache parent. This
> has to
> > > be changed (Try adding the iotdb-parent as parent and adding the
> missing
> > > configuration options from the spring-boot-starter-parent. This
> way you
> > > don’t have to redefine all the settings you can no longer inherit
> from the
> > > iotdb-parent.
> > > > * Module service-rpc and tsfile:
> > > > * You are defining a “release” profile which is obviously
> intended
> > > for releasing. This will not apply as apache releases are
> performed with
> > > the “apache-release” profile. So if important stuff is to be done
> here (it
> > > actually shouldn’t)
> > > > * From a quick look all defined there is also done by
> the
> > > profile defined in the “apache-release” profile in the “apache”
> parent pom.
> > > So all except the exclusion for package names including “thrift”
> could
> > > simply be removed. (Maybe it’s worth configuring the
> maven-javadoc-plugin
> > > in a pluginManagement section of the root to exclude them in
> general and
> > > get rid of the “release” profiles in general)
> > > > * Especially the staging part has to be removed as you
> will be
> > > staging to repository.apache.org instead of the oss sonatype repo.
> > > >
> > > > Things I personally would suggest to change:
> > > >
> > > >
> > > > * Don’t use a property to define the version of the
> artifacts that
> > > are part of the build (project.version) (The release plugin
> updates this
> > > for you and believe me I have encountered a lot of problems with
> this
> > > approach in my last 10 years of managing Maven projects as a
> consultant)
> > > > * Why are you generating code (thrift) and checking that in?
> I would
> > > strongly suggest to always generate the code, move the thrift
> files to
> > > “src/main/thrift” and have the build generate things ot its usual
> location
> > > “target/generated-sources/thrift” (or something like that … the
> last part
> > > of the path may differ)
> > > > * General observations:
> > > > * Replace the antrun plugin with the dependencies plugin
> for
> > > copying the libs to the iotdb/lib directory (antrun is highly
> deprecated)
> > > > * No need to re-define the license, scm, distribution
> management
> > > in sub-modules
> > > > * Try to unify the plugin definitions in the root pom.
> > > > * The “release” profile could be causing problems in a
> real
> > > release (will not be executed as with apache it’s called
> “apache-release”)
> > > > * You are relying on Java 8 so the Jodatime library is
> included in
> > > general there is no need to use the external library and the
> included
> > > classes could be used)
> > > > * It might we worth defining an “example” module which
> contains
> > > real maven buildable examples (some projects even move them to a
> separate
> > > examples repository) right now there is no way to automatically
> detect that
> > > the code has changed and the example no longer matches that API.
> > > > * I wouldn’t produce fat jars (using the assembly plugin)
> as this
> > > bundles classes in to a library that might be used in another
> application
> > > that has dependencies to the same libraries but in a different
> version …
> > > trust me … these are the worst problems to track down.
> > > > * Module iotdb:
> > > > * Have the MANIFEST.MF generated by maven instead.
> > > > * You have a dependency on the antlr3 maven plugin.
> Instead you
> > > should depend on the antlr-runtime artifact that matches your
> antlr3
> > > version.
> > > > * What is the “kvmatch-iotdb” artifact referring to?
> Seems it is
> > > not part of this project.
> > > > * Why are you manually configuring an additional
> compilation in
> > > the pre-integration-test phase? There is no code in “src/it/java”
> > > > * One test is failing because I’m in a different time
> zone: Failed
> > > tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest):
> > > expected:<…02-01T11:12:35.000+0[1]:00> but
> > > was:<…02-01T11:12:35.000+0[8]:00>
> > > (cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references
> ending
> > > with “+08:00” which makes the tests only run in one time-zone)
> > > > *
> > >
> cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses
> > > the deprecated internal SUN API which is not available in Java 11
> anymore
> > > > * Module jdbc:
> > > > * You are adding a “interface/thrift” directory to the
> sources,
> > > but that doesn’t exist.
> > > > * Module service-rpc:
> > > > * Move the interfaces thirft files to “src/main/thrift”
> and have
> > > the code generated on every build to “target/generated-sources”
> (default)
> > > > * Don’t check in generated code.
> > > > * Module tsfile_
> > > > * In general the same remarks apply as to the service-rpc
> module
> > > > * Module spark
> > > > * You have included *.tsfile in your “.gitglobal” but
> seem to have
> > > forced in a “tsfile”-file in the test resources (Things can get
> quite messy
> > > in such situations if the file is ever changed, as the IDE will
> never
> > > detect changes and never commit any changes to that file)
> > > > * Module Hadoop
> > > > * Looks a little unfinished … so I won’t go into details
> here.
> > > >
> > > > So much for a quick run through the project … If you want, I
> could be of
> > > assistance with the actual transformation … I know I could
> probably address
> > > most in half a day and that most people would probably take a lot
> longer.
> > > After all we want you to learn the Apache way … not to learn
> setting up
> > > projects ;-) (Other mentors please correct me if I’m wrong)
> > > >
> > > > So … I’ll call it a night or my girlfriend will get angry ;-)
> > > >
> > > > Chris
> > > >
> > > >
>
>
>
AW: Things I noticed
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi all,
ok ... so I'm working on it ... ok ... I'm mostly finished.
I tested things on Mac, Windows and Linux (Ubuntu 18.04)
The root pom now contains several profiles which should automatically activate themselves as soon as the conditions are met.
As soon as a maven project contains a directory "src/main/thrift" it will download and use the executable matching the OS type and generate the sources to the maven default of "target/generated-sources/thrift"
This automatically excludes those sources from several checks (Such as the RAT tool).
Using the Maven Wrapper there the only prerequisite for building is Java 8 or above. Instead of using "mvn" you can now simply use "mvnw" and the script will auto-download and configure the right Maven version.
Some things I noticed:
- When running on Windows in the iotdb module there are systematic test failures as there seem to be some file locking problems in the teardown methods of the tests
- In the iotdb there are multiple issues with the JavaDoc generation.
- the class referenced in the MANIFEST.MF of the iotdb module doesn't exist
- The vulnerable dependency check does list several problematic library versions (Lib versions for which public CVEs have been filed)
- With the fix in the Time test the build on my Mac and Ubuntu ran flawlessly
Chris
Am 12.12.18, 09:16 schrieb "Willem Jiang" <wi...@gmail.com>:
Hi Christofer,
Please go ahead. It could be great if you can give a hand on it.
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Wed, Dec 12, 2018 at 3:12 PM Christofer Dutz
<ch...@c-ware.de> wrote:
>
> Hi Willem,
>
> If you don't mind, I would like to give it a try to automate this. I think the should be options for it.
>
> Chris
>
> Outlook for Android<https://aka.ms/ghei36> herunterladen
>
> ________________________________
> From: Willem Jiang <wi...@gmail.com>
> Sent: Wednesday, December 12, 2018 3:49:54 AM
> To: dev@iotdb.apache.org
> Subject: Re: Things I noticed
>
> I just did a quick search of thrift generated tool. It looks like
> there is no across system platform tools to do the job.
> We still need to install the thrift binary before generate the thrift
> related code. In this case I think it's fine to check in the generated
> code there. But we still need to update the rat[1] tool to skip the
> License header checking for those codes.
>
> [1]https://creadur.apache.org/rat/
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
> On Wed, Dec 12, 2018 at 8:58 AM Xiangdong Huang <sa...@gmail.com> wrote:
> >
> > Hi,
> >
> > Christofer, thank you so much for your detailed suggestion. It is a good
> > chance to let us know how a clear and canonical maven project is.
> >
> > I have organized all of these issues and open 3 issues at Github: #502,
> > #503, #504
> >
> > * https://github.com/thulab/iotdb/issues/502
> > * https://github.com/thulab/iotdb/issues/503
> > * https://github.com/thulab/iotdb/issues/504
> >
> > And, all guys (current committers), it is time to fight.
> >
> > As for the release-profile, I have no experience about that. We will try
> > our best to fix it, and may ask for more help. :D
> >
> > As for the generated codes, especially for Thrfit, I have some question.
> > Generating Thrift code on Windows is easy, you just need a thrfit.exe file.
> > But for Linux and Mac OS, you have to install the thrift yourself. And you
> > must make sure that the version of the thrift is the same with us (v0.9).
> > If someone does not install thrift, Maven will says that something like
> > `/usr/local/bin/thrift does not exist`, and the maven build will failed.
> > Besides, we can find some other projects also check in the generated thrift
> > code, like Apache Cassandra v2.0[1].
> >
> > Reference:
> > [1] https://github.com/apache/cassandra/tree/cassandra-2.0/interface
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
> >
> >
> > Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三 上午4:49写道：
> > >
> > > Hi all,
> > >
> > > So I was curious, checked out the code and ran a build. I had to disable
> > one test (more details down the document), but in the end I got a
> > successful build. Congrats on that … some of us – especially one other of
> > your mentors - know at least one project where it takes a little more
> > setting up to get it to build ;-)
> > >
> > > Here comes a list of things we will need changing for Apache (And or in
> > order to avoid a lot of problems in the future):
> > >
> > >
> > > * The parent pom in the root has to be set to “org.apache:apache:21”
> > (or the latest version of that)
> > > * As far as I know the ASF doesn’t like the “developers” section in
> > the pom files
> > > * All non-binary files need the ASF header (Yes … all of them … I
> > know this will not be the most glorious job, but it has to be done)
> > > * Module Grafana
> > > * The parent of this module is not within the project scope. This
> > quite often causes problems with the maven tooling and when releasing, the
> > module it will not inherit the settings of the apache parent. This has to
> > be changed (Try adding the iotdb-parent as parent and adding the missing
> > configuration options from the spring-boot-starter-parent. This way you
> > don’t have to redefine all the settings you can no longer inherit from the
> > iotdb-parent.
> > > * Module service-rpc and tsfile:
> > > * You are defining a “release” profile which is obviously intended
> > for releasing. This will not apply as apache releases are performed with
> > the “apache-release” profile. So if important stuff is to be done here (it
> > actually shouldn’t)
> > > * From a quick look all defined there is also done by the
> > profile defined in the “apache-release” profile in the “apache” parent pom.
> > So all except the exclusion for package names including “thrift” could
> > simply be removed. (Maybe it’s worth configuring the maven-javadoc-plugin
> > in a pluginManagement section of the root to exclude them in general and
> > get rid of the “release” profiles in general)
> > > * Especially the staging part has to be removed as you will be
> > staging to repository.apache.org instead of the oss sonatype repo.
> > >
> > > Things I personally would suggest to change:
> > >
> > >
> > > * Don’t use a property to define the version of the artifacts that
> > are part of the build (project.version) (The release plugin updates this
> > for you and believe me I have encountered a lot of problems with this
> > approach in my last 10 years of managing Maven projects as a consultant)
> > > * Why are you generating code (thrift) and checking that in? I would
> > strongly suggest to always generate the code, move the thrift files to
> > “src/main/thrift” and have the build generate things ot its usual location
> > “target/generated-sources/thrift” (or something like that … the last part
> > of the path may differ)
> > > * General observations:
> > > * Replace the antrun plugin with the dependencies plugin for
> > copying the libs to the iotdb/lib directory (antrun is highly deprecated)
> > > * No need to re-define the license, scm, distribution management
> > in sub-modules
> > > * Try to unify the plugin definitions in the root pom.
> > > * The “release” profile could be causing problems in a real
> > release (will not be executed as with apache it’s called “apache-release”)
> > > * You are relying on Java 8 so the Jodatime library is included in
> > general there is no need to use the external library and the included
> > classes could be used)
> > > * It might we worth defining an “example” module which contains
> > real maven buildable examples (some projects even move them to a separate
> > examples repository) right now there is no way to automatically detect that
> > the code has changed and the example no longer matches that API.
> > > * I wouldn’t produce fat jars (using the assembly plugin) as this
> > bundles classes in to a library that might be used in another application
> > that has dependencies to the same libraries but in a different version …
> > trust me … these are the worst problems to track down.
> > > * Module iotdb:
> > > * Have the MANIFEST.MF generated by maven instead.
> > > * You have a dependency on the antlr3 maven plugin. Instead you
> > should depend on the antlr-runtime artifact that matches your antlr3
> > version.
> > > * What is the “kvmatch-iotdb” artifact referring to? Seems it is
> > not part of this project.
> > > * Why are you manually configuring an additional compilation in
> > the pre-integration-test phase? There is no code in “src/it/java”
> > > * One test is failing because I’m in a different time zone: Failed
> > tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest):
> > expected:<…02-01T11:12:35.000+0[1]:00> but
> > was:<…02-01T11:12:35.000+0[8]:00>
> > (cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references ending
> > with “+08:00” which makes the tests only run in one time-zone)
> > > *
> > cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses
> > the deprecated internal SUN API which is not available in Java 11 anymore
> > > * Module jdbc:
> > > * You are adding a “interface/thrift” directory to the sources,
> > but that doesn’t exist.
> > > * Module service-rpc:
> > > * Move the interfaces thirft files to “src/main/thrift” and have
> > the code generated on every build to “target/generated-sources” (default)
> > > * Don’t check in generated code.
> > > * Module tsfile_
> > > * In general the same remarks apply as to the service-rpc module
> > > * Module spark
> > > * You have included *.tsfile in your “.gitglobal” but seem to have
> > forced in a “tsfile”-file in the test resources (Things can get quite messy
> > in such situations if the file is ever changed, as the IDE will never
> > detect changes and never commit any changes to that file)
> > > * Module Hadoop
> > > * Looks a little unfinished … so I won’t go into details here.
> > >
> > > So much for a quick run through the project … If you want, I could be of
> > assistance with the actual transformation … I know I could probably address
> > most in half a day and that most people would probably take a lot longer.
> > After all we want you to learn the Apache way … not to learn setting up
> > projects ;-) (Other mentors please correct me if I’m wrong)
> > >
> > > So … I’ll call it a night or my girlfriend will get angry ;-)
> > >
> > > Chris
> > >
> > >
Re: Things I noticed
Posted by Willem Jiang <wi...@gmail.com>.
Hi Christofer,
Please go ahead. It could be great if you can give a hand on it.
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Wed, Dec 12, 2018 at 3:12 PM Christofer Dutz
<ch...@c-ware.de> wrote:
>
> Hi Willem,
>
> If you don't mind, I would like to give it a try to automate this. I think the should be options for it.
>
> Chris
>
> Outlook for Android<https://aka.ms/ghei36> herunterladen
>
> ________________________________
> From: Willem Jiang <wi...@gmail.com>
> Sent: Wednesday, December 12, 2018 3:49:54 AM
> To: dev@iotdb.apache.org
> Subject: Re: Things I noticed
>
> I just did a quick search of thrift generated tool. It looks like
> there is no across system platform tools to do the job.
> We still need to install the thrift binary before generate the thrift
> related code. In this case I think it's fine to check in the generated
> code there. But we still need to update the rat[1] tool to skip the
> License header checking for those codes.
>
> [1]https://creadur.apache.org/rat/
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
> On Wed, Dec 12, 2018 at 8:58 AM Xiangdong Huang <sa...@gmail.com> wrote:
> >
> > Hi,
> >
> > Christofer, thank you so much for your detailed suggestion. It is a good
> > chance to let us know how a clear and canonical maven project is.
> >
> > I have organized all of these issues and open 3 issues at Github: #502,
> > #503, #504
> >
> > * https://github.com/thulab/iotdb/issues/502
> > * https://github.com/thulab/iotdb/issues/503
> > * https://github.com/thulab/iotdb/issues/504
> >
> > And, all guys (current committers), it is time to fight.
> >
> > As for the release-profile, I have no experience about that. We will try
> > our best to fix it, and may ask for more help. :D
> >
> > As for the generated codes, especially for Thrfit, I have some question.
> > Generating Thrift code on Windows is easy, you just need a thrfit.exe file.
> > But for Linux and Mac OS, you have to install the thrift yourself. And you
> > must make sure that the version of the thrift is the same with us (v0.9).
> > If someone does not install thrift, Maven will says that something like
> > `/usr/local/bin/thrift does not exist`, and the maven build will failed.
> > Besides, we can find some other projects also check in the generated thrift
> > code, like Apache Cassandra v2.0[1].
> >
> > Reference:
> > [1] https://github.com/apache/cassandra/tree/cassandra-2.0/interface
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
> >
> >
> > Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三 上午4:49写道：
> > >
> > > Hi all,
> > >
> > > So I was curious, checked out the code and ran a build. I had to disable
> > one test (more details down the document), but in the end I got a
> > successful build. Congrats on that … some of us – especially one other of
> > your mentors - know at least one project where it takes a little more
> > setting up to get it to build ;-)
> > >
> > > Here comes a list of things we will need changing for Apache (And or in
> > order to avoid a lot of problems in the future):
> > >
> > >
> > > * The parent pom in the root has to be set to “org.apache:apache:21”
> > (or the latest version of that)
> > > * As far as I know the ASF doesn’t like the “developers” section in
> > the pom files
> > > * All non-binary files need the ASF header (Yes … all of them … I
> > know this will not be the most glorious job, but it has to be done)
> > > * Module Grafana
> > > * The parent of this module is not within the project scope. This
> > quite often causes problems with the maven tooling and when releasing, the
> > module it will not inherit the settings of the apache parent. This has to
> > be changed (Try adding the iotdb-parent as parent and adding the missing
> > configuration options from the spring-boot-starter-parent. This way you
> > don’t have to redefine all the settings you can no longer inherit from the
> > iotdb-parent.
> > > * Module service-rpc and tsfile:
> > > * You are defining a “release” profile which is obviously intended
> > for releasing. This will not apply as apache releases are performed with
> > the “apache-release” profile. So if important stuff is to be done here (it
> > actually shouldn’t)
> > > * From a quick look all defined there is also done by the
> > profile defined in the “apache-release” profile in the “apache” parent pom.
> > So all except the exclusion for package names including “thrift” could
> > simply be removed. (Maybe it’s worth configuring the maven-javadoc-plugin
> > in a pluginManagement section of the root to exclude them in general and
> > get rid of the “release” profiles in general)
> > > * Especially the staging part has to be removed as you will be
> > staging to repository.apache.org instead of the oss sonatype repo.
> > >
> > > Things I personally would suggest to change:
> > >
> > >
> > > * Don’t use a property to define the version of the artifacts that
> > are part of the build (project.version) (The release plugin updates this
> > for you and believe me I have encountered a lot of problems with this
> > approach in my last 10 years of managing Maven projects as a consultant)
> > > * Why are you generating code (thrift) and checking that in? I would
> > strongly suggest to always generate the code, move the thrift files to
> > “src/main/thrift” and have the build generate things ot its usual location
> > “target/generated-sources/thrift” (or something like that … the last part
> > of the path may differ)
> > > * General observations:
> > > * Replace the antrun plugin with the dependencies plugin for
> > copying the libs to the iotdb/lib directory (antrun is highly deprecated)
> > > * No need to re-define the license, scm, distribution management
> > in sub-modules
> > > * Try to unify the plugin definitions in the root pom.
> > > * The “release” profile could be causing problems in a real
> > release (will not be executed as with apache it’s called “apache-release”)
> > > * You are relying on Java 8 so the Jodatime library is included in
> > general there is no need to use the external library and the included
> > classes could be used)
> > > * It might we worth defining an “example” module which contains
> > real maven buildable examples (some projects even move them to a separate
> > examples repository) right now there is no way to automatically detect that
> > the code has changed and the example no longer matches that API.
> > > * I wouldn’t produce fat jars (using the assembly plugin) as this
> > bundles classes in to a library that might be used in another application
> > that has dependencies to the same libraries but in a different version …
> > trust me … these are the worst problems to track down.
> > > * Module iotdb:
> > > * Have the MANIFEST.MF generated by maven instead.
> > > * You have a dependency on the antlr3 maven plugin. Instead you
> > should depend on the antlr-runtime artifact that matches your antlr3
> > version.
> > > * What is the “kvmatch-iotdb” artifact referring to? Seems it is
> > not part of this project.
> > > * Why are you manually configuring an additional compilation in
> > the pre-integration-test phase? There is no code in “src/it/java”
> > > * One test is failing because I’m in a different time zone: Failed
> > tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest):
> > expected:<…02-01T11:12:35.000+0[1]:00> but
> > was:<…02-01T11:12:35.000+0[8]:00>
> > (cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references ending
> > with “+08:00” which makes the tests only run in one time-zone)
> > > *
> > cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses
> > the deprecated internal SUN API which is not available in Java 11 anymore
> > > * Module jdbc:
> > > * You are adding a “interface/thrift” directory to the sources,
> > but that doesn’t exist.
> > > * Module service-rpc:
> > > * Move the interfaces thirft files to “src/main/thrift” and have
> > the code generated on every build to “target/generated-sources” (default)
> > > * Don’t check in generated code.
> > > * Module tsfile_
> > > * In general the same remarks apply as to the service-rpc module
> > > * Module spark
> > > * You have included *.tsfile in your “.gitglobal” but seem to have
> > forced in a “tsfile”-file in the test resources (Things can get quite messy
> > in such situations if the file is ever changed, as the IDE will never
> > detect changes and never commit any changes to that file)
> > > * Module Hadoop
> > > * Looks a little unfinished … so I won’t go into details here.
> > >
> > > So much for a quick run through the project … If you want, I could be of
> > assistance with the actual transformation … I know I could probably address
> > most in half a day and that most people would probably take a lot longer.
> > After all we want you to learn the Apache way … not to learn setting up
> > projects ;-) (Other mentors please correct me if I’m wrong)
> > >
> > > So … I’ll call it a night or my girlfriend will get angry ;-)
> > >
> > > Chris
> > >
> > >
Re: Things I noticed
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Willem,
If you don't mind, I would like to give it a try to automate this. I think the should be options for it.
Chris
Outlook for Android<https://aka.ms/ghei36> herunterladen
________________________________
From: Willem Jiang <wi...@gmail.com>
Sent: Wednesday, December 12, 2018 3:49:54 AM
To: dev@iotdb.apache.org
Subject: Re: Things I noticed
I just did a quick search of thrift generated tool. It looks like
there is no across system platform tools to do the job.
We still need to install the thrift binary before generate the thrift
related code. In this case I think it's fine to check in the generated
code there. But we still need to update the rat[1] tool to skip the
License header checking for those codes.
[1]https://creadur.apache.org/rat/
Willem Jiang
Twitter: willemjiang
Weibo: ����willem
On Wed, Dec 12, 2018 at 8:58 AM Xiangdong Huang <sa...@gmail.com> wrote:
>
> Hi,
>
> Christofer, thank you so much for your detailed suggestion. It is a good
> chance to let us know how a clear and canonical maven project is.
>
> I have organized all of these issues and open 3 issues at Github: #502,
> #503, #504
>
> * https://github.com/thulab/iotdb/issues/502
> * https://github.com/thulab/iotdb/issues/503
> * https://github.com/thulab/iotdb/issues/504
>
> And, all guys (current committers), it is time to fight.
>
> As for the release-profile, I have no experience about that. We will try
> our best to fix it, and may ask for more help. :D
>
> As for the generated codes, especially for Thrfit, I have some question.
> Generating Thrift code on Windows is easy, you just need a thrfit.exe file.
> But for Linux and Mac OS, you have to install the thrift yourself. And you
> must make sure that the version of the thrift is the same with us (v0.9).
> If someone does not install thrift, Maven will says that something like
> `/usr/local/bin/thrift does not exist`, and the maven build will failed.
> Besides, we can find some other projects also check in the generated thrift
> code, like Apache Cassandra v2.0[1].
>
> Reference:
> [1] https://github.com/apache/cassandra/tree/cassandra-2.0/interface
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> ����
> �廪��ѧ ����ѧԺ
>
>
> Christofer Dutz <ch...@c-ware.de> ��2018��12��12������ ����4:49���
> >
> > Hi all,
> >
> > So I was curious, checked out the code and ran a build. I had to disable
> one test (more details down the document), but in the end I got a
> successful build. Congrats on that �� some of us �C especially one other of
> your mentors - know at least one project where it takes a little more
> setting up to get it to build ;-)
> >
> > Here comes a list of things we will need changing for Apache (And or in
> order to avoid a lot of problems in the future):
> >
> >
> > * The parent pom in the root has to be set to ��org.apache:apache:21��
> (or the latest version of that)
> > * As far as I know the ASF doesn��t like the ��developers�� section in
> the pom files
> > * All non-binary files need the ASF header (Yes �� all of them �� I
> know this will not be the most glorious job, but it has to be done)
> > * Module Grafana
> > * The parent of this module is not within the project scope. This
> quite often causes problems with the maven tooling and when releasing, the
> module it will not inherit the settings of the apache parent. This has to
> be changed (Try adding the iotdb-parent as parent and adding the missing
> configuration options from the spring-boot-starter-parent. This way you
> don��t have to redefine all the settings you can no longer inherit from the
> iotdb-parent.
> > * Module service-rpc and tsfile:
> > * You are defining a ��release�� profile which is obviously intended
> for releasing. This will not apply as apache releases are performed with
> the ��apache-release�� profile. So if important stuff is to be done here (it
> actually shouldn��t)
> > * From a quick look all defined there is also done by the
> profile defined in the ��apache-release�� profile in the ��apache�� parent pom.
> So all except the exclusion for package names including ��thrift�� could
> simply be removed. (Maybe it��s worth configuring the maven-javadoc-plugin
> in a pluginManagement section of the root to exclude them in general and
> get rid of the ��release�� profiles in general)
> > * Especially the staging part has to be removed as you will be
> staging to repository.apache.org instead of the oss sonatype repo.
> >
> > Things I personally would suggest to change:
> >
> >
> > * Don��t use a property to define the version of the artifacts that
> are part of the build (project.version) (The release plugin updates this
> for you and believe me I have encountered a lot of problems with this
> approach in my last 10 years of managing Maven projects as a consultant)
> > * Why are you generating code (thrift) and checking that in? I would
> strongly suggest to always generate the code, move the thrift files to
> ��src/main/thrift�� and have the build generate things ot its usual location
> ��target/generated-sources/thrift�� (or something like that �� the last part
> of the path may differ)
> > * General observations:
> > * Replace the antrun plugin with the dependencies plugin for
> copying the libs to the iotdb/lib directory (antrun is highly deprecated)
> > * No need to re-define the license, scm, distribution management
> in sub-modules
> > * Try to unify the plugin definitions in the root pom.
> > * The ��release�� profile could be causing problems in a real
> release (will not be executed as with apache it��s called ��apache-release��)
> > * You are relying on Java 8 so the Jodatime library is included in
> general there is no need to use the external library and the included
> classes could be used)
> > * It might we worth defining an ��example�� module which contains
> real maven buildable examples (some projects even move them to a separate
> examples repository) right now there is no way to automatically detect that
> the code has changed and the example no longer matches that API.
> > * I wouldn��t produce fat jars (using the assembly plugin) as this
> bundles classes in to a library that might be used in another application
> that has dependencies to the same libraries but in a different version ��
> trust me �� these are the worst problems to track down.
> > * Module iotdb:
> > * Have the MANIFEST.MF generated by maven instead.
> > * You have a dependency on the antlr3 maven plugin. Instead you
> should depend on the antlr-runtime artifact that matches your antlr3
> version.
> > * What is the ��kvmatch-iotdb�� artifact referring to? Seems it is
> not part of this project.
> > * Why are you manually configuring an additional compilation in
> the pre-integration-test phase? There is no code in ��src/it/java��
> > * One test is failing because I��m in a different time zone: Failed
> tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest):
> expected:<��02-01T11:12:35.000+0[1]:00> but
> was:<��02-01T11:12:35.000+0[8]:00>
> (cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references ending
> with ��+08:00�� which makes the tests only run in one time-zone)
> > *
> cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses
> the deprecated internal SUN API which is not available in Java 11 anymore
> > * Module jdbc:
> > * You are adding a ��interface/thrift�� directory to the sources,
> but that doesn��t exist.
> > * Module service-rpc:
> > * Move the interfaces thirft files to ��src/main/thrift�� and have
> the code generated on every build to ��target/generated-sources�� (default)
> > * Don��t check in generated code.
> > * Module tsfile_
> > * In general the same remarks apply as to the service-rpc module
> > * Module spark
> > * You have included *.tsfile in your ��.gitglobal�� but seem to have
> forced in a ��tsfile��-file in the test resources (Things can get quite messy
> in such situations if the file is ever changed, as the IDE will never
> detect changes and never commit any changes to that file)
> > * Module Hadoop
> > * Looks a little unfinished �� so I won��t go into details here.
> >
> > So much for a quick run through the project �� If you want, I could be of
> assistance with the actual transformation �� I know I could probably address
> most in half a day and that most people would probably take a lot longer.
> After all we want you to learn the Apache way �� not to learn setting up
> projects ;-) (Other mentors please correct me if I��m wrong)
> >
> > So �� I��ll call it a night or my girlfriend will get angry ;-)
> >
> > Chris
> >
> >
Re: Things I noticed
Posted by Willem Jiang <wi...@gmail.com>.
I just did a quick search of thrift generated tool. It looks like
there is no across system platform tools to do the job.
We still need to install the thrift binary before generate the thrift
related code. In this case I think it's fine to check in the generated
code there. But we still need to update the rat[1] tool to skip the
License header checking for those codes.
[1]https://creadur.apache.org/rat/
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Wed, Dec 12, 2018 at 8:58 AM Xiangdong Huang <sa...@gmail.com> wrote:
>
> Hi,
>
> Christofer, thank you so much for your detailed suggestion. It is a good
> chance to let us know how a clear and canonical maven project is.
>
> I have organized all of these issues and open 3 issues at Github: #502,
> #503, #504
>
> * https://github.com/thulab/iotdb/issues/502
> * https://github.com/thulab/iotdb/issues/503
> * https://github.com/thulab/iotdb/issues/504
>
> And, all guys (current committers), it is time to fight.
>
> As for the release-profile, I have no experience about that. We will try
> our best to fix it, and may ask for more help. :D
>
> As for the generated codes, especially for Thrfit, I have some question.
> Generating Thrift code on Windows is easy, you just need a thrfit.exe file.
> But for Linux and Mac OS, you have to install the thrift yourself. And you
> must make sure that the version of the thrift is the same with us (v0.9).
> If someone does not install thrift, Maven will says that something like
> `/usr/local/bin/thrift does not exist`, and the maven build will failed.
> Besides, we can find some other projects also check in the generated thrift
> code, like Apache Cassandra v2.0[1].
>
> Reference:
> [1] https://github.com/apache/cassandra/tree/cassandra-2.0/interface
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三 上午4:49写道：
> >
> > Hi all,
> >
> > So I was curious, checked out the code and ran a build. I had to disable
> one test (more details down the document), but in the end I got a
> successful build. Congrats on that … some of us – especially one other of
> your mentors - know at least one project where it takes a little more
> setting up to get it to build ;-)
> >
> > Here comes a list of things we will need changing for Apache (And or in
> order to avoid a lot of problems in the future):
> >
> >
> > * The parent pom in the root has to be set to “org.apache:apache:21”
> (or the latest version of that)
> > * As far as I know the ASF doesn’t like the “developers” section in
> the pom files
> > * All non-binary files need the ASF header (Yes … all of them … I
> know this will not be the most glorious job, but it has to be done)
> > * Module Grafana
> > * The parent of this module is not within the project scope. This
> quite often causes problems with the maven tooling and when releasing, the
> module it will not inherit the settings of the apache parent. This has to
> be changed (Try adding the iotdb-parent as parent and adding the missing
> configuration options from the spring-boot-starter-parent. This way you
> don’t have to redefine all the settings you can no longer inherit from the
> iotdb-parent.
> > * Module service-rpc and tsfile:
> > * You are defining a “release” profile which is obviously intended
> for releasing. This will not apply as apache releases are performed with
> the “apache-release” profile. So if important stuff is to be done here (it
> actually shouldn’t)
> > * From a quick look all defined there is also done by the
> profile defined in the “apache-release” profile in the “apache” parent pom.
> So all except the exclusion for package names including “thrift” could
> simply be removed. (Maybe it’s worth configuring the maven-javadoc-plugin
> in a pluginManagement section of the root to exclude them in general and
> get rid of the “release” profiles in general)
> > * Especially the staging part has to be removed as you will be
> staging to repository.apache.org instead of the oss sonatype repo.
> >
> > Things I personally would suggest to change:
> >
> >
> > * Don’t use a property to define the version of the artifacts that
> are part of the build (project.version) (The release plugin updates this
> for you and believe me I have encountered a lot of problems with this
> approach in my last 10 years of managing Maven projects as a consultant)
> > * Why are you generating code (thrift) and checking that in? I would
> strongly suggest to always generate the code, move the thrift files to
> “src/main/thrift” and have the build generate things ot its usual location
> “target/generated-sources/thrift” (or something like that … the last part
> of the path may differ)
> > * General observations:
> > * Replace the antrun plugin with the dependencies plugin for
> copying the libs to the iotdb/lib directory (antrun is highly deprecated)
> > * No need to re-define the license, scm, distribution management
> in sub-modules
> > * Try to unify the plugin definitions in the root pom.
> > * The “release” profile could be causing problems in a real
> release (will not be executed as with apache it’s called “apache-release”)
> > * You are relying on Java 8 so the Jodatime library is included in
> general there is no need to use the external library and the included
> classes could be used)
> > * It might we worth defining an “example” module which contains
> real maven buildable examples (some projects even move them to a separate
> examples repository) right now there is no way to automatically detect that
> the code has changed and the example no longer matches that API.
> > * I wouldn’t produce fat jars (using the assembly plugin) as this
> bundles classes in to a library that might be used in another application
> that has dependencies to the same libraries but in a different version …
> trust me … these are the worst problems to track down.
> > * Module iotdb:
> > * Have the MANIFEST.MF generated by maven instead.
> > * You have a dependency on the antlr3 maven plugin. Instead you
> should depend on the antlr-runtime artifact that matches your antlr3
> version.
> > * What is the “kvmatch-iotdb” artifact referring to? Seems it is
> not part of this project.
> > * Why are you manually configuring an additional compilation in
> the pre-integration-test phase? There is no code in “src/it/java”
> > * One test is failing because I’m in a different time zone: Failed
> tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest):
> expected:<…02-01T11:12:35.000+0[1]:00> but
> was:<…02-01T11:12:35.000+0[8]:00>
> (cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references ending
> with “+08:00” which makes the tests only run in one time-zone)
> > *
> cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses
> the deprecated internal SUN API which is not available in Java 11 anymore
> > * Module jdbc:
> > * You are adding a “interface/thrift” directory to the sources,
> but that doesn’t exist.
> > * Module service-rpc:
> > * Move the interfaces thirft files to “src/main/thrift” and have
> the code generated on every build to “target/generated-sources” (default)
> > * Don’t check in generated code.
> > * Module tsfile_
> > * In general the same remarks apply as to the service-rpc module
> > * Module spark
> > * You have included *.tsfile in your “.gitglobal” but seem to have
> forced in a “tsfile”-file in the test resources (Things can get quite messy
> in such situations if the file is ever changed, as the IDE will never
> detect changes and never commit any changes to that file)
> > * Module Hadoop
> > * Looks a little unfinished … so I won’t go into details here.
> >
> > So much for a quick run through the project … If you want, I could be of
> assistance with the actual transformation … I know I could probably address
> most in half a day and that most people would probably take a lot longer.
> After all we want you to learn the Apache way … not to learn setting up
> projects ;-) (Other mentors please correct me if I’m wrong)
> >
> > So … I’ll call it a night or my girlfriend will get angry ;-)
> >
> > Chris
> >
> >
Re: Things I noticed
Posted by Xiangdong Huang <sa...@gmail.com>.
As for pom file issues, they are at ISSUE #500 and #501
* https://github.com/thulab/iotdb/issues/500
* https://github.com/thulab/iotdb/issues/501
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Xiangdong Huang <sa...@gmail.com> 于2018年12月12日周三 上午8:58写道：
> Hi,
>
> Christofer, thank you so much for your detailed suggestion. It is a good
> chance to let us know how a clear and canonical maven project is.
>
> I have organized all of these issues and open 3 issues at Github: #502,
> #503, #504
>
> * https://github.com/thulab/iotdb/issues/502
> * https://github.com/thulab/iotdb/issues/503
> * https://github.com/thulab/iotdb/issues/504
>
> And, all guys (current committers), it is time to fight.
>
> As for the release-profile, I have no experience about that. We will try
> our best to fix it, and may ask for more help. :D
>
> As for the generated codes, especially for Thrfit, I have some question.
> Generating Thrift code on Windows is easy, you just need a thrfit.exe file.
> But for Linux and Mac OS, you have to install the thrift yourself. And you
> must make sure that the version of the thrift is the same with us (v0.9).
> If someone does not install thrift, Maven will says that something like
> `/usr/local/bin/thrift does not exist`, and the maven build will failed.
> Besides, we can find some other projects also check in the generated
> thrift code, like Apache Cassandra v2.0[1].
>
> Reference:
> [1] https://github.com/apache/cassandra/tree/cassandra-2.0/interface
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三 上午4:49写道：
> >
> > Hi all,
> >
> > So I was curious, checked out the code and ran a build. I had to disable
> one test (more details down the document), but in the end I got a
> successful build. Congrats on that … some of us – especially one other of
> your mentors - know at least one project where it takes a little more
> setting up to get it to build ;-)
> >
> > Here comes a list of things we will need changing for Apache (And or in
> order to avoid a lot of problems in the future):
> >
> >
> > * The parent pom in the root has to be set to “org.apache:apache:21”
> (or the latest version of that)
> > * As far as I know the ASF doesn’t like the “developers” section in
> the pom files
> > * All non-binary files need the ASF header (Yes … all of them … I
> know this will not be the most glorious job, but it has to be done)
> > * Module Grafana
> > * The parent of this module is not within the project scope. This
> quite often causes problems with the maven tooling and when releasing, the
> module it will not inherit the settings of the apache parent. This has to
> be changed (Try adding the iotdb-parent as parent and adding the missing
> configuration options from the spring-boot-starter-parent. This way you
> don’t have to redefine all the settings you can no longer inherit from the
> iotdb-parent.
> > * Module service-rpc and tsfile:
> > * You are defining a “release” profile which is obviously
> intended for releasing. This will not apply as apache releases are
> performed with the “apache-release” profile. So if important stuff is to be
> done here (it actually shouldn’t)
> > * From a quick look all defined there is also done by the
> profile defined in the “apache-release” profile in the “apache” parent pom.
> So all except the exclusion for package names including “thrift” could
> simply be removed. (Maybe it’s worth configuring the maven-javadoc-plugin
> in a pluginManagement section of the root to exclude them in general and
> get rid of the “release” profiles in general)
> > * Especially the staging part has to be removed as you will be
> staging to repository.apache.org instead of the oss sonatype repo.
> >
> > Things I personally would suggest to change:
> >
> >
> > * Don’t use a property to define the version of the artifacts that
> are part of the build (project.version) (The release plugin updates this
> for you and believe me I have encountered a lot of problems with this
> approach in my last 10 years of managing Maven projects as a consultant)
> > * Why are you generating code (thrift) and checking that in? I would
> strongly suggest to always generate the code, move the thrift files to
> “src/main/thrift” and have the build generate things ot its usual location
> “target/generated-sources/thrift” (or something like that … the last part
> of the path may differ)
> > * General observations:
> > * Replace the antrun plugin with the dependencies plugin for
> copying the libs to the iotdb/lib directory (antrun is highly deprecated)
> > * No need to re-define the license, scm, distribution management
> in sub-modules
> > * Try to unify the plugin definitions in the root pom.
> > * The “release” profile could be causing problems in a real
> release (will not be executed as with apache it’s called “apache-release”)
> > * You are relying on Java 8 so the Jodatime library is included
> in general there is no need to use the external library and the included
> classes could be used)
> > * It might we worth defining an “example” module which contains
> real maven buildable examples (some projects even move them to a separate
> examples repository) right now there is no way to automatically detect that
> the code has changed and the example no longer matches that API.
> > * I wouldn’t produce fat jars (using the assembly plugin) as this
> bundles classes in to a library that might be used in another application
> that has dependencies to the same libraries but in a different version …
> trust me … these are the worst problems to track down.
> > * Module iotdb:
> > * Have the MANIFEST.MF generated by maven instead.
> > * You have a dependency on the antlr3 maven plugin. Instead you
> should depend on the antlr-runtime artifact that matches your antlr3
> version.
> > * What is the “kvmatch-iotdb” artifact referring to? Seems it is
> not part of this project.
> > * Why are you manually configuring an additional compilation in
> the pre-integration-test phase? There is no code in “src/it/java”
> > * One test is failing because I’m in a different time zone:
> Failed tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest):
> expected:<…02-01T11:12:35.000+0[1]:00> but
> was:<…02-01T11:12:35.000+0[8]:00>
> (cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references ending
> with “+08:00” which makes the tests only run in one time-zone)
> > *
> cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses
> the deprecated internal SUN API which is not available in Java 11 anymore
> > * Module jdbc:
> > * You are adding a “interface/thrift” directory to the sources,
> but that doesn’t exist.
> > * Module service-rpc:
> > * Move the interfaces thirft files to “src/main/thrift” and have
> the code generated on every build to “target/generated-sources” (default)
> > * Don’t check in generated code.
> > * Module tsfile_
> > * In general the same remarks apply as to the service-rpc module
> > * Module spark
> > * You have included *.tsfile in your “.gitglobal” but seem to
> have forced in a “tsfile”-file in the test resources (Things can get quite
> messy in such situations if the file is ever changed, as the IDE will never
> detect changes and never commit any changes to that file)
> > * Module Hadoop
> > * Looks a little unfinished … so I won’t go into details here.
> >
> > So much for a quick run through the project … If you want, I could be of
> assistance with the actual transformation … I know I could probably address
> most in half a day and that most people would probably take a lot longer.
> After all we want you to learn the Apache way … not to learn setting up
> projects ;-) (Other mentors please correct me if I’m wrong)
> >
> > So … I’ll call it a night or my girlfriend will get angry ;-)
> >
> > Chris
> >
> >
>
Re: Things I noticed
Posted by Xiangdong Huang <sa...@gmail.com>.
Hi,
Christofer, thank you so much for your detailed suggestion. It is a good
chance to let us know how a clear and canonical maven project is.
I have organized all of these issues and open 3 issues at Github: #502,
#503, #504
* https://github.com/thulab/iotdb/issues/502
* https://github.com/thulab/iotdb/issues/503
* https://github.com/thulab/iotdb/issues/504
And, all guys (current committers), it is time to fight.
As for the release-profile, I have no experience about that. We will try
our best to fix it, and may ask for more help. :D
As for the generated codes, especially for Thrfit, I have some question.
Generating Thrift code on Windows is easy, you just need a thrfit.exe file.
But for Linux and Mac OS, you have to install the thrift yourself. And you
must make sure that the version of the thrift is the same with us (v0.9).
If someone does not install thrift, Maven will says that something like
`/usr/local/bin/thrift does not exist`, and the maven build will failed.
Besides, we can find some other projects also check in the generated thrift
code, like Apache Cassandra v2.0[1].
Reference:
[1] https://github.com/apache/cassandra/tree/cassandra-2.0/interface
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Christofer Dutz <ch...@c-ware.de> 于2018年12月12日周三 上午4:49写道：
>
> Hi all,
>
> So I was curious, checked out the code and ran a build. I had to disable
one test (more details down the document), but in the end I got a
successful build. Congrats on that … some of us – especially one other of
your mentors - know at least one project where it takes a little more
setting up to get it to build ;-)
>
> Here comes a list of things we will need changing for Apache (And or in
order to avoid a lot of problems in the future):
>
>
> * The parent pom in the root has to be set to “org.apache:apache:21”
(or the latest version of that)
> * As far as I know the ASF doesn’t like the “developers” section in
the pom files
> * All non-binary files need the ASF header (Yes … all of them … I
know this will not be the most glorious job, but it has to be done)
> * Module Grafana
> * The parent of this module is not within the project scope. This
quite often causes problems with the maven tooling and when releasing, the
module it will not inherit the settings of the apache parent. This has to
be changed (Try adding the iotdb-parent as parent and adding the missing
configuration options from the spring-boot-starter-parent. This way you
don’t have to redefine all the settings you can no longer inherit from the
iotdb-parent.
> * Module service-rpc and tsfile:
> * You are defining a “release” profile which is obviously intended
for releasing. This will not apply as apache releases are performed with
the “apache-release” profile. So if important stuff is to be done here (it
actually shouldn’t)
> * From a quick look all defined there is also done by the
profile defined in the “apache-release” profile in the “apache” parent pom.
So all except the exclusion for package names including “thrift” could
simply be removed. (Maybe it’s worth configuring the maven-javadoc-plugin
in a pluginManagement section of the root to exclude them in general and
get rid of the “release” profiles in general)
> * Especially the staging part has to be removed as you will be
staging to repository.apache.org instead of the oss sonatype repo.
>
> Things I personally would suggest to change:
>
>
> * Don’t use a property to define the version of the artifacts that
are part of the build (project.version) (The release plugin updates this
for you and believe me I have encountered a lot of problems with this
approach in my last 10 years of managing Maven projects as a consultant)
> * Why are you generating code (thrift) and checking that in? I would
strongly suggest to always generate the code, move the thrift files to
“src/main/thrift” and have the build generate things ot its usual location
“target/generated-sources/thrift” (or something like that … the last part
of the path may differ)
> * General observations:
> * Replace the antrun plugin with the dependencies plugin for
copying the libs to the iotdb/lib directory (antrun is highly deprecated)
> * No need to re-define the license, scm, distribution management
in sub-modules
> * Try to unify the plugin definitions in the root pom.
> * The “release” profile could be causing problems in a real
release (will not be executed as with apache it’s called “apache-release”)
> * You are relying on Java 8 so the Jodatime library is included in
general there is no need to use the external library and the included
classes could be used)
> * It might we worth defining an “example” module which contains
real maven buildable examples (some projects even move them to a separate
examples repository) right now there is no way to automatically detect that
the code has changed and the example no longer matches that API.
> * I wouldn’t produce fat jars (using the assembly plugin) as this
bundles classes in to a library that might be used in another application
that has dependencies to the same libraries but in a different version …
trust me … these are the worst problems to track down.
> * Module iotdb:
> * Have the MANIFEST.MF generated by maven instead.
> * You have a dependency on the antlr3 maven plugin. Instead you
should depend on the antlr-runtime artifact that matches your antlr3
version.
> * What is the “kvmatch-iotdb” artifact referring to? Seems it is
not part of this project.
> * Why are you manually configuring an additional compilation in
the pre-integration-test phase? There is no code in “src/it/java”
> * One test is failing because I’m in a different time zone: Failed
tests: test(cn.edu.tsinghua.iotdb.sql.DateFormatTest):
expected:<…02-01T11:12:35.000+0[1]:00> but
was:<…02-01T11:12:35.000+0[8]:00>
(cn.edu.tsinghua.iotdb.sql.DateFormatTest.test has most references ending
with “+08:00” which makes the tests only run in one time-zone)
> *
cn.edu.tsinghua.iotdb.queryV2.engine.control.OverflowFileStreamManager uses
the deprecated internal SUN API which is not available in Java 11 anymore
> * Module jdbc:
> * You are adding a “interface/thrift” directory to the sources,
but that doesn’t exist.
> * Module service-rpc:
> * Move the interfaces thirft files to “src/main/thrift” and have
the code generated on every build to “target/generated-sources” (default)
> * Don’t check in generated code.
> * Module tsfile_
> * In general the same remarks apply as to the service-rpc module
> * Module spark
> * You have included *.tsfile in your “.gitglobal” but seem to have
forced in a “tsfile”-file in the test resources (Things can get quite messy
in such situations if the file is ever changed, as the IDE will never
detect changes and never commit any changes to that file)
> * Module Hadoop
> * Looks a little unfinished … so I won’t go into details here.
>
> So much for a quick run through the project … If you want, I could be of
assistance with the actual transformation … I know I could probably address
most in half a day and that most people would probably take a lot longer.
After all we want you to learn the Apache way … not to learn setting up
projects ;-) (Other mentors please correct me if I’m wrong)
>
> So … I’ll call it a night or my girlfriend will get angry ;-)
>
> Chris
>
>