You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2020/11/13 15:48:20 UTC
[GitHub] [drill] nielsbasjes opened a new pull request #2115: [DRILL-7803] Docker based developer build environment
nielsbasjes opened a new pull request #2115:
URL: https://github.com/apache/drill/pull/2115
# [DRILL-7803](https://issues.apache.org/jira/browse/DRILL-7803): Docker based developer build environment
## Description
I was asked to review a change in Drill ( https://github.com/apache/drill/pull/2112 ).
When I tried to build the software it turns out that a lot of the build fails over the fact that I have Java 11 installed on my machine by default.
To "fix" this I copied/changed the Docker based build environment I initially created for Hadoop (many years ago) and recently updated for Apache Beam ( https://github.com/apache/beam/pull/13308 | [BEAM-10891](https://issues.apache.org/jira/browse/BEAM-10891) )
Because I think it makes sense to have this helper more easily available I put it in this pull request.
This is just an additional tool to make setting up the build environment for Drill a lot easier.
Options
## Documentation
Only a small addition for developers who want to build the sourcecode locally.
## Testing
On a "empty" (only git and docker are needed) Linux machine with Docker installed (i.e. RedHat/CentOS probably won't work) in the root of the Drill source tree run the `start-build-env.sh` script.
After a while (depends on machine and network speed) a bash command prompt should appear with all needed tools to build Apache Drill installed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre merged pull request #2115: [DRILL-7803] Docker based developer build environment
Posted by GitBox <gi...@apache.org>.
cgivre merged pull request #2115:
URL: https://github.com/apache/drill/pull/2115
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] nielsbasjes commented on pull request #2115: [DRILL-7803] Docker based developer build environment
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2115:
URL: https://github.com/apache/drill/pull/2115#issuecomment-727575725
Should be fixed now.
Added libncurses5 as indicated here https://github.com/wix/wix-embedded-mysql/issues/116
Now this test passes when run in isolation.
Going to do a full rerun now to be sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] nielsbasjes commented on pull request #2115: [DRILL-7803] Docker based developer build environment
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2115:
URL: https://github.com/apache/drill/pull/2115#issuecomment-727563553
NOTE: Doing a full test run to ensure it is correct now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] nielsbasjes commented on pull request #2115: [DRILL-7803] Docker based developer build environment
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2115:
URL: https://github.com/apache/drill/pull/2115#issuecomment-727583459
It worked on my machine
```
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Drill Root POM 1.19.0-SNAPSHOT:
[INFO]
[INFO] Apache Drill Root POM .............................. SUCCESS [ 6.664 s]
[INFO] tools/Parent Pom ................................... SUCCESS [ 0.495 s]
[INFO] tools/freemarker codegen tooling ................... SUCCESS [ 3.996 s]
[INFO] Drill Protocol ..................................... SUCCESS [ 5.408 s]
[INFO] Common (Logical Plan, Base expressions) ............ SUCCESS [ 6.158 s]
[INFO] Logical Plan, Base expressions ..................... SUCCESS [ 6.717 s]
[INFO] exec/Parent Pom .................................... SUCCESS [ 1.274 s]
[INFO] exec/memory/Parent Pom ............................. SUCCESS [ 0.283 s]
[INFO] exec/memory/base ................................... SUCCESS [ 4.175 s]
[INFO] exec/rpc ........................................... SUCCESS [ 2.831 s]
[INFO] exec/Vectors ....................................... SUCCESS [01:35 min]
[INFO] contrib/Parent Pom ................................. SUCCESS [ 0.514 s]
[INFO] contrib/data/Parent Pom ............................ SUCCESS [ 0.315 s]
[INFO] contrib/data/tpch-sample-data ...................... SUCCESS [ 1.862 s]
[INFO] metastore/Parent Pom ............................... SUCCESS [ 0.366 s]
[INFO] metastore/Drill Metastore API ...................... SUCCESS [ 6.318 s]
[INFO] metastore/Drill Iceberg Metastore .................. SUCCESS [ 11.373 s]
[INFO] exec/Java Execution Engine ......................... SUCCESS [22:56 min]
[INFO] exec/JDBC Driver using dependencies ................ SUCCESS [01:12 min]
[INFO] JDBC JAR with all dependencies ..................... SUCCESS [ 22.328 s]
[INFO] Drill-on-YARN ...................................... SUCCESS [ 7.593 s]
[INFO] metastore/Drill RDBMS Metastore .................... SUCCESS [ 9.037 s]
[INFO] contrib/kudu-storage-plugin ........................ SUCCESS [ 6.361 s]
[INFO] contrib/http-storage-plugin ........................ SUCCESS [ 13.912 s]
[INFO] contrib/opentsdb-storage-plugin .................... SUCCESS [ 12.608 s]
[INFO] contrib/mongo-storage-plugin ....................... SUCCESS [ 22.082 s]
[INFO] contrib/hbase-storage-plugin ....................... SUCCESS [01:29 min]
[INFO] contrib/jdbc-storage-plugin ........................ SUCCESS [01:00 min]
[INFO] contrib/hive-storage-plugin/Parent Pom ............. SUCCESS [ 0.423 s]
[INFO] contrib/hive-storage-plugin/hive-exec-shaded ....... SUCCESS [ 19.934 s]
[INFO] contrib/hive-storage-plugin/core ................... SUCCESS [03:34 min]
[INFO] contrib/kafka-storage-plugin ....................... SUCCESS [03:53 min]
[INFO] contrib/drill-udfs ................................. SUCCESS [ 23.610 s]
[INFO] contrib/format-syslog .............................. SUCCESS [ 10.682 s]
[INFO] contrib/format-hdf5 ................................ SUCCESS [ 14.429 s]
[INFO] contrib/format-spss ................................ SUCCESS [ 11.198 s]
[INFO] contrib/ltsv-format-plugin ......................... SUCCESS [ 10.541 s]
[INFO] contrib/format-esri ................................ SUCCESS [ 11.475 s]
[INFO] contrib/format-excel ............................... SUCCESS [ 13.508 s]
[INFO] contrib/druid-storage-plugin ....................... SUCCESS [ 7.319 s]
[INFO] Packaging and Distribution Assembly ................ SUCCESS [ 25.364 s]
[INFO] contrib/mapr-format-plugin ......................... SUCCESS [ 6.800 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 40:49 min
[INFO] Finished at: 2020-11-15T15:00:32Z
[INFO] ------------------------------------------------------------------------
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] nielsbasjes edited a comment on pull request #2115: [DRILL-7803] Docker based developer build environment
Posted by GitBox <gi...@apache.org>.
nielsbasjes edited a comment on pull request #2115:
URL: https://github.com/apache/drill/pull/2115#issuecomment-727575725
Should be fixed now.
Added libncurses5 as indicated here https://github.com/wix/wix-embedded-mysql/issues/116
Now this test passes when run in isolation.
Going to do a full rerun now to be sure.
Doing `mvn clean install -Drat.skip=false -Dlicense.skip=false`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] nielsbasjes commented on pull request #2115: [DRILL-7803] Docker based developer build environment
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2115:
URL: https://github.com/apache/drill/pull/2115#issuecomment-727577144
Added MAVEN_OPTS settings from the Release documentation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] nielsbasjes commented on pull request #2115: [DRILL-7803] Docker based developer build environment
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2115:
URL: https://github.com/apache/drill/pull/2115#issuecomment-727572723
At this point the entire build passes with one exception: TestJdbcPluginWithMySQLIT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on pull request #2115: [DRILL-7803] Docker based developer build environment
Posted by GitBox <gi...@apache.org>.
cgivre commented on pull request #2115:
URL: https://github.com/apache/drill/pull/2115#issuecomment-727276453
@nielsbasjes
Thanks for the contribution and the code review. If you can fix the license headers, I'll approve this. (Drill is picky about them)
Best,
-- C
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] nielsbasjes commented on pull request #2115: [DRILL-7803] Docker based developer build environment
Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2115:
URL: https://github.com/apache/drill/pull/2115#issuecomment-727571164
Having trouble getting the JDBC test to pass that uses the EmbeddedMySQL.
The required dependencies are there but it still fails during the setup of the test.
```
[INFO] Running org.apache.drill.exec.store.jdbc.TestJdbcPluginWithMySQLIT
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! - in org.apache.drill.exec.store.jdbc.TestJdbcPluginWithMySQLIT
[ERROR] org.apache.drill.exec.store.jdbc.TestJdbcPluginWithMySQLIT Time elapsed: 0.001 s <<< ERROR!
com.wix.mysql.exceptions.CommandFailedException: Command 'CREATE USER 'mysqlUser'@'%' IDENTIFIED BY 'mysqlPass';' on schema 'information_schema' failed with message 'Stream closed'
at org.apache.drill.exec.store.jdbc.TestJdbcPluginWithMySQLIT.initMysql(TestJdbcPluginWithMySQLIT.java:70)
Caused by: java.io.IOException: Stream closed
at org.apache.drill.exec.store.jdbc.TestJdbcPluginWithMySQLIT.initMysql(TestJdbcPluginWithMySQLIT.java:70)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org