You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Sudheesh Katkam <sk...@maprtech.com> on 2015/10/26 22:34:47 UTC
Increased build time and failure?
Hey y’all,
I use “mvn clean install -DskipTests” to build Drill on my Mac. On the latest master, I am getting this error now. Also notice that the build time is ~23 minutes; it used to be ~7 minutes.
@Julien what are the implications of your changes in DRILL-3742?
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Drill Root POM .............................. SUCCESS [02:29 min]
[INFO] tools/Parent Pom ................................... SUCCESS [ 23.473 s]
[INFO] tools/freemarker codegen tooling ................... SUCCESS [01:41 min]
[INFO] Drill Protocol ..................................... SUCCESS [ 34.936 s]
[INFO] Common (Logical Plan, Base expressions) ............ SUCCESS [01:03 min]
[INFO] contrib/Parent Pom ................................. SUCCESS [ 3.533 s]
[INFO] contrib/data/Parent Pom ............................ SUCCESS [ 10.862 s]
[INFO] contrib/data/tpch-sample-data ...................... SUCCESS [ 17.306 s]
[INFO] exec/Parent Pom .................................... SUCCESS [ 3.997 s]
[INFO] exec/Java Execution Engine ......................... SUCCESS [08:16 min]
[INFO] exec/JDBC Driver using dependencies ................ SUCCESS [ 52.674 s]
[INFO] JDBC JAR with all dependencies ..................... SUCCESS [01:17 min]
[INFO] contrib/mongo-storage-plugin ....................... SUCCESS [ 53.952 s]
[INFO] contrib/hbase-storage-plugin ....................... SUCCESS [ 51.397 s]
[INFO] contrib/jdbc-storage-plugin ........................ SUCCESS [ 14.904 s]
[INFO] contrib/hive-storage-plugin/Parent Pom ............. SUCCESS [ 1.964 s]
[INFO] contrib/hive-storage-plugin/hive-exec-shaded ....... SUCCESS [ 41.461 s]
[INFO] contrib/hive-storage-plugin/core ................... SUCCESS [01:34 min]
[INFO] contrib/drill-gis-plugin ........................... SUCCESS [ 41.361 s]
[INFO] Packaging and Distribution Assembly ................ SKIPPED
[INFO] contrib/sqlline .................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22:51 min
[INFO] Finished at: 2015-10-26T14:22:50-07:00
[INFO] Final Memory: 139M/1714M
[INFO] ------------------------------------------------------------------------
[ERROR] PermGen space -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
Thank you,
Sudheesh
Re: Increased build time and failure?
Posted by Jacques Nadeau <ja...@dremio.com>.
Some things to note:
FMPP does an incremental build now (not replacing the generated files if
they haven't changed, thus meaning javac also avoids compilation). This
makes an incremental build way faster. (The enhancement also avoids double
copying/compiling the generated classes.) Since you did clean, you won't
see that improvement.
The bigger win on build time is with tests running: startup and shutdown of
Drillbits is faster (due to classpath scanning savings, function
implementation registry creation and reduced shutdown delays in netty).
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Mon, Oct 26, 2015 at 5:45 PM, Sudheesh Katkam <sk...@maprtech.com>
wrote:
> I added MAVEN_OPTS and now the build time completes successfully in ~7
> minutes on multiple runs. The commit message says “reduces build time by
> 30%”, but I do not see any improvement. What am I doing wrong?
>
> - Sudheesh
>
> > On Oct 26, 2015, at 2:51 PM, Jacques Nadeau <ja...@dremio.com> wrote:
> >
> > Have you seen this multiple times? The first time you may build with the
> > changes I merged today include downloading a new hadoop version (which
> > takes forever the first time).
> >
> > In general, the build without tests is substantially faster than before.
> I
> > have seen PermGen issues and have set my environment variables to provide
> > more permgen. I use MAVEN_OPTS="-Xmx1g -XX:MaxPermSize=256m".
> >
> >
> >
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Mon, Oct 26, 2015 at 2:34 PM, Sudheesh Katkam <sk...@maprtech.com>
> > wrote:
> >
> >> Hey y’all,
> >>
> >> I use “mvn clean install -DskipTests” to build Drill on my Mac. On the
> >> latest master, I am getting this error now. Also notice that the build
> time
> >> is ~23 minutes; it used to be ~7 minutes.
> >>
> >> @Julien what are the implications of your changes in DRILL-3742?
> >>
> >> [INFO]
> >> ------------------------------------------------------------------------
> >> [INFO] Reactor Summary:
> >> [INFO]
> >> [INFO] Apache Drill Root POM .............................. SUCCESS
> [02:29
> >> min]
> >> [INFO] tools/Parent Pom ................................... SUCCESS [
> >> 23.473 s]
> >> [INFO] tools/freemarker codegen tooling ................... SUCCESS
> [01:41
> >> min]
> >> [INFO] Drill Protocol ..................................... SUCCESS [
> >> 34.936 s]
> >> [INFO] Common (Logical Plan, Base expressions) ............ SUCCESS
> [01:03
> >> min]
> >> [INFO] contrib/Parent Pom ................................. SUCCESS [
> >> 3.533 s]
> >> [INFO] contrib/data/Parent Pom ............................ SUCCESS [
> >> 10.862 s]
> >> [INFO] contrib/data/tpch-sample-data ...................... SUCCESS [
> >> 17.306 s]
> >> [INFO] exec/Parent Pom .................................... SUCCESS [
> >> 3.997 s]
> >> [INFO] exec/Java Execution Engine ......................... SUCCESS
> [08:16
> >> min]
> >> [INFO] exec/JDBC Driver using dependencies ................ SUCCESS [
> >> 52.674 s]
> >> [INFO] JDBC JAR with all dependencies ..................... SUCCESS
> [01:17
> >> min]
> >> [INFO] contrib/mongo-storage-plugin ....................... SUCCESS [
> >> 53.952 s]
> >> [INFO] contrib/hbase-storage-plugin ....................... SUCCESS [
> >> 51.397 s]
> >> [INFO] contrib/jdbc-storage-plugin ........................ SUCCESS [
> >> 14.904 s]
> >> [INFO] contrib/hive-storage-plugin/Parent Pom ............. SUCCESS [
> >> 1.964 s]
> >> [INFO] contrib/hive-storage-plugin/hive-exec-shaded ....... SUCCESS [
> >> 41.461 s]
> >> [INFO] contrib/hive-storage-plugin/core ................... SUCCESS
> [01:34
> >> min]
> >> [INFO] contrib/drill-gis-plugin ........................... SUCCESS [
> >> 41.361 s]
> >> [INFO] Packaging and Distribution Assembly ................ SKIPPED
> >> [INFO] contrib/sqlline .................................... SKIPPED
> >> [INFO]
> >> ------------------------------------------------------------------------
> >> [INFO] BUILD SUCCESS
> >> [INFO]
> >> ------------------------------------------------------------------------
> >> [INFO] Total time: 22:51 min
> >> [INFO] Finished at: 2015-10-26T14:22:50-07:00
> >> [INFO] Final Memory: 139M/1714M
> >> [INFO]
> >> ------------------------------------------------------------------------
> >> [ERROR] PermGen space -> [Help 1]
> >> [ERROR]
> >> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> >> -e switch.
> >> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> >> [ERROR]
> >> [ERROR] For more information about the errors and possible solutions,
> >> please read the following articles:
> >> [ERROR] [Help 1]
> >> http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
> >>
> >> Thank you,
> >> Sudheesh
>
>
Re: Increased build time and failure?
Posted by Sudheesh Katkam <sk...@maprtech.com>.
I added MAVEN_OPTS and now the build time completes successfully in ~7 minutes on multiple runs. The commit message says “reduces build time by 30%”, but I do not see any improvement. What am I doing wrong?
- Sudheesh
> On Oct 26, 2015, at 2:51 PM, Jacques Nadeau <ja...@dremio.com> wrote:
>
> Have you seen this multiple times? The first time you may build with the
> changes I merged today include downloading a new hadoop version (which
> takes forever the first time).
>
> In general, the build without tests is substantially faster than before. I
> have seen PermGen issues and have set my environment variables to provide
> more permgen. I use MAVEN_OPTS="-Xmx1g -XX:MaxPermSize=256m".
>
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Oct 26, 2015 at 2:34 PM, Sudheesh Katkam <sk...@maprtech.com>
> wrote:
>
>> Hey y’all,
>>
>> I use “mvn clean install -DskipTests” to build Drill on my Mac. On the
>> latest master, I am getting this error now. Also notice that the build time
>> is ~23 minutes; it used to be ~7 minutes.
>>
>> @Julien what are the implications of your changes in DRILL-3742?
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] Apache Drill Root POM .............................. SUCCESS [02:29
>> min]
>> [INFO] tools/Parent Pom ................................... SUCCESS [
>> 23.473 s]
>> [INFO] tools/freemarker codegen tooling ................... SUCCESS [01:41
>> min]
>> [INFO] Drill Protocol ..................................... SUCCESS [
>> 34.936 s]
>> [INFO] Common (Logical Plan, Base expressions) ............ SUCCESS [01:03
>> min]
>> [INFO] contrib/Parent Pom ................................. SUCCESS [
>> 3.533 s]
>> [INFO] contrib/data/Parent Pom ............................ SUCCESS [
>> 10.862 s]
>> [INFO] contrib/data/tpch-sample-data ...................... SUCCESS [
>> 17.306 s]
>> [INFO] exec/Parent Pom .................................... SUCCESS [
>> 3.997 s]
>> [INFO] exec/Java Execution Engine ......................... SUCCESS [08:16
>> min]
>> [INFO] exec/JDBC Driver using dependencies ................ SUCCESS [
>> 52.674 s]
>> [INFO] JDBC JAR with all dependencies ..................... SUCCESS [01:17
>> min]
>> [INFO] contrib/mongo-storage-plugin ....................... SUCCESS [
>> 53.952 s]
>> [INFO] contrib/hbase-storage-plugin ....................... SUCCESS [
>> 51.397 s]
>> [INFO] contrib/jdbc-storage-plugin ........................ SUCCESS [
>> 14.904 s]
>> [INFO] contrib/hive-storage-plugin/Parent Pom ............. SUCCESS [
>> 1.964 s]
>> [INFO] contrib/hive-storage-plugin/hive-exec-shaded ....... SUCCESS [
>> 41.461 s]
>> [INFO] contrib/hive-storage-plugin/core ................... SUCCESS [01:34
>> min]
>> [INFO] contrib/drill-gis-plugin ........................... SUCCESS [
>> 41.361 s]
>> [INFO] Packaging and Distribution Assembly ................ SKIPPED
>> [INFO] contrib/sqlline .................................... SKIPPED
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESS
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 22:51 min
>> [INFO] Finished at: 2015-10-26T14:22:50-07:00
>> [INFO] Final Memory: 139M/1714M
>> [INFO]
>> ------------------------------------------------------------------------
>> [ERROR] PermGen space -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>> -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1]
>> http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
>>
>> Thank you,
>> Sudheesh
Re: Increased build time and failure?
Posted by Jacques Nadeau <ja...@dremio.com>.
Have you seen this multiple times? The first time you may build with the
changes I merged today include downloading a new hadoop version (which
takes forever the first time).
In general, the build without tests is substantially faster than before. I
have seen PermGen issues and have set my environment variables to provide
more permgen. I use MAVEN_OPTS="-Xmx1g -XX:MaxPermSize=256m".
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Mon, Oct 26, 2015 at 2:34 PM, Sudheesh Katkam <sk...@maprtech.com>
wrote:
> Hey y’all,
>
> I use “mvn clean install -DskipTests” to build Drill on my Mac. On the
> latest master, I am getting this error now. Also notice that the build time
> is ~23 minutes; it used to be ~7 minutes.
>
> @Julien what are the implications of your changes in DRILL-3742?
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Drill Root POM .............................. SUCCESS [02:29
> min]
> [INFO] tools/Parent Pom ................................... SUCCESS [
> 23.473 s]
> [INFO] tools/freemarker codegen tooling ................... SUCCESS [01:41
> min]
> [INFO] Drill Protocol ..................................... SUCCESS [
> 34.936 s]
> [INFO] Common (Logical Plan, Base expressions) ............ SUCCESS [01:03
> min]
> [INFO] contrib/Parent Pom ................................. SUCCESS [
> 3.533 s]
> [INFO] contrib/data/Parent Pom ............................ SUCCESS [
> 10.862 s]
> [INFO] contrib/data/tpch-sample-data ...................... SUCCESS [
> 17.306 s]
> [INFO] exec/Parent Pom .................................... SUCCESS [
> 3.997 s]
> [INFO] exec/Java Execution Engine ......................... SUCCESS [08:16
> min]
> [INFO] exec/JDBC Driver using dependencies ................ SUCCESS [
> 52.674 s]
> [INFO] JDBC JAR with all dependencies ..................... SUCCESS [01:17
> min]
> [INFO] contrib/mongo-storage-plugin ....................... SUCCESS [
> 53.952 s]
> [INFO] contrib/hbase-storage-plugin ....................... SUCCESS [
> 51.397 s]
> [INFO] contrib/jdbc-storage-plugin ........................ SUCCESS [
> 14.904 s]
> [INFO] contrib/hive-storage-plugin/Parent Pom ............. SUCCESS [
> 1.964 s]
> [INFO] contrib/hive-storage-plugin/hive-exec-shaded ....... SUCCESS [
> 41.461 s]
> [INFO] contrib/hive-storage-plugin/core ................... SUCCESS [01:34
> min]
> [INFO] contrib/drill-gis-plugin ........................... SUCCESS [
> 41.361 s]
> [INFO] Packaging and Distribution Assembly ................ SKIPPED
> [INFO] contrib/sqlline .................................... SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 22:51 min
> [INFO] Finished at: 2015-10-26T14:22:50-07:00
> [INFO] Final Memory: 139M/1714M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] PermGen space -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
>
> Thank you,
> Sudheesh