You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jean-Marc Spaggiari <je...@spaggiari.org> on 2013/02/25 19:42:06 UTC
Build performances
Hi,
Buliding HBase locally is taking 1h for me:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 59:36.659s
[INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
[INFO] Final Memory: 64M/593M
[INFO] ------------------------------------------------------------------------
But from that our, it seems to be spending most of the time doing that:
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @ hbase ---
[INFO] Reading assembly descriptor: src/assembly/all.xml
[INFO] Copying files to
/home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0 too,
and disks are almost not working.
I have started the build with mvn clean site install assembly:assembly
-DskipTests -Prelease.
Is that really spending all this time copŷing files? I will move the
build to an SSD drive and retry, but is there anyway to reduce the
build time?
Thanks,
JM
Re: Build performances
Posted by Stack <st...@duboce.net>.
I would be interested in folks experience running it in production.
St.Ack
On Tue, Feb 26, 2013 at 5:36 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Stack,
>
> 3 tests failed and 1271 passed.
>
> So it seems to be pretty stable with openJDK1.7.
>
> I retried and got only 2 failed.
>
> In both cases testDeleteEditUnknownColumnFamilyAndOrTable was on the list.
>
> I'm retrying another time, but for small tests, I will say that open
> JDK is stable enought.
>
> JM
>
> 2013/2/25 Jean-Marc Spaggiari <je...@spaggiari.org>:
> > Hi Stack,
> >
> > When I'm testing locally with VMs I'm using openJDK for the standalone
> > mode so I don't have to install any additionnal JVM. And it's working
> > for my needs. I have not faced any issue so far.
> >
> > I just started the entire test suite with OpenJDK1.7. I will keep you
> posted.
> >
> > JM
> >
> > 2013/2/25 Stack <st...@duboce.net>:
> >> Does hbase work on openjdk?
> >> St.Ack
> >>
> >>
> >> On Mon, Feb 25, 2013 at 12:54 PM, Jean-Marc Spaggiari <
> >> jean-marc@spaggiari.org> wrote:
> >>
> >>> Hi Stack,
> >>>
> >>> Thanks for pointing to stracing.
> >>>
> >>> However, I fixed my issue...
> >>>
> >>> Initially I was running with that:
> >>> Java version: 1.6.0_24, vendor: Sun Microsystems Inc.
> >>> Java home: /usr/lib/jvm/java-6-openjdk-amd64/jre
> >>>
> >>>
> >>> With "export JAVA_HOME=/usr/local/jdk1.7.0_13/":
> >>> [INFO]
> >>>
> ------------------------------------------------------------------------
> >>> [INFO] BUILD SUCCESS
> >>> [INFO]
> >>>
> ------------------------------------------------------------------------
> >>> [INFO] Total time: 5:37.573s
> >>> [INFO] Finished at: Mon Feb 25 15:50:26 EST 2013
> >>> [INFO] Final Memory: 62M/3020M
> >>> [INFO]
> >>>
> ------------------------------------------------------------------------
> >>>
> >>> Less than 6 minutes. 10 times faster!!!!
> >>>
> >>> So I don't really know why it was so slow with the 1.6 OpenJDK, but
> >>> it's now running way faster!
> >>>
> >>> Thanks,
> >>>
> >>> JM
> >>>
> >>> 2013/2/25 Stack <st...@duboce.net>:
> >>> > Try stracing the process [1]?
> >>> > St.Ack
> >>> >
> >>> > 1. http://www.cyberciti.biz/tips/linux-strace-command-examples.html
> >>> >
> >>> >
> >>> > On Mon, Feb 25, 2013 at 12:09 PM, Jean-Marc Spaggiari <
> >>> > jean-marc@spaggiari.org> wrote:
> >>> >
> >>> >> When it's stuck on "Copying", there is nothing more displayed.
> >>> >>
> >>> >> Before that, it's verbose.
> >>> >>
> >>> >> [DEBUG] Adding file-set in: /mnt/security to archive location:
> >>> >> hbase-0.94.6-SNAPSHOT/security/
> >>> >> [DEBUG] FileSet[conf/] dir perms: 47777 file perms: 107777
> >>> >> [DEBUG] The archive base directory is 'null'
> >>> >> [DEBUG] NOT reformatting any files in /mnt/conf
> >>> >> [DEBUG] Adding file-set from directory: '/mnt/conf'
> >>> >> assembly output directory is: 'conf/'
> >>> >> [DEBUG] Adding file-set in: /mnt/conf to archive location:
> >>> >> hbase-0.94.6-SNAPSHOT/conf/
> >>> >> [DEBUG] FileSet[bin/] dir perms: 47777 file perms: 107777
> >>> >> [DEBUG] The archive base directory is 'null'
> >>> >> [DEBUG] NOT reformatting any files in /mnt/bin
> >>> >> [DEBUG] Adding file-set from directory: '/mnt/bin'
> >>> >> assembly output directory is: 'bin/'
> >>> >> [DEBUG] Adding file-set in: /mnt/bin to archive location:
> >>> >> hbase-0.94.6-SNAPSHOT/bin/
> >>> >> [DEBUG] FileSet[lib/ruby/] dir perms: 47777 file perms: 107777
> >>> >> [DEBUG] The archive base directory is 'null'
> >>> >> [DEBUG] NOT reformatting any files in /mnt/src/main/ruby
> >>> >> [DEBUG] Adding file-set from directory: '/mnt/src/main/ruby'
> >>> >> assembly output directory is: 'lib/ruby/'
> >>> >> [DEBUG] Adding file-set in: /mnt/src/main/ruby to archive location:
> >>> >> hbase-0.94.6-SNAPSHOT/lib/ruby/
> >>> >> [DEBUG] FileSet[] dir perms: 47777 file perms: 107777
> >>> >> [DEBUG] The archive base directory is 'null'
> >>> >> [DEBUG] NOT reformatting any files in /mnt/target
> >>> >> [DEBUG] Adding file-set from directory: '/mnt/target'
> >>> >> assembly output directory is: ''
> >>> >> [DEBUG] Adding file-set in: /mnt/target to archive location:
> >>> >> hbase-0.94.6-SNAPSHOT/
> >>> >> [DEBUG] FileSet[hbase-webapps/] dir perms: 47777 file perms: 107777
> >>> >> [DEBUG] The archive base directory is 'null'
> >>> >> [DEBUG] NOT reformatting any files in /mnt/target/hbase-webapps
> >>> >> [DEBUG] Adding file-set from directory: '/mnt/target/hbase-webapps'
> >>> >> assembly output directory is: 'hbase-webapps/'
> >>> >> [DEBUG] Adding file-set in: /mnt/target/hbase-webapps to archive
> >>> >> location: hbase-0.94.6-SNAPSHOT/hbase-webapps/
> >>> >> [DEBUG] FileSet[docs/] dir perms: 47777 file perms: 107777
> >>> >> [DEBUG] The archive base directory is 'null'
> >>> >> [DEBUG] NOT reformatting any files in /mnt/target/site
> >>> >> [DEBUG] Adding file-set from directory: '/mnt/target/site'
> >>> >> assembly output directory is: 'docs/'
> >>> >> [DEBUG] Adding file-set in: /mnt/target/site to archive location:
> >>> >> hbase-0.94.6-SNAPSHOT/docs/
> >>> >> [DEBUG] FileSet[sbin/] dir perms: 47777 file perms: 107777
> >>> >> [DEBUG] The archive base directory is 'null'
> >>> >> [DEBUG] NOT reformatting any files in /mnt/src/packages
> >>> >> [DEBUG] Adding file-set from directory: '/mnt/src/packages'
> >>> >> assembly output directory is: 'sbin/'
> >>> >> [DEBUG] Adding file-set in: /mnt/src/packages to archive location:
> >>> >> hbase-0.94.6-SNAPSHOT/sbin/
> >>> >> [INFO] Copying files to /mnt/target/hbase-0.94.6-SNAPSHOT
> >>> >>
> >>> >>
> >>> >> But now it's stuck again on the last line, not giving more
> >>> >> information. I killed it and I moved everything on tmpfs but it's
> not
> >>> >> faster. I have nothing else running.
> >>> >>
> >>> >> How long should it normaly take?
> >>> >>
> >>> >> Entire log are avaiable there: http://www.spaggiari.org/log.txt
> >>> >>
> >>> >> JM
> >>> >>
> >>> >> 2013/2/25 Stack <st...@duboce.net>:
> >>> >> > Following on from Andrew, does the debug show you anything?
> (Copying
> >>> >> > individual files?)
> >>> >> > St.Ack
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Feb 25, 2013 at 11:09 AM, Jean-Marc Spaggiari <
> >>> >> > jean-marc@spaggiari.org> wrote:
> >>> >> >
> >>> >> >> Hi Andy.
> >>> >> >>
> >>> >> >> With DEBUG enabled, what seems to be slow is that:
> >>> >> >> [DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1
> with
> >>> >> >> file:
> >>> >> >>
> >>> >>
> >>>
> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
> >>> >> >> to assembly location: lib/commons-math-2.1.jar.
> >>> >> >> [DEBUG] Adding file:
> >>> >> >>
> >>> >> >>
> >>> >>
> >>>
> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
> >>> >> >> to archive location:
> hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
> >>> >> >> [DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1 with
> file:
> >>> >> >>
> >>> >> >>
> >>> >>
> >>>
> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
> >>> >> >> to assembly location: lib/commons-net-1.4.1.jar.
> >>> >> >> [DEBUG] Adding file:
> >>> >> >>
> >>> >> >>
> >>> >>
> >>>
> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
> >>> >> >> to archive location:
> hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
> >>> >> >> About 3 minutes, displaying one line every 1 to 2 seconds.
> >>> >> >>
> >>> >> >> Then this is taking a lot of time:
> >>> >> >> [INFO] Copying files to
> /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
> >>> >> >>
> >>> >> >> It's now running for 10 minutes and seems it's doing nothing.
> This is
> >>> >> >> what I saw without the debug enabled. I'm building on an SSD
> drive.
> >>> >> >> It's only about 250MB. I can even build it in memory if
> required. But
> >>> >> >> I don't think it's because of the disk.
> >>> >> >>
> >>> >> >> JM
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> 2013/2/25 Andrew Purtell <ap...@apache.org>:
> >>> >> >> > If you run Maven with debug logging enabled, do you see it
> taking
> >>> ~3
> >>> >> >> > seconds per dependency jar to decide if it wants to include it
> in
> >>> the
> >>> >> >> > assembly or not?
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
> >>> >> >> > jean-marc@spaggiari.org> wrote:
> >>> >> >> >
> >>> >> >> >> Hi,
> >>> >> >> >>
> >>> >> >> >> Buliding HBase locally is taking 1h for me:
> >>> >> >> >> [INFO]
> >>> >> >> >>
> >>> >>
> ------------------------------------------------------------------------
> >>> >> >> >> [INFO] BUILD SUCCESS
> >>> >> >> >> [INFO]
> >>> >> >> >>
> >>> >>
> ------------------------------------------------------------------------
> >>> >> >> >> [INFO] Total time: 59:36.659s
> >>> >> >> >> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
> >>> >> >> >> [INFO] Final Memory: 64M/593M
> >>> >> >> >> [INFO]
> >>> >> >> >>
> >>> >>
> ------------------------------------------------------------------------
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >> But from that our, it seems to be spending most of the time
> doing
> >>> >> that:
> >>> >> >> >> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @
> >>> hbase
> >>> >> ---
> >>> >> >> >> [INFO] Reading assembly descriptor: src/assembly/all.xml
> >>> >> >> >> [INFO] Copying files to
> >>> >> >> >>
> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
> >>> >> >> >>
> >>> >> >> >> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is
> almost 0
> >>> too,
> >>> >> >> >> and disks are almost not working.
> >>> >> >> >>
> >>> >> >> >> I have started the build with mvn clean site install
> >>> >> assembly:assembly
> >>> >> >> >> -DskipTests -Prelease.
> >>> >> >> >>
> >>> >> >> >> Is that really spending all this time copŷing files? I will
> move
> >>> the
> >>> >> >> >> build to an SSD drive and retry, but is there anyway to
> reduce the
> >>> >> >> >> build time?
> >>> >> >> >>
> >>> >> >> >> Thanks,
> >>> >> >> >>
> >>> >> >> >> JM
> >>> >> >> >>
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > --
> >>> >> >> > Best regards,
> >>> >> >> >
> >>> >> >> > - Andy
> >>> >> >> >
> >>> >> >> > Problems worthy of attack prove their worth by hitting back. -
> Piet
> >>> >> Hein
> >>> >> >> > (via Tom White)
> >>> >> >>
> >>> >>
> >>>
>
Re: Build performances
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Stack,
3 tests failed and 1271 passed.
So it seems to be pretty stable with openJDK1.7.
I retried and got only 2 failed.
In both cases testDeleteEditUnknownColumnFamilyAndOrTable was on the list.
I'm retrying another time, but for small tests, I will say that open
JDK is stable enought.
JM
2013/2/25 Jean-Marc Spaggiari <je...@spaggiari.org>:
> Hi Stack,
>
> When I'm testing locally with VMs I'm using openJDK for the standalone
> mode so I don't have to install any additionnal JVM. And it's working
> for my needs. I have not faced any issue so far.
>
> I just started the entire test suite with OpenJDK1.7. I will keep you posted.
>
> JM
>
> 2013/2/25 Stack <st...@duboce.net>:
>> Does hbase work on openjdk?
>> St.Ack
>>
>>
>> On Mon, Feb 25, 2013 at 12:54 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> Hi Stack,
>>>
>>> Thanks for pointing to stracing.
>>>
>>> However, I fixed my issue...
>>>
>>> Initially I was running with that:
>>> Java version: 1.6.0_24, vendor: Sun Microsystems Inc.
>>> Java home: /usr/lib/jvm/java-6-openjdk-amd64/jre
>>>
>>>
>>> With "export JAVA_HOME=/usr/local/jdk1.7.0_13/":
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 5:37.573s
>>> [INFO] Finished at: Mon Feb 25 15:50:26 EST 2013
>>> [INFO] Final Memory: 62M/3020M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> Less than 6 minutes. 10 times faster!!!!
>>>
>>> So I don't really know why it was so slow with the 1.6 OpenJDK, but
>>> it's now running way faster!
>>>
>>> Thanks,
>>>
>>> JM
>>>
>>> 2013/2/25 Stack <st...@duboce.net>:
>>> > Try stracing the process [1]?
>>> > St.Ack
>>> >
>>> > 1. http://www.cyberciti.biz/tips/linux-strace-command-examples.html
>>> >
>>> >
>>> > On Mon, Feb 25, 2013 at 12:09 PM, Jean-Marc Spaggiari <
>>> > jean-marc@spaggiari.org> wrote:
>>> >
>>> >> When it's stuck on "Copying", there is nothing more displayed.
>>> >>
>>> >> Before that, it's verbose.
>>> >>
>>> >> [DEBUG] Adding file-set in: /mnt/security to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/security/
>>> >> [DEBUG] FileSet[conf/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/conf
>>> >> [DEBUG] Adding file-set from directory: '/mnt/conf'
>>> >> assembly output directory is: 'conf/'
>>> >> [DEBUG] Adding file-set in: /mnt/conf to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/conf/
>>> >> [DEBUG] FileSet[bin/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/bin
>>> >> [DEBUG] Adding file-set from directory: '/mnt/bin'
>>> >> assembly output directory is: 'bin/'
>>> >> [DEBUG] Adding file-set in: /mnt/bin to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/bin/
>>> >> [DEBUG] FileSet[lib/ruby/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/src/main/ruby
>>> >> [DEBUG] Adding file-set from directory: '/mnt/src/main/ruby'
>>> >> assembly output directory is: 'lib/ruby/'
>>> >> [DEBUG] Adding file-set in: /mnt/src/main/ruby to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/lib/ruby/
>>> >> [DEBUG] FileSet[] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/target
>>> >> [DEBUG] Adding file-set from directory: '/mnt/target'
>>> >> assembly output directory is: ''
>>> >> [DEBUG] Adding file-set in: /mnt/target to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/
>>> >> [DEBUG] FileSet[hbase-webapps/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/target/hbase-webapps
>>> >> [DEBUG] Adding file-set from directory: '/mnt/target/hbase-webapps'
>>> >> assembly output directory is: 'hbase-webapps/'
>>> >> [DEBUG] Adding file-set in: /mnt/target/hbase-webapps to archive
>>> >> location: hbase-0.94.6-SNAPSHOT/hbase-webapps/
>>> >> [DEBUG] FileSet[docs/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/target/site
>>> >> [DEBUG] Adding file-set from directory: '/mnt/target/site'
>>> >> assembly output directory is: 'docs/'
>>> >> [DEBUG] Adding file-set in: /mnt/target/site to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/docs/
>>> >> [DEBUG] FileSet[sbin/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/src/packages
>>> >> [DEBUG] Adding file-set from directory: '/mnt/src/packages'
>>> >> assembly output directory is: 'sbin/'
>>> >> [DEBUG] Adding file-set in: /mnt/src/packages to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/sbin/
>>> >> [INFO] Copying files to /mnt/target/hbase-0.94.6-SNAPSHOT
>>> >>
>>> >>
>>> >> But now it's stuck again on the last line, not giving more
>>> >> information. I killed it and I moved everything on tmpfs but it's not
>>> >> faster. I have nothing else running.
>>> >>
>>> >> How long should it normaly take?
>>> >>
>>> >> Entire log are avaiable there: http://www.spaggiari.org/log.txt
>>> >>
>>> >> JM
>>> >>
>>> >> 2013/2/25 Stack <st...@duboce.net>:
>>> >> > Following on from Andrew, does the debug show you anything? (Copying
>>> >> > individual files?)
>>> >> > St.Ack
>>> >> >
>>> >> >
>>> >> > On Mon, Feb 25, 2013 at 11:09 AM, Jean-Marc Spaggiari <
>>> >> > jean-marc@spaggiari.org> wrote:
>>> >> >
>>> >> >> Hi Andy.
>>> >> >>
>>> >> >> With DEBUG enabled, what seems to be slow is that:
>>> >> >> [DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1 with
>>> >> >> file:
>>> >> >>
>>> >>
>>> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
>>> >> >> to assembly location: lib/commons-math-2.1.jar.
>>> >> >> [DEBUG] Adding file:
>>> >> >>
>>> >> >>
>>> >>
>>> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
>>> >> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
>>> >> >> [DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1 with file:
>>> >> >>
>>> >> >>
>>> >>
>>> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
>>> >> >> to assembly location: lib/commons-net-1.4.1.jar.
>>> >> >> [DEBUG] Adding file:
>>> >> >>
>>> >> >>
>>> >>
>>> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
>>> >> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
>>> >> >> About 3 minutes, displaying one line every 1 to 2 seconds.
>>> >> >>
>>> >> >> Then this is taking a lot of time:
>>> >> >> [INFO] Copying files to /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>>> >> >>
>>> >> >> It's now running for 10 minutes and seems it's doing nothing. This is
>>> >> >> what I saw without the debug enabled. I'm building on an SSD drive.
>>> >> >> It's only about 250MB. I can even build it in memory if required. But
>>> >> >> I don't think it's because of the disk.
>>> >> >>
>>> >> >> JM
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> 2013/2/25 Andrew Purtell <ap...@apache.org>:
>>> >> >> > If you run Maven with debug logging enabled, do you see it taking
>>> ~3
>>> >> >> > seconds per dependency jar to decide if it wants to include it in
>>> the
>>> >> >> > assembly or not?
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
>>> >> >> > jean-marc@spaggiari.org> wrote:
>>> >> >> >
>>> >> >> >> Hi,
>>> >> >> >>
>>> >> >> >> Buliding HBase locally is taking 1h for me:
>>> >> >> >> [INFO]
>>> >> >> >>
>>> >> ------------------------------------------------------------------------
>>> >> >> >> [INFO] BUILD SUCCESS
>>> >> >> >> [INFO]
>>> >> >> >>
>>> >> ------------------------------------------------------------------------
>>> >> >> >> [INFO] Total time: 59:36.659s
>>> >> >> >> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
>>> >> >> >> [INFO] Final Memory: 64M/593M
>>> >> >> >> [INFO]
>>> >> >> >>
>>> >> ------------------------------------------------------------------------
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> But from that our, it seems to be spending most of the time doing
>>> >> that:
>>> >> >> >> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @
>>> hbase
>>> >> ---
>>> >> >> >> [INFO] Reading assembly descriptor: src/assembly/all.xml
>>> >> >> >> [INFO] Copying files to
>>> >> >> >> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>>> >> >> >>
>>> >> >> >> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0
>>> too,
>>> >> >> >> and disks are almost not working.
>>> >> >> >>
>>> >> >> >> I have started the build with mvn clean site install
>>> >> assembly:assembly
>>> >> >> >> -DskipTests -Prelease.
>>> >> >> >>
>>> >> >> >> Is that really spending all this time copŷing files? I will move
>>> the
>>> >> >> >> build to an SSD drive and retry, but is there anyway to reduce the
>>> >> >> >> build time?
>>> >> >> >>
>>> >> >> >> Thanks,
>>> >> >> >>
>>> >> >> >> JM
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> > Best regards,
>>> >> >> >
>>> >> >> > - Andy
>>> >> >> >
>>> >> >> > Problems worthy of attack prove their worth by hitting back. - Piet
>>> >> Hein
>>> >> >> > (via Tom White)
>>> >> >>
>>> >>
>>>
Re: Build performances
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Stack,
When I'm testing locally with VMs I'm using openJDK for the standalone
mode so I don't have to install any additionnal JVM. And it's working
for my needs. I have not faced any issue so far.
I just started the entire test suite with OpenJDK1.7. I will keep you posted.
JM
2013/2/25 Stack <st...@duboce.net>:
> Does hbase work on openjdk?
> St.Ack
>
>
> On Mon, Feb 25, 2013 at 12:54 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Hi Stack,
>>
>> Thanks for pointing to stracing.
>>
>> However, I fixed my issue...
>>
>> Initially I was running with that:
>> Java version: 1.6.0_24, vendor: Sun Microsystems Inc.
>> Java home: /usr/lib/jvm/java-6-openjdk-amd64/jre
>>
>>
>> With "export JAVA_HOME=/usr/local/jdk1.7.0_13/":
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESS
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 5:37.573s
>> [INFO] Finished at: Mon Feb 25 15:50:26 EST 2013
>> [INFO] Final Memory: 62M/3020M
>> [INFO]
>> ------------------------------------------------------------------------
>>
>> Less than 6 minutes. 10 times faster!!!!
>>
>> So I don't really know why it was so slow with the 1.6 OpenJDK, but
>> it's now running way faster!
>>
>> Thanks,
>>
>> JM
>>
>> 2013/2/25 Stack <st...@duboce.net>:
>> > Try stracing the process [1]?
>> > St.Ack
>> >
>> > 1. http://www.cyberciti.biz/tips/linux-strace-command-examples.html
>> >
>> >
>> > On Mon, Feb 25, 2013 at 12:09 PM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org> wrote:
>> >
>> >> When it's stuck on "Copying", there is nothing more displayed.
>> >>
>> >> Before that, it's verbose.
>> >>
>> >> [DEBUG] Adding file-set in: /mnt/security to archive location:
>> >> hbase-0.94.6-SNAPSHOT/security/
>> >> [DEBUG] FileSet[conf/] dir perms: 47777 file perms: 107777
>> >> [DEBUG] The archive base directory is 'null'
>> >> [DEBUG] NOT reformatting any files in /mnt/conf
>> >> [DEBUG] Adding file-set from directory: '/mnt/conf'
>> >> assembly output directory is: 'conf/'
>> >> [DEBUG] Adding file-set in: /mnt/conf to archive location:
>> >> hbase-0.94.6-SNAPSHOT/conf/
>> >> [DEBUG] FileSet[bin/] dir perms: 47777 file perms: 107777
>> >> [DEBUG] The archive base directory is 'null'
>> >> [DEBUG] NOT reformatting any files in /mnt/bin
>> >> [DEBUG] Adding file-set from directory: '/mnt/bin'
>> >> assembly output directory is: 'bin/'
>> >> [DEBUG] Adding file-set in: /mnt/bin to archive location:
>> >> hbase-0.94.6-SNAPSHOT/bin/
>> >> [DEBUG] FileSet[lib/ruby/] dir perms: 47777 file perms: 107777
>> >> [DEBUG] The archive base directory is 'null'
>> >> [DEBUG] NOT reformatting any files in /mnt/src/main/ruby
>> >> [DEBUG] Adding file-set from directory: '/mnt/src/main/ruby'
>> >> assembly output directory is: 'lib/ruby/'
>> >> [DEBUG] Adding file-set in: /mnt/src/main/ruby to archive location:
>> >> hbase-0.94.6-SNAPSHOT/lib/ruby/
>> >> [DEBUG] FileSet[] dir perms: 47777 file perms: 107777
>> >> [DEBUG] The archive base directory is 'null'
>> >> [DEBUG] NOT reformatting any files in /mnt/target
>> >> [DEBUG] Adding file-set from directory: '/mnt/target'
>> >> assembly output directory is: ''
>> >> [DEBUG] Adding file-set in: /mnt/target to archive location:
>> >> hbase-0.94.6-SNAPSHOT/
>> >> [DEBUG] FileSet[hbase-webapps/] dir perms: 47777 file perms: 107777
>> >> [DEBUG] The archive base directory is 'null'
>> >> [DEBUG] NOT reformatting any files in /mnt/target/hbase-webapps
>> >> [DEBUG] Adding file-set from directory: '/mnt/target/hbase-webapps'
>> >> assembly output directory is: 'hbase-webapps/'
>> >> [DEBUG] Adding file-set in: /mnt/target/hbase-webapps to archive
>> >> location: hbase-0.94.6-SNAPSHOT/hbase-webapps/
>> >> [DEBUG] FileSet[docs/] dir perms: 47777 file perms: 107777
>> >> [DEBUG] The archive base directory is 'null'
>> >> [DEBUG] NOT reformatting any files in /mnt/target/site
>> >> [DEBUG] Adding file-set from directory: '/mnt/target/site'
>> >> assembly output directory is: 'docs/'
>> >> [DEBUG] Adding file-set in: /mnt/target/site to archive location:
>> >> hbase-0.94.6-SNAPSHOT/docs/
>> >> [DEBUG] FileSet[sbin/] dir perms: 47777 file perms: 107777
>> >> [DEBUG] The archive base directory is 'null'
>> >> [DEBUG] NOT reformatting any files in /mnt/src/packages
>> >> [DEBUG] Adding file-set from directory: '/mnt/src/packages'
>> >> assembly output directory is: 'sbin/'
>> >> [DEBUG] Adding file-set in: /mnt/src/packages to archive location:
>> >> hbase-0.94.6-SNAPSHOT/sbin/
>> >> [INFO] Copying files to /mnt/target/hbase-0.94.6-SNAPSHOT
>> >>
>> >>
>> >> But now it's stuck again on the last line, not giving more
>> >> information. I killed it and I moved everything on tmpfs but it's not
>> >> faster. I have nothing else running.
>> >>
>> >> How long should it normaly take?
>> >>
>> >> Entire log are avaiable there: http://www.spaggiari.org/log.txt
>> >>
>> >> JM
>> >>
>> >> 2013/2/25 Stack <st...@duboce.net>:
>> >> > Following on from Andrew, does the debug show you anything? (Copying
>> >> > individual files?)
>> >> > St.Ack
>> >> >
>> >> >
>> >> > On Mon, Feb 25, 2013 at 11:09 AM, Jean-Marc Spaggiari <
>> >> > jean-marc@spaggiari.org> wrote:
>> >> >
>> >> >> Hi Andy.
>> >> >>
>> >> >> With DEBUG enabled, what seems to be slow is that:
>> >> >> [DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1 with
>> >> >> file:
>> >> >>
>> >>
>> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
>> >> >> to assembly location: lib/commons-math-2.1.jar.
>> >> >> [DEBUG] Adding file:
>> >> >>
>> >> >>
>> >>
>> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
>> >> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
>> >> >> [DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1 with file:
>> >> >>
>> >> >>
>> >>
>> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
>> >> >> to assembly location: lib/commons-net-1.4.1.jar.
>> >> >> [DEBUG] Adding file:
>> >> >>
>> >> >>
>> >>
>> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
>> >> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
>> >> >> About 3 minutes, displaying one line every 1 to 2 seconds.
>> >> >>
>> >> >> Then this is taking a lot of time:
>> >> >> [INFO] Copying files to /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>> >> >>
>> >> >> It's now running for 10 minutes and seems it's doing nothing. This is
>> >> >> what I saw without the debug enabled. I'm building on an SSD drive.
>> >> >> It's only about 250MB. I can even build it in memory if required. But
>> >> >> I don't think it's because of the disk.
>> >> >>
>> >> >> JM
>> >> >>
>> >> >>
>> >> >>
>> >> >> 2013/2/25 Andrew Purtell <ap...@apache.org>:
>> >> >> > If you run Maven with debug logging enabled, do you see it taking
>> ~3
>> >> >> > seconds per dependency jar to decide if it wants to include it in
>> the
>> >> >> > assembly or not?
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
>> >> >> > jean-marc@spaggiari.org> wrote:
>> >> >> >
>> >> >> >> Hi,
>> >> >> >>
>> >> >> >> Buliding HBase locally is taking 1h for me:
>> >> >> >> [INFO]
>> >> >> >>
>> >> ------------------------------------------------------------------------
>> >> >> >> [INFO] BUILD SUCCESS
>> >> >> >> [INFO]
>> >> >> >>
>> >> ------------------------------------------------------------------------
>> >> >> >> [INFO] Total time: 59:36.659s
>> >> >> >> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
>> >> >> >> [INFO] Final Memory: 64M/593M
>> >> >> >> [INFO]
>> >> >> >>
>> >> ------------------------------------------------------------------------
>> >> >> >>
>> >> >> >>
>> >> >> >> But from that our, it seems to be spending most of the time doing
>> >> that:
>> >> >> >> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @
>> hbase
>> >> ---
>> >> >> >> [INFO] Reading assembly descriptor: src/assembly/all.xml
>> >> >> >> [INFO] Copying files to
>> >> >> >> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>> >> >> >>
>> >> >> >> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0
>> too,
>> >> >> >> and disks are almost not working.
>> >> >> >>
>> >> >> >> I have started the build with mvn clean site install
>> >> assembly:assembly
>> >> >> >> -DskipTests -Prelease.
>> >> >> >>
>> >> >> >> Is that really spending all this time copŷing files? I will move
>> the
>> >> >> >> build to an SSD drive and retry, but is there anyway to reduce the
>> >> >> >> build time?
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >>
>> >> >> >> JM
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Best regards,
>> >> >> >
>> >> >> > - Andy
>> >> >> >
>> >> >> > Problems worthy of attack prove their worth by hitting back. - Piet
>> >> Hein
>> >> >> > (via Tom White)
>> >> >>
>> >>
>>
Re: Build performances
Posted by Stack <st...@duboce.net>.
Does hbase work on openjdk?
St.Ack
On Mon, Feb 25, 2013 at 12:54 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Stack,
>
> Thanks for pointing to stracing.
>
> However, I fixed my issue...
>
> Initially I was running with that:
> Java version: 1.6.0_24, vendor: Sun Microsystems Inc.
> Java home: /usr/lib/jvm/java-6-openjdk-amd64/jre
>
>
> With "export JAVA_HOME=/usr/local/jdk1.7.0_13/":
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 5:37.573s
> [INFO] Finished at: Mon Feb 25 15:50:26 EST 2013
> [INFO] Final Memory: 62M/3020M
> [INFO]
> ------------------------------------------------------------------------
>
> Less than 6 minutes. 10 times faster!!!!
>
> So I don't really know why it was so slow with the 1.6 OpenJDK, but
> it's now running way faster!
>
> Thanks,
>
> JM
>
> 2013/2/25 Stack <st...@duboce.net>:
> > Try stracing the process [1]?
> > St.Ack
> >
> > 1. http://www.cyberciti.biz/tips/linux-strace-command-examples.html
> >
> >
> > On Mon, Feb 25, 2013 at 12:09 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> When it's stuck on "Copying", there is nothing more displayed.
> >>
> >> Before that, it's verbose.
> >>
> >> [DEBUG] Adding file-set in: /mnt/security to archive location:
> >> hbase-0.94.6-SNAPSHOT/security/
> >> [DEBUG] FileSet[conf/] dir perms: 47777 file perms: 107777
> >> [DEBUG] The archive base directory is 'null'
> >> [DEBUG] NOT reformatting any files in /mnt/conf
> >> [DEBUG] Adding file-set from directory: '/mnt/conf'
> >> assembly output directory is: 'conf/'
> >> [DEBUG] Adding file-set in: /mnt/conf to archive location:
> >> hbase-0.94.6-SNAPSHOT/conf/
> >> [DEBUG] FileSet[bin/] dir perms: 47777 file perms: 107777
> >> [DEBUG] The archive base directory is 'null'
> >> [DEBUG] NOT reformatting any files in /mnt/bin
> >> [DEBUG] Adding file-set from directory: '/mnt/bin'
> >> assembly output directory is: 'bin/'
> >> [DEBUG] Adding file-set in: /mnt/bin to archive location:
> >> hbase-0.94.6-SNAPSHOT/bin/
> >> [DEBUG] FileSet[lib/ruby/] dir perms: 47777 file perms: 107777
> >> [DEBUG] The archive base directory is 'null'
> >> [DEBUG] NOT reformatting any files in /mnt/src/main/ruby
> >> [DEBUG] Adding file-set from directory: '/mnt/src/main/ruby'
> >> assembly output directory is: 'lib/ruby/'
> >> [DEBUG] Adding file-set in: /mnt/src/main/ruby to archive location:
> >> hbase-0.94.6-SNAPSHOT/lib/ruby/
> >> [DEBUG] FileSet[] dir perms: 47777 file perms: 107777
> >> [DEBUG] The archive base directory is 'null'
> >> [DEBUG] NOT reformatting any files in /mnt/target
> >> [DEBUG] Adding file-set from directory: '/mnt/target'
> >> assembly output directory is: ''
> >> [DEBUG] Adding file-set in: /mnt/target to archive location:
> >> hbase-0.94.6-SNAPSHOT/
> >> [DEBUG] FileSet[hbase-webapps/] dir perms: 47777 file perms: 107777
> >> [DEBUG] The archive base directory is 'null'
> >> [DEBUG] NOT reformatting any files in /mnt/target/hbase-webapps
> >> [DEBUG] Adding file-set from directory: '/mnt/target/hbase-webapps'
> >> assembly output directory is: 'hbase-webapps/'
> >> [DEBUG] Adding file-set in: /mnt/target/hbase-webapps to archive
> >> location: hbase-0.94.6-SNAPSHOT/hbase-webapps/
> >> [DEBUG] FileSet[docs/] dir perms: 47777 file perms: 107777
> >> [DEBUG] The archive base directory is 'null'
> >> [DEBUG] NOT reformatting any files in /mnt/target/site
> >> [DEBUG] Adding file-set from directory: '/mnt/target/site'
> >> assembly output directory is: 'docs/'
> >> [DEBUG] Adding file-set in: /mnt/target/site to archive location:
> >> hbase-0.94.6-SNAPSHOT/docs/
> >> [DEBUG] FileSet[sbin/] dir perms: 47777 file perms: 107777
> >> [DEBUG] The archive base directory is 'null'
> >> [DEBUG] NOT reformatting any files in /mnt/src/packages
> >> [DEBUG] Adding file-set from directory: '/mnt/src/packages'
> >> assembly output directory is: 'sbin/'
> >> [DEBUG] Adding file-set in: /mnt/src/packages to archive location:
> >> hbase-0.94.6-SNAPSHOT/sbin/
> >> [INFO] Copying files to /mnt/target/hbase-0.94.6-SNAPSHOT
> >>
> >>
> >> But now it's stuck again on the last line, not giving more
> >> information. I killed it and I moved everything on tmpfs but it's not
> >> faster. I have nothing else running.
> >>
> >> How long should it normaly take?
> >>
> >> Entire log are avaiable there: http://www.spaggiari.org/log.txt
> >>
> >> JM
> >>
> >> 2013/2/25 Stack <st...@duboce.net>:
> >> > Following on from Andrew, does the debug show you anything? (Copying
> >> > individual files?)
> >> > St.Ack
> >> >
> >> >
> >> > On Mon, Feb 25, 2013 at 11:09 AM, Jean-Marc Spaggiari <
> >> > jean-marc@spaggiari.org> wrote:
> >> >
> >> >> Hi Andy.
> >> >>
> >> >> With DEBUG enabled, what seems to be slow is that:
> >> >> [DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1 with
> >> >> file:
> >> >>
> >>
> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
> >> >> to assembly location: lib/commons-math-2.1.jar.
> >> >> [DEBUG] Adding file:
> >> >>
> >> >>
> >>
> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
> >> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
> >> >> [DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1 with file:
> >> >>
> >> >>
> >>
> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
> >> >> to assembly location: lib/commons-net-1.4.1.jar.
> >> >> [DEBUG] Adding file:
> >> >>
> >> >>
> >>
> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
> >> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
> >> >> About 3 minutes, displaying one line every 1 to 2 seconds.
> >> >>
> >> >> Then this is taking a lot of time:
> >> >> [INFO] Copying files to /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
> >> >>
> >> >> It's now running for 10 minutes and seems it's doing nothing. This is
> >> >> what I saw without the debug enabled. I'm building on an SSD drive.
> >> >> It's only about 250MB. I can even build it in memory if required. But
> >> >> I don't think it's because of the disk.
> >> >>
> >> >> JM
> >> >>
> >> >>
> >> >>
> >> >> 2013/2/25 Andrew Purtell <ap...@apache.org>:
> >> >> > If you run Maven with debug logging enabled, do you see it taking
> ~3
> >> >> > seconds per dependency jar to decide if it wants to include it in
> the
> >> >> > assembly or not?
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
> >> >> > jean-marc@spaggiari.org> wrote:
> >> >> >
> >> >> >> Hi,
> >> >> >>
> >> >> >> Buliding HBase locally is taking 1h for me:
> >> >> >> [INFO]
> >> >> >>
> >> ------------------------------------------------------------------------
> >> >> >> [INFO] BUILD SUCCESS
> >> >> >> [INFO]
> >> >> >>
> >> ------------------------------------------------------------------------
> >> >> >> [INFO] Total time: 59:36.659s
> >> >> >> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
> >> >> >> [INFO] Final Memory: 64M/593M
> >> >> >> [INFO]
> >> >> >>
> >> ------------------------------------------------------------------------
> >> >> >>
> >> >> >>
> >> >> >> But from that our, it seems to be spending most of the time doing
> >> that:
> >> >> >> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @
> hbase
> >> ---
> >> >> >> [INFO] Reading assembly descriptor: src/assembly/all.xml
> >> >> >> [INFO] Copying files to
> >> >> >> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
> >> >> >>
> >> >> >> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0
> too,
> >> >> >> and disks are almost not working.
> >> >> >>
> >> >> >> I have started the build with mvn clean site install
> >> assembly:assembly
> >> >> >> -DskipTests -Prelease.
> >> >> >>
> >> >> >> Is that really spending all this time copŷing files? I will move
> the
> >> >> >> build to an SSD drive and retry, but is there anyway to reduce the
> >> >> >> build time?
> >> >> >>
> >> >> >> Thanks,
> >> >> >>
> >> >> >> JM
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Best regards,
> >> >> >
> >> >> > - Andy
> >> >> >
> >> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> >> Hein
> >> >> > (via Tom White)
> >> >>
> >>
>
Re: Build performances
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Stack,
Thanks for pointing to stracing.
However, I fixed my issue...
Initially I was running with that:
Java version: 1.6.0_24, vendor: Sun Microsystems Inc.
Java home: /usr/lib/jvm/java-6-openjdk-amd64/jre
With "export JAVA_HOME=/usr/local/jdk1.7.0_13/":
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5:37.573s
[INFO] Finished at: Mon Feb 25 15:50:26 EST 2013
[INFO] Final Memory: 62M/3020M
[INFO] ------------------------------------------------------------------------
Less than 6 minutes. 10 times faster!!!!
So I don't really know why it was so slow with the 1.6 OpenJDK, but
it's now running way faster!
Thanks,
JM
2013/2/25 Stack <st...@duboce.net>:
> Try stracing the process [1]?
> St.Ack
>
> 1. http://www.cyberciti.biz/tips/linux-strace-command-examples.html
>
>
> On Mon, Feb 25, 2013 at 12:09 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> When it's stuck on "Copying", there is nothing more displayed.
>>
>> Before that, it's verbose.
>>
>> [DEBUG] Adding file-set in: /mnt/security to archive location:
>> hbase-0.94.6-SNAPSHOT/security/
>> [DEBUG] FileSet[conf/] dir perms: 47777 file perms: 107777
>> [DEBUG] The archive base directory is 'null'
>> [DEBUG] NOT reformatting any files in /mnt/conf
>> [DEBUG] Adding file-set from directory: '/mnt/conf'
>> assembly output directory is: 'conf/'
>> [DEBUG] Adding file-set in: /mnt/conf to archive location:
>> hbase-0.94.6-SNAPSHOT/conf/
>> [DEBUG] FileSet[bin/] dir perms: 47777 file perms: 107777
>> [DEBUG] The archive base directory is 'null'
>> [DEBUG] NOT reformatting any files in /mnt/bin
>> [DEBUG] Adding file-set from directory: '/mnt/bin'
>> assembly output directory is: 'bin/'
>> [DEBUG] Adding file-set in: /mnt/bin to archive location:
>> hbase-0.94.6-SNAPSHOT/bin/
>> [DEBUG] FileSet[lib/ruby/] dir perms: 47777 file perms: 107777
>> [DEBUG] The archive base directory is 'null'
>> [DEBUG] NOT reformatting any files in /mnt/src/main/ruby
>> [DEBUG] Adding file-set from directory: '/mnt/src/main/ruby'
>> assembly output directory is: 'lib/ruby/'
>> [DEBUG] Adding file-set in: /mnt/src/main/ruby to archive location:
>> hbase-0.94.6-SNAPSHOT/lib/ruby/
>> [DEBUG] FileSet[] dir perms: 47777 file perms: 107777
>> [DEBUG] The archive base directory is 'null'
>> [DEBUG] NOT reformatting any files in /mnt/target
>> [DEBUG] Adding file-set from directory: '/mnt/target'
>> assembly output directory is: ''
>> [DEBUG] Adding file-set in: /mnt/target to archive location:
>> hbase-0.94.6-SNAPSHOT/
>> [DEBUG] FileSet[hbase-webapps/] dir perms: 47777 file perms: 107777
>> [DEBUG] The archive base directory is 'null'
>> [DEBUG] NOT reformatting any files in /mnt/target/hbase-webapps
>> [DEBUG] Adding file-set from directory: '/mnt/target/hbase-webapps'
>> assembly output directory is: 'hbase-webapps/'
>> [DEBUG] Adding file-set in: /mnt/target/hbase-webapps to archive
>> location: hbase-0.94.6-SNAPSHOT/hbase-webapps/
>> [DEBUG] FileSet[docs/] dir perms: 47777 file perms: 107777
>> [DEBUG] The archive base directory is 'null'
>> [DEBUG] NOT reformatting any files in /mnt/target/site
>> [DEBUG] Adding file-set from directory: '/mnt/target/site'
>> assembly output directory is: 'docs/'
>> [DEBUG] Adding file-set in: /mnt/target/site to archive location:
>> hbase-0.94.6-SNAPSHOT/docs/
>> [DEBUG] FileSet[sbin/] dir perms: 47777 file perms: 107777
>> [DEBUG] The archive base directory is 'null'
>> [DEBUG] NOT reformatting any files in /mnt/src/packages
>> [DEBUG] Adding file-set from directory: '/mnt/src/packages'
>> assembly output directory is: 'sbin/'
>> [DEBUG] Adding file-set in: /mnt/src/packages to archive location:
>> hbase-0.94.6-SNAPSHOT/sbin/
>> [INFO] Copying files to /mnt/target/hbase-0.94.6-SNAPSHOT
>>
>>
>> But now it's stuck again on the last line, not giving more
>> information. I killed it and I moved everything on tmpfs but it's not
>> faster. I have nothing else running.
>>
>> How long should it normaly take?
>>
>> Entire log are avaiable there: http://www.spaggiari.org/log.txt
>>
>> JM
>>
>> 2013/2/25 Stack <st...@duboce.net>:
>> > Following on from Andrew, does the debug show you anything? (Copying
>> > individual files?)
>> > St.Ack
>> >
>> >
>> > On Mon, Feb 25, 2013 at 11:09 AM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org> wrote:
>> >
>> >> Hi Andy.
>> >>
>> >> With DEBUG enabled, what seems to be slow is that:
>> >> [DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1 with
>> >> file:
>> >>
>> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
>> >> to assembly location: lib/commons-math-2.1.jar.
>> >> [DEBUG] Adding file:
>> >>
>> >>
>> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
>> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
>> >> [DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1 with file:
>> >>
>> >>
>> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
>> >> to assembly location: lib/commons-net-1.4.1.jar.
>> >> [DEBUG] Adding file:
>> >>
>> >>
>> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
>> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
>> >> About 3 minutes, displaying one line every 1 to 2 seconds.
>> >>
>> >> Then this is taking a lot of time:
>> >> [INFO] Copying files to /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>> >>
>> >> It's now running for 10 minutes and seems it's doing nothing. This is
>> >> what I saw without the debug enabled. I'm building on an SSD drive.
>> >> It's only about 250MB. I can even build it in memory if required. But
>> >> I don't think it's because of the disk.
>> >>
>> >> JM
>> >>
>> >>
>> >>
>> >> 2013/2/25 Andrew Purtell <ap...@apache.org>:
>> >> > If you run Maven with debug logging enabled, do you see it taking ~3
>> >> > seconds per dependency jar to decide if it wants to include it in the
>> >> > assembly or not?
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
>> >> > jean-marc@spaggiari.org> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> Buliding HBase locally is taking 1h for me:
>> >> >> [INFO]
>> >> >>
>> ------------------------------------------------------------------------
>> >> >> [INFO] BUILD SUCCESS
>> >> >> [INFO]
>> >> >>
>> ------------------------------------------------------------------------
>> >> >> [INFO] Total time: 59:36.659s
>> >> >> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
>> >> >> [INFO] Final Memory: 64M/593M
>> >> >> [INFO]
>> >> >>
>> ------------------------------------------------------------------------
>> >> >>
>> >> >>
>> >> >> But from that our, it seems to be spending most of the time doing
>> that:
>> >> >> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @ hbase
>> ---
>> >> >> [INFO] Reading assembly descriptor: src/assembly/all.xml
>> >> >> [INFO] Copying files to
>> >> >> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>> >> >>
>> >> >> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0 too,
>> >> >> and disks are almost not working.
>> >> >>
>> >> >> I have started the build with mvn clean site install
>> assembly:assembly
>> >> >> -DskipTests -Prelease.
>> >> >>
>> >> >> Is that really spending all this time copŷing files? I will move the
>> >> >> build to an SSD drive and retry, but is there anyway to reduce the
>> >> >> build time?
>> >> >>
>> >> >> Thanks,
>> >> >>
>> >> >> JM
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Best regards,
>> >> >
>> >> > - Andy
>> >> >
>> >> > Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>> >> > (via Tom White)
>> >>
>>
Re: Build performances
Posted by Stack <st...@duboce.net>.
Try stracing the process [1]?
St.Ack
1. http://www.cyberciti.biz/tips/linux-strace-command-examples.html
On Mon, Feb 25, 2013 at 12:09 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> When it's stuck on "Copying", there is nothing more displayed.
>
> Before that, it's verbose.
>
> [DEBUG] Adding file-set in: /mnt/security to archive location:
> hbase-0.94.6-SNAPSHOT/security/
> [DEBUG] FileSet[conf/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/conf
> [DEBUG] Adding file-set from directory: '/mnt/conf'
> assembly output directory is: 'conf/'
> [DEBUG] Adding file-set in: /mnt/conf to archive location:
> hbase-0.94.6-SNAPSHOT/conf/
> [DEBUG] FileSet[bin/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/bin
> [DEBUG] Adding file-set from directory: '/mnt/bin'
> assembly output directory is: 'bin/'
> [DEBUG] Adding file-set in: /mnt/bin to archive location:
> hbase-0.94.6-SNAPSHOT/bin/
> [DEBUG] FileSet[lib/ruby/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/src/main/ruby
> [DEBUG] Adding file-set from directory: '/mnt/src/main/ruby'
> assembly output directory is: 'lib/ruby/'
> [DEBUG] Adding file-set in: /mnt/src/main/ruby to archive location:
> hbase-0.94.6-SNAPSHOT/lib/ruby/
> [DEBUG] FileSet[] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/target
> [DEBUG] Adding file-set from directory: '/mnt/target'
> assembly output directory is: ''
> [DEBUG] Adding file-set in: /mnt/target to archive location:
> hbase-0.94.6-SNAPSHOT/
> [DEBUG] FileSet[hbase-webapps/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/target/hbase-webapps
> [DEBUG] Adding file-set from directory: '/mnt/target/hbase-webapps'
> assembly output directory is: 'hbase-webapps/'
> [DEBUG] Adding file-set in: /mnt/target/hbase-webapps to archive
> location: hbase-0.94.6-SNAPSHOT/hbase-webapps/
> [DEBUG] FileSet[docs/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/target/site
> [DEBUG] Adding file-set from directory: '/mnt/target/site'
> assembly output directory is: 'docs/'
> [DEBUG] Adding file-set in: /mnt/target/site to archive location:
> hbase-0.94.6-SNAPSHOT/docs/
> [DEBUG] FileSet[sbin/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/src/packages
> [DEBUG] Adding file-set from directory: '/mnt/src/packages'
> assembly output directory is: 'sbin/'
> [DEBUG] Adding file-set in: /mnt/src/packages to archive location:
> hbase-0.94.6-SNAPSHOT/sbin/
> [INFO] Copying files to /mnt/target/hbase-0.94.6-SNAPSHOT
>
>
> But now it's stuck again on the last line, not giving more
> information. I killed it and I moved everything on tmpfs but it's not
> faster. I have nothing else running.
>
> How long should it normaly take?
>
> Entire log are avaiable there: http://www.spaggiari.org/log.txt
>
> JM
>
> 2013/2/25 Stack <st...@duboce.net>:
> > Following on from Andrew, does the debug show you anything? (Copying
> > individual files?)
> > St.Ack
> >
> >
> > On Mon, Feb 25, 2013 at 11:09 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> Hi Andy.
> >>
> >> With DEBUG enabled, what seems to be slow is that:
> >> [DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1 with
> >> file:
> >>
> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
> >> to assembly location: lib/commons-math-2.1.jar.
> >> [DEBUG] Adding file:
> >>
> >>
> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
> >> [DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1 with file:
> >>
> >>
> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
> >> to assembly location: lib/commons-net-1.4.1.jar.
> >> [DEBUG] Adding file:
> >>
> >>
> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
> >> About 3 minutes, displaying one line every 1 to 2 seconds.
> >>
> >> Then this is taking a lot of time:
> >> [INFO] Copying files to /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
> >>
> >> It's now running for 10 minutes and seems it's doing nothing. This is
> >> what I saw without the debug enabled. I'm building on an SSD drive.
> >> It's only about 250MB. I can even build it in memory if required. But
> >> I don't think it's because of the disk.
> >>
> >> JM
> >>
> >>
> >>
> >> 2013/2/25 Andrew Purtell <ap...@apache.org>:
> >> > If you run Maven with debug logging enabled, do you see it taking ~3
> >> > seconds per dependency jar to decide if it wants to include it in the
> >> > assembly or not?
> >> >
> >> >
> >> >
> >> > On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
> >> > jean-marc@spaggiari.org> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> Buliding HBase locally is taking 1h for me:
> >> >> [INFO]
> >> >>
> ------------------------------------------------------------------------
> >> >> [INFO] BUILD SUCCESS
> >> >> [INFO]
> >> >>
> ------------------------------------------------------------------------
> >> >> [INFO] Total time: 59:36.659s
> >> >> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
> >> >> [INFO] Final Memory: 64M/593M
> >> >> [INFO]
> >> >>
> ------------------------------------------------------------------------
> >> >>
> >> >>
> >> >> But from that our, it seems to be spending most of the time doing
> that:
> >> >> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @ hbase
> ---
> >> >> [INFO] Reading assembly descriptor: src/assembly/all.xml
> >> >> [INFO] Copying files to
> >> >> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
> >> >>
> >> >> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0 too,
> >> >> and disks are almost not working.
> >> >>
> >> >> I have started the build with mvn clean site install
> assembly:assembly
> >> >> -DskipTests -Prelease.
> >> >>
> >> >> Is that really spending all this time copŷing files? I will move the
> >> >> build to an SSD drive and retry, but is there anyway to reduce the
> >> >> build time?
> >> >>
> >> >> Thanks,
> >> >>
> >> >> JM
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> > - Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> > (via Tom White)
> >>
>
Re: Build performances
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
When it's stuck on "Copying", there is nothing more displayed.
Before that, it's verbose.
[DEBUG] Adding file-set in: /mnt/security to archive location:
hbase-0.94.6-SNAPSHOT/security/
[DEBUG] FileSet[conf/] dir perms: 47777 file perms: 107777
[DEBUG] The archive base directory is 'null'
[DEBUG] NOT reformatting any files in /mnt/conf
[DEBUG] Adding file-set from directory: '/mnt/conf'
assembly output directory is: 'conf/'
[DEBUG] Adding file-set in: /mnt/conf to archive location:
hbase-0.94.6-SNAPSHOT/conf/
[DEBUG] FileSet[bin/] dir perms: 47777 file perms: 107777
[DEBUG] The archive base directory is 'null'
[DEBUG] NOT reformatting any files in /mnt/bin
[DEBUG] Adding file-set from directory: '/mnt/bin'
assembly output directory is: 'bin/'
[DEBUG] Adding file-set in: /mnt/bin to archive location:
hbase-0.94.6-SNAPSHOT/bin/
[DEBUG] FileSet[lib/ruby/] dir perms: 47777 file perms: 107777
[DEBUG] The archive base directory is 'null'
[DEBUG] NOT reformatting any files in /mnt/src/main/ruby
[DEBUG] Adding file-set from directory: '/mnt/src/main/ruby'
assembly output directory is: 'lib/ruby/'
[DEBUG] Adding file-set in: /mnt/src/main/ruby to archive location:
hbase-0.94.6-SNAPSHOT/lib/ruby/
[DEBUG] FileSet[] dir perms: 47777 file perms: 107777
[DEBUG] The archive base directory is 'null'
[DEBUG] NOT reformatting any files in /mnt/target
[DEBUG] Adding file-set from directory: '/mnt/target'
assembly output directory is: ''
[DEBUG] Adding file-set in: /mnt/target to archive location:
hbase-0.94.6-SNAPSHOT/
[DEBUG] FileSet[hbase-webapps/] dir perms: 47777 file perms: 107777
[DEBUG] The archive base directory is 'null'
[DEBUG] NOT reformatting any files in /mnt/target/hbase-webapps
[DEBUG] Adding file-set from directory: '/mnt/target/hbase-webapps'
assembly output directory is: 'hbase-webapps/'
[DEBUG] Adding file-set in: /mnt/target/hbase-webapps to archive
location: hbase-0.94.6-SNAPSHOT/hbase-webapps/
[DEBUG] FileSet[docs/] dir perms: 47777 file perms: 107777
[DEBUG] The archive base directory is 'null'
[DEBUG] NOT reformatting any files in /mnt/target/site
[DEBUG] Adding file-set from directory: '/mnt/target/site'
assembly output directory is: 'docs/'
[DEBUG] Adding file-set in: /mnt/target/site to archive location:
hbase-0.94.6-SNAPSHOT/docs/
[DEBUG] FileSet[sbin/] dir perms: 47777 file perms: 107777
[DEBUG] The archive base directory is 'null'
[DEBUG] NOT reformatting any files in /mnt/src/packages
[DEBUG] Adding file-set from directory: '/mnt/src/packages'
assembly output directory is: 'sbin/'
[DEBUG] Adding file-set in: /mnt/src/packages to archive location:
hbase-0.94.6-SNAPSHOT/sbin/
[INFO] Copying files to /mnt/target/hbase-0.94.6-SNAPSHOT
But now it's stuck again on the last line, not giving more
information. I killed it and I moved everything on tmpfs but it's not
faster. I have nothing else running.
How long should it normaly take?
Entire log are avaiable there: http://www.spaggiari.org/log.txt
JM
2013/2/25 Stack <st...@duboce.net>:
> Following on from Andrew, does the debug show you anything? (Copying
> individual files?)
> St.Ack
>
>
> On Mon, Feb 25, 2013 at 11:09 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Hi Andy.
>>
>> With DEBUG enabled, what seems to be slow is that:
>> [DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1 with
>> file:
>> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
>> to assembly location: lib/commons-math-2.1.jar.
>> [DEBUG] Adding file:
>>
>> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
>> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
>> [DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1 with file:
>>
>> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
>> to assembly location: lib/commons-net-1.4.1.jar.
>> [DEBUG] Adding file:
>>
>> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
>> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
>> About 3 minutes, displaying one line every 1 to 2 seconds.
>>
>> Then this is taking a lot of time:
>> [INFO] Copying files to /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>>
>> It's now running for 10 minutes and seems it's doing nothing. This is
>> what I saw without the debug enabled. I'm building on an SSD drive.
>> It's only about 250MB. I can even build it in memory if required. But
>> I don't think it's because of the disk.
>>
>> JM
>>
>>
>>
>> 2013/2/25 Andrew Purtell <ap...@apache.org>:
>> > If you run Maven with debug logging enabled, do you see it taking ~3
>> > seconds per dependency jar to decide if it wants to include it in the
>> > assembly or not?
>> >
>> >
>> >
>> > On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org> wrote:
>> >
>> >> Hi,
>> >>
>> >> Buliding HBase locally is taking 1h for me:
>> >> [INFO]
>> >> ------------------------------------------------------------------------
>> >> [INFO] BUILD SUCCESS
>> >> [INFO]
>> >> ------------------------------------------------------------------------
>> >> [INFO] Total time: 59:36.659s
>> >> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
>> >> [INFO] Final Memory: 64M/593M
>> >> [INFO]
>> >> ------------------------------------------------------------------------
>> >>
>> >>
>> >> But from that our, it seems to be spending most of the time doing that:
>> >> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @ hbase ---
>> >> [INFO] Reading assembly descriptor: src/assembly/all.xml
>> >> [INFO] Copying files to
>> >> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>> >>
>> >> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0 too,
>> >> and disks are almost not working.
>> >>
>> >> I have started the build with mvn clean site install assembly:assembly
>> >> -DskipTests -Prelease.
>> >>
>> >> Is that really spending all this time copŷing files? I will move the
>> >> build to an SSD drive and retry, but is there anyway to reduce the
>> >> build time?
>> >>
>> >> Thanks,
>> >>
>> >> JM
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> > - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>>
Re: Build performances
Posted by Stack <st...@duboce.net>.
Following on from Andrew, does the debug show you anything? (Copying
individual files?)
St.Ack
On Mon, Feb 25, 2013 at 11:09 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Andy.
>
> With DEBUG enabled, what seems to be slow is that:
> [DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1 with
> file:
> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
> to assembly location: lib/commons-math-2.1.jar.
> [DEBUG] Adding file:
>
> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
> [DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1 with file:
>
> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
> to assembly location: lib/commons-net-1.4.1.jar.
> [DEBUG] Adding file:
>
> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
> About 3 minutes, displaying one line every 1 to 2 seconds.
>
> Then this is taking a lot of time:
> [INFO] Copying files to /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>
> It's now running for 10 minutes and seems it's doing nothing. This is
> what I saw without the debug enabled. I'm building on an SSD drive.
> It's only about 250MB. I can even build it in memory if required. But
> I don't think it's because of the disk.
>
> JM
>
>
>
> 2013/2/25 Andrew Purtell <ap...@apache.org>:
> > If you run Maven with debug logging enabled, do you see it taking ~3
> > seconds per dependency jar to decide if it wants to include it in the
> > assembly or not?
> >
> >
> >
> > On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> Hi,
> >>
> >> Buliding HBase locally is taking 1h for me:
> >> [INFO]
> >> ------------------------------------------------------------------------
> >> [INFO] BUILD SUCCESS
> >> [INFO]
> >> ------------------------------------------------------------------------
> >> [INFO] Total time: 59:36.659s
> >> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
> >> [INFO] Final Memory: 64M/593M
> >> [INFO]
> >> ------------------------------------------------------------------------
> >>
> >>
> >> But from that our, it seems to be spending most of the time doing that:
> >> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @ hbase ---
> >> [INFO] Reading assembly descriptor: src/assembly/all.xml
> >> [INFO] Copying files to
> >> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
> >>
> >> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0 too,
> >> and disks are almost not working.
> >>
> >> I have started the build with mvn clean site install assembly:assembly
> >> -DskipTests -Prelease.
> >>
> >> Is that really spending all this time copŷing files? I will move the
> >> build to an SSD drive and retry, but is there anyway to reduce the
> >> build time?
> >>
> >> Thanks,
> >>
> >> JM
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>
Re: Build performances
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Andy.
With DEBUG enabled, what seems to be slow is that:
[DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1 with
file: /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
to assembly location: lib/commons-math-2.1.jar.
[DEBUG] Adding file:
/home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
[DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1 with file:
/home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
to assembly location: lib/commons-net-1.4.1.jar.
[DEBUG] Adding file:
/home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
About 3 minutes, displaying one line every 1 to 2 seconds.
Then this is taking a lot of time:
[INFO] Copying files to /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
It's now running for 10 minutes and seems it's doing nothing. This is
what I saw without the debug enabled. I'm building on an SSD drive.
It's only about 250MB. I can even build it in memory if required. But
I don't think it's because of the disk.
JM
2013/2/25 Andrew Purtell <ap...@apache.org>:
> If you run Maven with debug logging enabled, do you see it taking ~3
> seconds per dependency jar to decide if it wants to include it in the
> assembly or not?
>
>
>
> On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Hi,
>>
>> Buliding HBase locally is taking 1h for me:
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESS
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 59:36.659s
>> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
>> [INFO] Final Memory: 64M/593M
>> [INFO]
>> ------------------------------------------------------------------------
>>
>>
>> But from that our, it seems to be spending most of the time doing that:
>> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @ hbase ---
>> [INFO] Reading assembly descriptor: src/assembly/all.xml
>> [INFO] Copying files to
>> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>>
>> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0 too,
>> and disks are almost not working.
>>
>> I have started the build with mvn clean site install assembly:assembly
>> -DskipTests -Prelease.
>>
>> Is that really spending all this time copŷing files? I will move the
>> build to an SSD drive and retry, but is there anyway to reduce the
>> build time?
>>
>> Thanks,
>>
>> JM
>>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: Build performances
Posted by Andrew Purtell <ap...@apache.org>.
If you run Maven with debug logging enabled, do you see it taking ~3
seconds per dependency jar to decide if it wants to include it in the
assembly or not?
On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi,
>
> Buliding HBase locally is taking 1h for me:
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 59:36.659s
> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
> [INFO] Final Memory: 64M/593M
> [INFO]
> ------------------------------------------------------------------------
>
>
> But from that our, it seems to be spending most of the time doing that:
> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @ hbase ---
> [INFO] Reading assembly descriptor: src/assembly/all.xml
> [INFO] Copying files to
> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>
> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0 too,
> and disks are almost not working.
>
> I have started the build with mvn clean site install assembly:assembly
> -DskipTests -Prelease.
>
> Is that really spending all this time copŷing files? I will move the
> build to an SSD drive and retry, but is there anyway to reduce the
> build time?
>
> Thanks,
>
> JM
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)