You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by Dongjin Lee <do...@apache.org> on 2018/07/19 05:37:40 UTC
Build failure on 0.13.SNAPSHOT
Hello. I am trying to build druid, but it fails. My environment is like the
following:
- CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
- RAM: 7704 MB
- OS: ubuntu 18.04
- JDK: openjdk version "1.8.0_171" (default configuration, with MaxHeapSize
= 1928 MB)
- Branch: master (commit: cd8ea3d)
The error message I got is:
[INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] io.druid:druid ..................................... SUCCESS [
> 50.258 s]
> [INFO] java-util .......................................... SUCCESS [03:57
> min]
> [INFO] druid-api .......................................... SUCCESS [
> 22.694 s]
> [INFO] druid-common ....................................... SUCCESS [
> 14.083 s]
> [INFO] druid-hll .......................................... SUCCESS [
> 17.126 s]
> [INFO] extendedset ........................................ SUCCESS [
> 10.856 s]
>
> *[INFO] druid-processing ................................... FAILURE
> [04:36 min]*[INFO] druid-aws-common ...................................
> SKIPPED
> [INFO] druid-server ....................................... SKIPPED
> [INFO] druid-examples ..................................... SKIPPED
> ...
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 10:29 min
> [INFO] Finished at: 2018-07-19T13:23:31+09:00
> [INFO] Final Memory: 88M/777M
> [INFO]
> ------------------------------------------------------------------------
>
> *[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test)
> on project druid-processing: Execution default-test of goal
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test failed: The
> forked VM terminated without properly saying goodbye. VM crash or
> System.exit called?*[ERROR] Command was /bin/sh -c cd
> /home/djlee/workspace/java/druid/processing &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m -Duser.language=en
> -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
> -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> -Ddruid.indexing.doubleStorage=double -jar
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
It seems like it fails when it runs tests on `druid-processing` module but
I can't certain. Is there anyone who can give me some hints? Thanks in
advance.
Best,
Dongjin
--
*Dongjin Lee*
*A hitchhiker in the mathematical world.*
*github: <http://goog_969573159/>github.com/dongjinleekr
<http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>slideshare:
www.slideshare.net/dongjinleekr
<http://www.slideshare.net/dongjinleekr>*
Re: Build failure on 0.13.SNAPSHOT
Posted by Dongjin Lee <do...@apache.org>.
Oh, it seems like apache mailing server ate the attached image as Gian
said; here is the table:
total used free shared buff/cache available
Mem: 7704 6147 122 161 1435 1123
Swap: 4095 1667 2428
Frankly, this result is not accurate; since I closed all the other
applications except the build process, it shows the memory usage of `system
processes + build process` - Jihoon is right, if the test process uses 8gb
of memory as I said first, the test can't be run on Travis. The reason
would be the other something as Jihoon said - I am working on a desktop
version of Ubuntu with the different implementation of JDK (openjdk 8) to
Travis configuration. Now I guess these differences caused some side
effects. Since this is not a problem with the code, you don't need a fix.
Thanks again for your kind guidance.
Thanks,
Dongjin
On Thu, Jul 26, 2018 at 3:24 AM Jihoon Son <ji...@apache.org> wrote:
> @Dongjin, thank you for looking into this. I got some follow-up questions.
>
> > Druid uses above 8gb of memory for testing.
>
> How did you get this result? Did you use some tools to get the memory usage
> of only the maven process for Druid tests? When I checked last time using
> JMC, I could see that running all unit tests of each Travis Job usually use
> up to 512 MB of memory (you can find our Travis Job configuration here:
> https://github.com/apache/incubator-druid/blob/master/.travis.yml). That's
> why I set the memory limit to 512 MB for our Travis CI. Maybe we need to
> check again and raise the memory limit to avoid frequent OOM failures in
> groupBy query tests.
>
> Also, if it does use 8 GB of memory, I suspect it's a sort of bug or
> something. I don't think even our integration tests don't use such a large
> memory which run all Druid processes, ZooKeeper, and Kafka. So, we should
> figure out which tests use such a lot of memory and fix it instead of
> fixing documents.
>
> > Fwiw I think 8GB should be enough memory with the right settings, since
> our Travis CI build environment only has about 7.5GB.
>
> @Gian, the memory of the Travis instance is shared by all processes
> including OS. AFAIK, the OS requires about 1 GB of memory. It means that
> our unit tests can't be run on Travis if it needs 8 GB of memory.
>
> Best,
> Jihoon
>
> On Wed, Jul 25, 2018 at 10:01 AM Charles Allen
> <ch...@snap.com.invalid> wrote:
>
> > OOME seems to be showing up in some of the Travis testing as well for
> group
> > by related stuff. Unsure what's going on there.
> >
> > On Tue, Jul 24, 2018 at 9:46 PM Dongjin Lee <do...@apache.org> wrote:
> >
> > > After some experiments, I figured out the following:
> > >
> > > 1. Druid uses above 8gb of memory for testing. (building-druid.png)
> > > 2. With 8gb(physical)+4gb(swap) of memory, the test succeeds regardless
> > of
> > > maven version (3.3.9, 3.5.2, 3.5.4) or MAVEN_OPTS. However, with
> > > 8gb(physical)+2gb(swap) of memory[^1], some tests failed. The list of
> > > failing tests differs between maven 3.3.9 and 3.5.2.
> > >
> > > In short, retaining sufficient memory solved the problem - *It seems
> like
> > > 12gb of memory is a recommended setting for building druid.* (I guess
> > > lots of you are working with the MacBook Pro with 16gb RAM, right? In
> > that
> > > case, you must not have encountered this problem.)
> > >
> > > If you are okay, may I update the building documentation for the
> newbies
> > > like me?
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > +1. While building Druid, I found another problem. But this issue
> should
> > > be discussed in another thread.
> > >
> > > [^1]: You know, the other processes also occupy the memory.
> > >
> > >
> > > On Tue, Jul 24, 2018 at 3:07 AM Jihoon Son <ji...@apache.org>
> wrote:
> > >
> > >> I'm also using Maven 3.5.2 and not using any special configurations
> for
> > >> Maven, but I have never seen that error too.
> > >> Most of our Travis jobs have been working with only 512 MB of direct
> > >> memory. Only the 'strict compilation' Travis job requires 3 GB of
> > memory.
> > >>
> > >> I think it's worthwhile to look into this more. Maybe we somehow use
> > more
> > >> memory when we run all tests by 'mvn install'. Maybe this relates to
> the
> > >> frequent transient failures of 'processing module test', one of our
> > Travis
> > >> jobs.
> > >>
> > >> Jihoon
> > >>
> > >> On Mon, Jul 23, 2018 at 9:32 AM Gian Merlino <gi...@apache.org> wrote:
> > >>
> > >> > Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it
> > has
> > >> > been working for for me. I don't think I"m using any special Maven
> > >> > overrides (at least, I don't see anything interesting in my ~/.m2
> > >> directory
> > >> > or in my environment variables). It might have to do with how much
> > >> memory
> > >> > our machines have? I do most of my builds on a Mac with 16GB RAM.
> > Maybe
> > >> try
> > >> > checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn
> > >> install
> > >> > commands, which might be needed for more low memory environments.
> > >> >
> > >> > $ mvn --version
> > >> > Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
> > >> > 2017-10-18T00:58:13-07:00)
> > >> > Maven home: /usr/local/Cellar/maven/3.5.2/libexec
> > >> > Java version: 1.8.0_161, vendor: Oracle Corporation
> > >> > Java home:
> > >> > /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre
> > >> > Default locale: en_US, platform encoding: UTF-8
> > >> > OS name: "mac os x", version: "10.13.5", arch: "x86_64", family:
> "mac"
> > >> >
> > >> > On Mon, Jul 23, 2018 at 6:40 AM Dongjin Lee <do...@apache.org>
> > wrote:
> > >> >
> > >> > > Finally, it seems like I found the reason. It was a composition of
> > >> > several
> > >> > > problems:
> > >> > >
> > >> > > - Druid should not be built with maven 3.5.x. With 3.5.2, Test
> > suites
> > >> > like
> > >> > > `GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9
> > >> which
> > >> > is
> > >> > > built in the latest version of IntelliJ, those errors disappeared.
> > It
> > >> > seems
> > >> > > like maven 3.5.x is not stable yet - it applied a drastic change,
> > and
> > >> it
> > >> > is
> > >> > > also why they skipped 3.4.x.
> > >> > > - It seems like Druid requires some MaxDirectMemorySize
> > configuration
> > >> for
> > >> > > some test suites. With some JVM parameter like
> > >> > `-XX:MaxDirectMemorySize=4g`
> > >> > > some test suites were passed, but not all. I am now trying the
> other
> > >> > > options with enlarged swap space.
> > >> > >
> > >> > > Question: How much MaxDirectMemorySize configuration are you
> using?
> > >> > >
> > >> > > Best,
> > >> > > Dongjin
> > >> > >
> > >> > > On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <gh...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > > > Hi Dongjin,
> > >> > > >
> > >> > > > that is weird. It looks like the vm crashed because of out of
> > memory
> > >> > > while
> > >> > > > testing.
> > >> > > > It might be a real issue or not.
> > >> > > > Have you set any memory configuration for your maven?
> > >> > > >
> > >> > > > Jihoon
> > >> > > >
> > >> > > > On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <dongjin@apache.org
> >
> > >> > wrote:
> > >> > > >
> > >> > > > > Hi Jihoon,
> > >> > > > >
> > >> > > > > I ran `mvn clean package` following development/build
> > >> > > > > <
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
> > >> > > > > >
> > >> > > > > .
> > >> > > > >
> > >> > > > > Dongjin
> > >> > > > >
> > >> > > > > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <
> > jihoonson@apache.org
> > >> >
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > Hi Dongjin,
> > >> > > > > >
> > >> > > > > > what maven command did you run?
> > >> > > > > >
> > >> > > > > > Jihoon
> > >> > > > > >
> > >> > > > > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <
> > >> dongjin@apache.org>
> > >> > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hello. I am trying to build druid, but it fails. My
> > >> environment
> > >> > is
> > >> > > > like
> > >> > > > > > the
> > >> > > > > > > following:
> > >> > > > > > >
> > >> > > > > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> > >> > > > > > > - RAM: 7704 MB
> > >> > > > > > > - OS: ubuntu 18.04
> > >> > > > > > > - JDK: openjdk version "1.8.0_171" (default configuration,
> > >> with
> > >> > > > > > MaxHeapSize
> > >> > > > > > > = 1928 MB)
> > >> > > > > > > - Branch: master (commit: cd8ea3d)
> > >> > > > > > >
> > >> > > > > > > The error message I got is:
> > >> > > > > > >
> > >> > > > > > > [INFO]
> > >> > > > > > > >
> > >> > > > > >
> > >> > > >
> > >> >
> > ------------------------------------------------------------------------
> > >> > > > > > > > [INFO] Reactor Summary:
> > >> > > > > > > > [INFO]
> > >> > > > > > > > [INFO] io.druid:druid
> > .....................................
> > >> > > > SUCCESS [
> > >> > > > > > > > 50.258 s]
> > >> > > > > > > > [INFO] java-util
> > ..........................................
> > >> > > SUCCESS
> > >> > > > > > > [03:57
> > >> > > > > > > > min]
> > >> > > > > > > > [INFO] druid-api
> > ..........................................
> > >> > > > SUCCESS [
> > >> > > > > > > > 22.694 s]
> > >> > > > > > > > [INFO] druid-common
> > .......................................
> > >> > > > SUCCESS [
> > >> > > > > > > > 14.083 s]
> > >> > > > > > > > [INFO] druid-hll
> > ..........................................
> > >> > > > SUCCESS [
> > >> > > > > > > > 17.126 s]
> > >> > > > > > > > [INFO] extendedset
> > ........................................
> > >> > > > SUCCESS [
> > >> > > > > > > > 10.856 s]
> > >> > > > > > > >
> > >> > > > > > > > *[INFO] druid-processing
> > ...................................
> > >> > > > FAILURE
> > >> > > > > > > > [04:36 min]*[INFO] druid-aws-common
> > >> > > > > ...................................
> > >> > > > > > > > SKIPPED
> > >> > > > > > > > [INFO] druid-server
> > .......................................
> > >> > > SKIPPED
> > >> > > > > > > > [INFO] druid-examples
> > .....................................
> > >> > > SKIPPED
> > >> > > > > > > > ...
> > >> > > > > > > > [INFO]
> > >> > > > > > > >
> > >> > > > > >
> > >> > > >
> > >> >
> > ------------------------------------------------------------------------
> > >> > > > > > > > [INFO] BUILD FAILURE
> > >> > > > > > > > [INFO]
> > >> > > > > > > >
> > >> > > > > >
> > >> > > >
> > >> >
> > ------------------------------------------------------------------------
> > >> > > > > > > > [INFO] Total time: 10:29 min
> > >> > > > > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> > >> > > > > > > > [INFO] Final Memory: 88M/777M
> > >> > > > > > > > [INFO]
> > >> > > > > > > >
> > >> > > > > >
> > >> > > >
> > >> >
> > ------------------------------------------------------------------------
> > >> > > > > > > >
> > >> > > > > > > > *[ERROR] Failed to execute goal
> > >> > > > > > > >
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > >> > > > > > (default-test)
> > >> > > > > > > > on project druid-processing: Execution default-test of
> > goal
> > >> > > > > > > >
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > >> > > failed:
> > >> > > > > The
> > >> > > > > > > > forked VM terminated without properly saying goodbye. VM
> > >> crash
> > >> > or
> > >> > > > > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
> > >> > > > > > > > /home/djlee/workspace/java/druid/processing &&
> > >> > > > > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> > >> > > > > > > -Duser.language=en
> > >> > > > > > > > -Duser.country=US -Dfile.encoding=UTF-8
> > -Duser.timezone=UTC
> > >> > > > > > > >
> > >> > >
> -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> > >> > > > > > > > -Ddruid.indexing.doubleStorage=double -jar
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > It seems like it fails when it runs tests on
> > >> `druid-processing`
> > >> > > > module
> > >> > > > > > but
> > >> > > > > > > I can't certain. Is there anyone who can give me some
> hints?
> > >> > Thanks
> > >> > > > in
> > >> > > > > > > advance.
> > >> > > > > > >
> > >> > > > > > > Best,
> > >> > > > > > > Dongjin
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > *Dongjin Lee*
> > >> > > > > > >
> > >> > > > > > > *A hitchhiker in the mathematical world.*
> > >> > > > > > >
> > >>
> > > > > > > > > *github: <http://goog_969573159/
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> > >
> > >> >github.com/dongjinleekr
> > >> > > > > > > <http://github.com/dongjinleekr>linkedin:
> > >> > > > > > kr.linkedin.com/in/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> > >
> > >> > > > > > > <http://kr.linkedin.com/in/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> > >
> > >> >slideshare:
> > >> > > > > > > www.slideshare.net/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> > >
> > >> > > > > > > <http://www.slideshare.net/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> > >
> > >> >*
> > >
> > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > > --
> > >> > > > > > *Dongjin Lee*
> > >> > > > > >
> > >> > > > > > *A hitchhiker in the mathematical world.*
> > >> > > > > >
> > >>
> > > > > > > > *github: <http://goog_969573159/
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> > >
> > >> >github.com/dongjinleekr
> > >> > > > > > <http://github.com/dongjinleekr>linkedin:
> > >> > > > > kr.linkedin.com/in/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> > >
> > >> > > > > > <http://kr.linkedin.com/in/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> > >
> > >> >slideshare:
> > >> > > > > www.slideshare.net/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> > >
> > >> > > > > > <http://www.slideshare.net/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> > >
> > >> >*
> > >
> > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > > > --
> > >> > > > *Dongjin Lee*
> > >> > > >
> > >> > > > *A hitchhiker in the mathematical world.*
> > >> > > >
> > >>
> > > > > > *github: <http://goog_969573159/
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> > >
> > >> >github.com/dongjinleekr
> > >> > > > <http://github.com/dongjinleekr>linkedin:
> > >> > > kr.linkedin.com/in/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> > >
> > >> > > > <http://kr.linkedin.com/in/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> > >
> > >> >slideshare:
> > >> > > www.slideshare.net/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> > >
> > >> > > > <http://www.slideshare.net/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> > >
> > >> >*
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >> --
> > >> *Dongjin Lee*
> > >>
> > >> *A hitchhiker in the mathematical world.*
> > >>
> > >> *github:
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> > >github.com/dongjinleekr
> > >> <http://github.com/dongjinleekr>linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >slideshare:
> > www.slideshare.net/dongjinleekr
> > >> <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> > >*
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > For additional commands, e-mail: dev-help@druid.apache.org
> >
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github: <http://goog_969573159/>github.com/dongjinleekr
> <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <http://kr.linkedin.com/in/dongjinleekr>slideshare: www.slideshare.net/dongjinleekr
> <http://www.slideshare.net/dongjinleekr>*
>
Re: Build failure on 0.13.SNAPSHOT
Posted by Jihoon Son <ji...@apache.org>.
@Dongjin, thank you for looking into this. I got some follow-up questions.
> Druid uses above 8gb of memory for testing.
How did you get this result? Did you use some tools to get the memory usage
of only the maven process for Druid tests? When I checked last time using
JMC, I could see that running all unit tests of each Travis Job usually use
up to 512 MB of memory (you can find our Travis Job configuration here:
https://github.com/apache/incubator-druid/blob/master/.travis.yml). That's
why I set the memory limit to 512 MB for our Travis CI. Maybe we need to
check again and raise the memory limit to avoid frequent OOM failures in
groupBy query tests.
Also, if it does use 8 GB of memory, I suspect it's a sort of bug or
something. I don't think even our integration tests don't use such a large
memory which run all Druid processes, ZooKeeper, and Kafka. So, we should
figure out which tests use such a lot of memory and fix it instead of
fixing documents.
> Fwiw I think 8GB should be enough memory with the right settings, since
our Travis CI build environment only has about 7.5GB.
@Gian, the memory of the Travis instance is shared by all processes
including OS. AFAIK, the OS requires about 1 GB of memory. It means that
our unit tests can't be run on Travis if it needs 8 GB of memory.
Best,
Jihoon
On Wed, Jul 25, 2018 at 10:01 AM Charles Allen
<ch...@snap.com.invalid> wrote:
> OOME seems to be showing up in some of the Travis testing as well for group
> by related stuff. Unsure what's going on there.
>
> On Tue, Jul 24, 2018 at 9:46 PM Dongjin Lee <do...@apache.org> wrote:
>
> > After some experiments, I figured out the following:
> >
> > 1. Druid uses above 8gb of memory for testing. (building-druid.png)
> > 2. With 8gb(physical)+4gb(swap) of memory, the test succeeds regardless
> of
> > maven version (3.3.9, 3.5.2, 3.5.4) or MAVEN_OPTS. However, with
> > 8gb(physical)+2gb(swap) of memory[^1], some tests failed. The list of
> > failing tests differs between maven 3.3.9 and 3.5.2.
> >
> > In short, retaining sufficient memory solved the problem - *It seems like
> > 12gb of memory is a recommended setting for building druid.* (I guess
> > lots of you are working with the MacBook Pro with 16gb RAM, right? In
> that
> > case, you must not have encountered this problem.)
> >
> > If you are okay, may I update the building documentation for the newbies
> > like me?
> >
> > Thanks,
> > Dongjin
> >
> > +1. While building Druid, I found another problem. But this issue should
> > be discussed in another thread.
> >
> > [^1]: You know, the other processes also occupy the memory.
> >
> >
> > On Tue, Jul 24, 2018 at 3:07 AM Jihoon Son <ji...@apache.org> wrote:
> >
> >> I'm also using Maven 3.5.2 and not using any special configurations for
> >> Maven, but I have never seen that error too.
> >> Most of our Travis jobs have been working with only 512 MB of direct
> >> memory. Only the 'strict compilation' Travis job requires 3 GB of
> memory.
> >>
> >> I think it's worthwhile to look into this more. Maybe we somehow use
> more
> >> memory when we run all tests by 'mvn install'. Maybe this relates to the
> >> frequent transient failures of 'processing module test', one of our
> Travis
> >> jobs.
> >>
> >> Jihoon
> >>
> >> On Mon, Jul 23, 2018 at 9:32 AM Gian Merlino <gi...@apache.org> wrote:
> >>
> >> > Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it
> has
> >> > been working for for me. I don't think I"m using any special Maven
> >> > overrides (at least, I don't see anything interesting in my ~/.m2
> >> directory
> >> > or in my environment variables). It might have to do with how much
> >> memory
> >> > our machines have? I do most of my builds on a Mac with 16GB RAM.
> Maybe
> >> try
> >> > checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn
> >> install
> >> > commands, which might be needed for more low memory environments.
> >> >
> >> > $ mvn --version
> >> > Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
> >> > 2017-10-18T00:58:13-07:00)
> >> > Maven home: /usr/local/Cellar/maven/3.5.2/libexec
> >> > Java version: 1.8.0_161, vendor: Oracle Corporation
> >> > Java home:
> >> > /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre
> >> > Default locale: en_US, platform encoding: UTF-8
> >> > OS name: "mac os x", version: "10.13.5", arch: "x86_64", family: "mac"
> >> >
> >> > On Mon, Jul 23, 2018 at 6:40 AM Dongjin Lee <do...@apache.org>
> wrote:
> >> >
> >> > > Finally, it seems like I found the reason. It was a composition of
> >> > several
> >> > > problems:
> >> > >
> >> > > - Druid should not be built with maven 3.5.x. With 3.5.2, Test
> suites
> >> > like
> >> > > `GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9
> >> which
> >> > is
> >> > > built in the latest version of IntelliJ, those errors disappeared.
> It
> >> > seems
> >> > > like maven 3.5.x is not stable yet - it applied a drastic change,
> and
> >> it
> >> > is
> >> > > also why they skipped 3.4.x.
> >> > > - It seems like Druid requires some MaxDirectMemorySize
> configuration
> >> for
> >> > > some test suites. With some JVM parameter like
> >> > `-XX:MaxDirectMemorySize=4g`
> >> > > some test suites were passed, but not all. I am now trying the other
> >> > > options with enlarged swap space.
> >> > >
> >> > > Question: How much MaxDirectMemorySize configuration are you using?
> >> > >
> >> > > Best,
> >> > > Dongjin
> >> > >
> >> > > On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <gh...@gmail.com>
> >> wrote:
> >> > >
> >> > > > Hi Dongjin,
> >> > > >
> >> > > > that is weird. It looks like the vm crashed because of out of
> memory
> >> > > while
> >> > > > testing.
> >> > > > It might be a real issue or not.
> >> > > > Have you set any memory configuration for your maven?
> >> > > >
> >> > > > Jihoon
> >> > > >
> >> > > > On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <do...@apache.org>
> >> > wrote:
> >> > > >
> >> > > > > Hi Jihoon,
> >> > > > >
> >> > > > > I ran `mvn clean package` following development/build
> >> > > > > <
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
> >> > > > > >
> >> > > > > .
> >> > > > >
> >> > > > > Dongjin
> >> > > > >
> >> > > > > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <
> jihoonson@apache.org
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > > Hi Dongjin,
> >> > > > > >
> >> > > > > > what maven command did you run?
> >> > > > > >
> >> > > > > > Jihoon
> >> > > > > >
> >> > > > > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <
> >> dongjin@apache.org>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Hello. I am trying to build druid, but it fails. My
> >> environment
> >> > is
> >> > > > like
> >> > > > > > the
> >> > > > > > > following:
> >> > > > > > >
> >> > > > > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> >> > > > > > > - RAM: 7704 MB
> >> > > > > > > - OS: ubuntu 18.04
> >> > > > > > > - JDK: openjdk version "1.8.0_171" (default configuration,
> >> with
> >> > > > > > MaxHeapSize
> >> > > > > > > = 1928 MB)
> >> > > > > > > - Branch: master (commit: cd8ea3d)
> >> > > > > > >
> >> > > > > > > The error message I got is:
> >> > > > > > >
> >> > > > > > > [INFO]
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> ------------------------------------------------------------------------
> >> > > > > > > > [INFO] Reactor Summary:
> >> > > > > > > > [INFO]
> >> > > > > > > > [INFO] io.druid:druid
> .....................................
> >> > > > SUCCESS [
> >> > > > > > > > 50.258 s]
> >> > > > > > > > [INFO] java-util
> ..........................................
> >> > > SUCCESS
> >> > > > > > > [03:57
> >> > > > > > > > min]
> >> > > > > > > > [INFO] druid-api
> ..........................................
> >> > > > SUCCESS [
> >> > > > > > > > 22.694 s]
> >> > > > > > > > [INFO] druid-common
> .......................................
> >> > > > SUCCESS [
> >> > > > > > > > 14.083 s]
> >> > > > > > > > [INFO] druid-hll
> ..........................................
> >> > > > SUCCESS [
> >> > > > > > > > 17.126 s]
> >> > > > > > > > [INFO] extendedset
> ........................................
> >> > > > SUCCESS [
> >> > > > > > > > 10.856 s]
> >> > > > > > > >
> >> > > > > > > > *[INFO] druid-processing
> ...................................
> >> > > > FAILURE
> >> > > > > > > > [04:36 min]*[INFO] druid-aws-common
> >> > > > > ...................................
> >> > > > > > > > SKIPPED
> >> > > > > > > > [INFO] druid-server
> .......................................
> >> > > SKIPPED
> >> > > > > > > > [INFO] druid-examples
> .....................................
> >> > > SKIPPED
> >> > > > > > > > ...
> >> > > > > > > > [INFO]
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> ------------------------------------------------------------------------
> >> > > > > > > > [INFO] BUILD FAILURE
> >> > > > > > > > [INFO]
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> ------------------------------------------------------------------------
> >> > > > > > > > [INFO] Total time: 10:29 min
> >> > > > > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> >> > > > > > > > [INFO] Final Memory: 88M/777M
> >> > > > > > > > [INFO]
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> ------------------------------------------------------------------------
> >> > > > > > > >
> >> > > > > > > > *[ERROR] Failed to execute goal
> >> > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> >> > > > > > (default-test)
> >> > > > > > > > on project druid-processing: Execution default-test of
> goal
> >> > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> >> > > failed:
> >> > > > > The
> >> > > > > > > > forked VM terminated without properly saying goodbye. VM
> >> crash
> >> > or
> >> > > > > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
> >> > > > > > > > /home/djlee/workspace/java/druid/processing &&
> >> > > > > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> >> > > > > > > -Duser.language=en
> >> > > > > > > > -Duser.country=US -Dfile.encoding=UTF-8
> -Duser.timezone=UTC
> >> > > > > > > >
> >> > > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> >> > > > > > > > -Ddruid.indexing.doubleStorage=double -jar
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > It seems like it fails when it runs tests on
> >> `druid-processing`
> >> > > > module
> >> > > > > > but
> >> > > > > > > I can't certain. Is there anyone who can give me some hints?
> >> > Thanks
> >> > > > in
> >> > > > > > > advance.
> >> > > > > > >
> >> > > > > > > Best,
> >> > > > > > > Dongjin
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > *Dongjin Lee*
> >> > > > > > >
> >> > > > > > > *A hitchhiker in the mathematical world.*
> >> > > > > > >
> >>
> > > > > > > > *github: <http://goog_969573159/
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> >
> >> >github.com/dongjinleekr
> >> > > > > > > <http://github.com/dongjinleekr>linkedin:
> >> > > > > > kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> > > > > > > <http://kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> >slideshare:
> >> > > > > > > www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> > > > > > > <http://www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> >*
> >
> >
> >> > > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > *Dongjin Lee*
> >> > > > > >
> >> > > > > > *A hitchhiker in the mathematical world.*
> >> > > > > >
> >>
> > > > > > > *github: <http://goog_969573159/
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> >
> >> >github.com/dongjinleekr
> >> > > > > > <http://github.com/dongjinleekr>linkedin:
> >> > > > > kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> > > > > > <http://kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> >slideshare:
> >> > > > > www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> > > > > > <http://www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> >*
> >
> >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > > --
> >> > > > *Dongjin Lee*
> >> > > >
> >> > > > *A hitchhiker in the mathematical world.*
> >> > > >
> >>
> > > > > *github: <http://goog_969573159/
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> >
> >> >github.com/dongjinleekr
> >> > > > <http://github.com/dongjinleekr>linkedin:
> >> > > kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> > > > <http://kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> >slideshare:
> >> > > www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> > > > <http://www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> >*
> >> > > >
> >> > >
> >> >
> >>
> >> --
> >> *Dongjin Lee*
> >>
> >> *A hitchhiker in the mathematical world.*
> >>
> >> *github:
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> >github.com/dongjinleekr
> >> <http://github.com/dongjinleekr>linkedin:
> kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=>slideshare:
> www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >*
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > For additional commands, e-mail: dev-help@druid.apache.org
>
Re: Build failure on 0.13.SNAPSHOT
Posted by Charles Allen <ch...@snap.com.INVALID>.
OOME seems to be showing up in some of the Travis testing as well for group
by related stuff. Unsure what's going on there.
On Tue, Jul 24, 2018 at 9:46 PM Dongjin Lee <do...@apache.org> wrote:
> After some experiments, I figured out the following:
>
> 1. Druid uses above 8gb of memory for testing. (building-druid.png)
> 2. With 8gb(physical)+4gb(swap) of memory, the test succeeds regardless of
> maven version (3.3.9, 3.5.2, 3.5.4) or MAVEN_OPTS. However, with
> 8gb(physical)+2gb(swap) of memory[^1], some tests failed. The list of
> failing tests differs between maven 3.3.9 and 3.5.2.
>
> In short, retaining sufficient memory solved the problem - *It seems like
> 12gb of memory is a recommended setting for building druid.* (I guess
> lots of you are working with the MacBook Pro with 16gb RAM, right? In that
> case, you must not have encountered this problem.)
>
> If you are okay, may I update the building documentation for the newbies
> like me?
>
> Thanks,
> Dongjin
>
> +1. While building Druid, I found another problem. But this issue should
> be discussed in another thread.
>
> [^1]: You know, the other processes also occupy the memory.
>
>
> On Tue, Jul 24, 2018 at 3:07 AM Jihoon Son <ji...@apache.org> wrote:
>
>> I'm also using Maven 3.5.2 and not using any special configurations for
>> Maven, but I have never seen that error too.
>> Most of our Travis jobs have been working with only 512 MB of direct
>> memory. Only the 'strict compilation' Travis job requires 3 GB of memory.
>>
>> I think it's worthwhile to look into this more. Maybe we somehow use more
>> memory when we run all tests by 'mvn install'. Maybe this relates to the
>> frequent transient failures of 'processing module test', one of our Travis
>> jobs.
>>
>> Jihoon
>>
>> On Mon, Jul 23, 2018 at 9:32 AM Gian Merlino <gi...@apache.org> wrote:
>>
>> > Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it has
>> > been working for for me. I don't think I"m using any special Maven
>> > overrides (at least, I don't see anything interesting in my ~/.m2
>> directory
>> > or in my environment variables). It might have to do with how much
>> memory
>> > our machines have? I do most of my builds on a Mac with 16GB RAM. Maybe
>> try
>> > checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn
>> install
>> > commands, which might be needed for more low memory environments.
>> >
>> > $ mvn --version
>> > Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
>> > 2017-10-18T00:58:13-07:00)
>> > Maven home: /usr/local/Cellar/maven/3.5.2/libexec
>> > Java version: 1.8.0_161, vendor: Oracle Corporation
>> > Java home:
>> > /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre
>> > Default locale: en_US, platform encoding: UTF-8
>> > OS name: "mac os x", version: "10.13.5", arch: "x86_64", family: "mac"
>> >
>> > On Mon, Jul 23, 2018 at 6:40 AM Dongjin Lee <do...@apache.org> wrote:
>> >
>> > > Finally, it seems like I found the reason. It was a composition of
>> > several
>> > > problems:
>> > >
>> > > - Druid should not be built with maven 3.5.x. With 3.5.2, Test suites
>> > like
>> > > `GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9
>> which
>> > is
>> > > built in the latest version of IntelliJ, those errors disappeared. It
>> > seems
>> > > like maven 3.5.x is not stable yet - it applied a drastic change, and
>> it
>> > is
>> > > also why they skipped 3.4.x.
>> > > - It seems like Druid requires some MaxDirectMemorySize configuration
>> for
>> > > some test suites. With some JVM parameter like
>> > `-XX:MaxDirectMemorySize=4g`
>> > > some test suites were passed, but not all. I am now trying the other
>> > > options with enlarged swap space.
>> > >
>> > > Question: How much MaxDirectMemorySize configuration are you using?
>> > >
>> > > Best,
>> > > Dongjin
>> > >
>> > > On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <gh...@gmail.com>
>> wrote:
>> > >
>> > > > Hi Dongjin,
>> > > >
>> > > > that is weird. It looks like the vm crashed because of out of memory
>> > > while
>> > > > testing.
>> > > > It might be a real issue or not.
>> > > > Have you set any memory configuration for your maven?
>> > > >
>> > > > Jihoon
>> > > >
>> > > > On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <do...@apache.org>
>> > wrote:
>> > > >
>> > > > > Hi Jihoon,
>> > > > >
>> > > > > I ran `mvn clean package` following development/build
>> > > > > <
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
>> > > > > >
>> > > > > .
>> > > > >
>> > > > > Dongjin
>> > > > >
>> > > > > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <jihoonson@apache.org
>> >
>> > > > wrote:
>> > > > >
>> > > > > > Hi Dongjin,
>> > > > > >
>> > > > > > what maven command did you run?
>> > > > > >
>> > > > > > Jihoon
>> > > > > >
>> > > > > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <
>> dongjin@apache.org>
>> > > > wrote:
>> > > > > >
>> > > > > > > Hello. I am trying to build druid, but it fails. My
>> environment
>> > is
>> > > > like
>> > > > > > the
>> > > > > > > following:
>> > > > > > >
>> > > > > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
>> > > > > > > - RAM: 7704 MB
>> > > > > > > - OS: ubuntu 18.04
>> > > > > > > - JDK: openjdk version "1.8.0_171" (default configuration,
>> with
>> > > > > > MaxHeapSize
>> > > > > > > = 1928 MB)
>> > > > > > > - Branch: master (commit: cd8ea3d)
>> > > > > > >
>> > > > > > > The error message I got is:
>> > > > > > >
>> > > > > > > [INFO]
>> > > > > > > >
>> > > > > >
>> > > >
>> > ------------------------------------------------------------------------
>> > > > > > > > [INFO] Reactor Summary:
>> > > > > > > > [INFO]
>> > > > > > > > [INFO] io.druid:druid .....................................
>> > > > SUCCESS [
>> > > > > > > > 50.258 s]
>> > > > > > > > [INFO] java-util ..........................................
>> > > SUCCESS
>> > > > > > > [03:57
>> > > > > > > > min]
>> > > > > > > > [INFO] druid-api ..........................................
>> > > > SUCCESS [
>> > > > > > > > 22.694 s]
>> > > > > > > > [INFO] druid-common .......................................
>> > > > SUCCESS [
>> > > > > > > > 14.083 s]
>> > > > > > > > [INFO] druid-hll ..........................................
>> > > > SUCCESS [
>> > > > > > > > 17.126 s]
>> > > > > > > > [INFO] extendedset ........................................
>> > > > SUCCESS [
>> > > > > > > > 10.856 s]
>> > > > > > > >
>> > > > > > > > *[INFO] druid-processing ...................................
>> > > > FAILURE
>> > > > > > > > [04:36 min]*[INFO] druid-aws-common
>> > > > > ...................................
>> > > > > > > > SKIPPED
>> > > > > > > > [INFO] druid-server .......................................
>> > > SKIPPED
>> > > > > > > > [INFO] druid-examples .....................................
>> > > SKIPPED
>> > > > > > > > ...
>> > > > > > > > [INFO]
>> > > > > > > >
>> > > > > >
>> > > >
>> > ------------------------------------------------------------------------
>> > > > > > > > [INFO] BUILD FAILURE
>> > > > > > > > [INFO]
>> > > > > > > >
>> > > > > >
>> > > >
>> > ------------------------------------------------------------------------
>> > > > > > > > [INFO] Total time: 10:29 min
>> > > > > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
>> > > > > > > > [INFO] Final Memory: 88M/777M
>> > > > > > > > [INFO]
>> > > > > > > >
>> > > > > >
>> > > >
>> > ------------------------------------------------------------------------
>> > > > > > > >
>> > > > > > > > *[ERROR] Failed to execute goal
>> > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
>> > > > > > (default-test)
>> > > > > > > > on project druid-processing: Execution default-test of goal
>> > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
>> > > failed:
>> > > > > The
>> > > > > > > > forked VM terminated without properly saying goodbye. VM
>> crash
>> > or
>> > > > > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
>> > > > > > > > /home/djlee/workspace/java/druid/processing &&
>> > > > > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
>> > > > > > > -Duser.language=en
>> > > > > > > > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
>> > > > > > > >
>> > > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
>> > > > > > > > -Ddruid.indexing.doubleStorage=double -jar
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
>> > > > > > >
>> > > > > > >
>> > > > > > > It seems like it fails when it runs tests on
>> `druid-processing`
>> > > > module
>> > > > > > but
>> > > > > > > I can't certain. Is there anyone who can give me some hints?
>> > Thanks
>> > > > in
>> > > > > > > advance.
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Dongjin
>> > > > > > >
>> > > > > > > --
>> > > > > > > *Dongjin Lee*
>> > > > > > >
>> > > > > > > *A hitchhiker in the mathematical world.*
>> > > > > > >
>>
> > > > > > > *github: <http://goog_969573159/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=>
>> >github.com/dongjinleekr
>> > > > > > > <http://github.com/dongjinleekr>linkedin:
>> > > > > > kr.linkedin.com/in/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=>
>> > > > > > > <http://kr.linkedin.com/in/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=>
>> >slideshare:
>> > > > > > > www.slideshare.net/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=>
>> > > > > > > <http://www.slideshare.net/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=>
>> >*
>
>
>> > > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > *Dongjin Lee*
>> > > > > >
>> > > > > > *A hitchhiker in the mathematical world.*
>> > > > > >
>>
> > > > > > *github: <http://goog_969573159/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=>
>> >github.com/dongjinleekr
>> > > > > > <http://github.com/dongjinleekr>linkedin:
>> > > > > kr.linkedin.com/in/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=>
>> > > > > > <http://kr.linkedin.com/in/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=>
>> >slideshare:
>> > > > > www.slideshare.net/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=>
>> > > > > > <http://www.slideshare.net/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=>
>> >*
>
>
>> > > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > *Dongjin Lee*
>> > > >
>> > > > *A hitchhiker in the mathematical world.*
>> > > >
>>
> > > > *github: <http://goog_969573159/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=>
>> >github.com/dongjinleekr
>> > > > <http://github.com/dongjinleekr>linkedin:
>> > > kr.linkedin.com/in/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=>
>> > > > <http://kr.linkedin.com/in/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=>
>> >slideshare:
>> > > www.slideshare.net/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=>
>> > > > <http://www.slideshare.net/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=>
>> >*
>> > > >
>> > >
>> >
>>
>> --
>> *Dongjin Lee*
>>
>> *A hitchhiker in the mathematical world.*
>>
>> *github:
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=>github.com/dongjinleekr
>> <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=>slideshare: www.slideshare.net/dongjinleekr
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=>*
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
Re: Build failure on 0.13.SNAPSHOT
Posted by Gian Merlino <gi...@apache.org>.
Hi Dongjin,
Thanks for doing this research. I don't see building-druid.png attached to
your mail -- maybe the ASF mailing list ate it? Please do send in a pull
request to update the documentation. Fwiw I think 8GB should be enough
memory with the right settings, since our Travis CI build environment only
has about 7.5GB. We use the sudo-enabled trusty environment described here:
https://docs.travis-ci.com/user/reference/overview/. Our Travis config is
here: https://github.com/apache/incubator-druid/blob/master/.travis.yml
Separately it would also be interesting to see what this memory is being
used for during the build. I wonder if it's unavoidable or if some of the
memory use is 'silly' and can be reduced.
On Tue, Jul 24, 2018 at 9:46 PM Dongjin Lee <do...@apache.org> wrote:
> After some experiments, I figured out the following:
>
> 1. Druid uses above 8gb of memory for testing. (building-druid.png)
> 2. With 8gb(physical)+4gb(swap) of memory, the test succeeds regardless of
> maven version (3.3.9, 3.5.2, 3.5.4) or MAVEN_OPTS. However, with
> 8gb(physical)+2gb(swap) of memory[^1], some tests failed. The list of
> failing tests differs between maven 3.3.9 and 3.5.2.
>
> In short, retaining sufficient memory solved the problem - *It seems like
> 12gb of memory is a recommended setting for building druid.* (I guess
> lots of you are working with the MacBook Pro with 16gb RAM, right? In that
> case, you must not have encountered this problem.)
>
> If you are okay, may I update the building documentation for the newbies
> like me?
>
> Thanks,
> Dongjin
>
> +1. While building Druid, I found another problem. But this issue should
> be discussed in another thread.
>
> [^1]: You know, the other processes also occupy the memory.
>
>
> On Tue, Jul 24, 2018 at 3:07 AM Jihoon Son <ji...@apache.org> wrote:
>
>> I'm also using Maven 3.5.2 and not using any special configurations for
>> Maven, but I have never seen that error too.
>> Most of our Travis jobs have been working with only 512 MB of direct
>> memory. Only the 'strict compilation' Travis job requires 3 GB of memory.
>>
>> I think it's worthwhile to look into this more. Maybe we somehow use more
>> memory when we run all tests by 'mvn install'. Maybe this relates to the
>> frequent transient failures of 'processing module test', one of our Travis
>> jobs.
>>
>> Jihoon
>>
>> On Mon, Jul 23, 2018 at 9:32 AM Gian Merlino <gi...@apache.org> wrote:
>>
>> > Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it has
>> > been working for for me. I don't think I"m using any special Maven
>> > overrides (at least, I don't see anything interesting in my ~/.m2
>> directory
>> > or in my environment variables). It might have to do with how much
>> memory
>> > our machines have? I do most of my builds on a Mac with 16GB RAM. Maybe
>> try
>> > checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn
>> install
>> > commands, which might be needed for more low memory environments.
>> >
>> > $ mvn --version
>> > Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
>> > 2017-10-18T00:58:13-07:00)
>> > Maven home: /usr/local/Cellar/maven/3.5.2/libexec
>> > Java version: 1.8.0_161, vendor: Oracle Corporation
>> > Java home:
>> > /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre
>> > Default locale: en_US, platform encoding: UTF-8
>> > OS name: "mac os x", version: "10.13.5", arch: "x86_64", family: "mac"
>> >
>> > On Mon, Jul 23, 2018 at 6:40 AM Dongjin Lee <do...@apache.org> wrote:
>> >
>> > > Finally, it seems like I found the reason. It was a composition of
>> > several
>> > > problems:
>> > >
>> > > - Druid should not be built with maven 3.5.x. With 3.5.2, Test suites
>> > like
>> > > `GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9
>> which
>> > is
>> > > built in the latest version of IntelliJ, those errors disappeared. It
>> > seems
>> > > like maven 3.5.x is not stable yet - it applied a drastic change, and
>> it
>> > is
>> > > also why they skipped 3.4.x.
>> > > - It seems like Druid requires some MaxDirectMemorySize configuration
>> for
>> > > some test suites. With some JVM parameter like
>> > `-XX:MaxDirectMemorySize=4g`
>> > > some test suites were passed, but not all. I am now trying the other
>> > > options with enlarged swap space.
>> > >
>> > > Question: How much MaxDirectMemorySize configuration are you using?
>> > >
>> > > Best,
>> > > Dongjin
>> > >
>> > > On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <gh...@gmail.com>
>> wrote:
>> > >
>> > > > Hi Dongjin,
>> > > >
>> > > > that is weird. It looks like the vm crashed because of out of memory
>> > > while
>> > > > testing.
>> > > > It might be a real issue or not.
>> > > > Have you set any memory configuration for your maven?
>> > > >
>> > > > Jihoon
>> > > >
>> > > > On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <do...@apache.org>
>> > wrote:
>> > > >
>> > > > > Hi Jihoon,
>> > > > >
>> > > > > I ran `mvn clean package` following development/build
>> > > > > <
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
>> > > > > >
>> > > > > .
>> > > > >
>> > > > > Dongjin
>> > > > >
>> > > > > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <jihoonson@apache.org
>> >
>> > > > wrote:
>> > > > >
>> > > > > > Hi Dongjin,
>> > > > > >
>> > > > > > what maven command did you run?
>> > > > > >
>> > > > > > Jihoon
>> > > > > >
>> > > > > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <
>> dongjin@apache.org>
>> > > > wrote:
>> > > > > >
>> > > > > > > Hello. I am trying to build druid, but it fails. My
>> environment
>> > is
>> > > > like
>> > > > > > the
>> > > > > > > following:
>> > > > > > >
>> > > > > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
>> > > > > > > - RAM: 7704 MB
>> > > > > > > - OS: ubuntu 18.04
>> > > > > > > - JDK: openjdk version "1.8.0_171" (default configuration,
>> with
>> > > > > > MaxHeapSize
>> > > > > > > = 1928 MB)
>> > > > > > > - Branch: master (commit: cd8ea3d)
>> > > > > > >
>> > > > > > > The error message I got is:
>> > > > > > >
>> > > > > > > [INFO]
>> > > > > > > >
>> > > > > >
>> > > >
>> > ------------------------------------------------------------------------
>> > > > > > > > [INFO] Reactor Summary:
>> > > > > > > > [INFO]
>> > > > > > > > [INFO] io.druid:druid .....................................
>> > > > SUCCESS [
>> > > > > > > > 50.258 s]
>> > > > > > > > [INFO] java-util ..........................................
>> > > SUCCESS
>> > > > > > > [03:57
>> > > > > > > > min]
>> > > > > > > > [INFO] druid-api ..........................................
>> > > > SUCCESS [
>> > > > > > > > 22.694 s]
>> > > > > > > > [INFO] druid-common .......................................
>> > > > SUCCESS [
>> > > > > > > > 14.083 s]
>> > > > > > > > [INFO] druid-hll ..........................................
>> > > > SUCCESS [
>> > > > > > > > 17.126 s]
>> > > > > > > > [INFO] extendedset ........................................
>> > > > SUCCESS [
>> > > > > > > > 10.856 s]
>> > > > > > > >
>> > > > > > > > *[INFO] druid-processing ...................................
>> > > > FAILURE
>> > > > > > > > [04:36 min]*[INFO] druid-aws-common
>> > > > > ...................................
>> > > > > > > > SKIPPED
>> > > > > > > > [INFO] druid-server .......................................
>> > > SKIPPED
>> > > > > > > > [INFO] druid-examples .....................................
>> > > SKIPPED
>> > > > > > > > ...
>> > > > > > > > [INFO]
>> > > > > > > >
>> > > > > >
>> > > >
>> > ------------------------------------------------------------------------
>> > > > > > > > [INFO] BUILD FAILURE
>> > > > > > > > [INFO]
>> > > > > > > >
>> > > > > >
>> > > >
>> > ------------------------------------------------------------------------
>> > > > > > > > [INFO] Total time: 10:29 min
>> > > > > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
>> > > > > > > > [INFO] Final Memory: 88M/777M
>> > > > > > > > [INFO]
>> > > > > > > >
>> > > > > >
>> > > >
>> > ------------------------------------------------------------------------
>> > > > > > > >
>> > > > > > > > *[ERROR] Failed to execute goal
>> > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
>> > > > > > (default-test)
>> > > > > > > > on project druid-processing: Execution default-test of goal
>> > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
>> > > failed:
>> > > > > The
>> > > > > > > > forked VM terminated without properly saying goodbye. VM
>> crash
>> > or
>> > > > > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
>> > > > > > > > /home/djlee/workspace/java/druid/processing &&
>> > > > > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
>> > > > > > > -Duser.language=en
>> > > > > > > > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
>> > > > > > > >
>> > > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
>> > > > > > > > -Ddruid.indexing.doubleStorage=double -jar
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
>> > > > > > >
>> > > > > > >
>> > > > > > > It seems like it fails when it runs tests on
>> `druid-processing`
>> > > > module
>> > > > > > but
>> > > > > > > I can't certain. Is there anyone who can give me some hints?
>> > Thanks
>> > > > in
>> > > > > > > advance.
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Dongjin
>> > > > > > >
>> > > > > > > --
>> > > > > > > *Dongjin Lee*
>> > > > > > >
>> > > > > > > *A hitchhiker in the mathematical world.*
>> > > > > > >
>> > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
>> > > > > > > <http://github.com/dongjinleekr>linkedin:
>> > > > > > kr.linkedin.com/in/dongjinleekr
>> > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
>> > > > > > > www.slideshare.net/dongjinleekr
>> > > > > > > <http://www.slideshare.net/dongjinleekr>*
>> > > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > *Dongjin Lee*
>> > > > > >
>> > > > > > *A hitchhiker in the mathematical world.*
>> > > > > >
>> > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
>> > > > > > <http://github.com/dongjinleekr>linkedin:
>> > > > > kr.linkedin.com/in/dongjinleekr
>> > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
>> > > > > www.slideshare.net/dongjinleekr
>> > > > > > <http://www.slideshare.net/dongjinleekr>*
>> > > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > *Dongjin Lee*
>> > > >
>> > > > *A hitchhiker in the mathematical world.*
>> > > >
>> > > > *github: <http://goog_969573159/>github.com/dongjinleekr
>> > > > <http://github.com/dongjinleekr>linkedin:
>> > > kr.linkedin.com/in/dongjinleekr
>> > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
>> > > www.slideshare.net/dongjinleekr
>> > > > <http://www.slideshare.net/dongjinleekr>*
>> > > >
>> > >
>> >
>>
>> --
>> *Dongjin Lee*
>>
>> *A hitchhiker in the mathematical world.*
>>
>> *github: <http://goog_969573159/>github.com/dongjinleekr
>> <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
>> <http://kr.linkedin.com/in/dongjinleekr>slideshare: www.slideshare.net/dongjinleekr
>> <http://www.slideshare.net/dongjinleekr>*
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
Re: Build failure on 0.13.SNAPSHOT
Posted by Dongjin Lee <do...@apache.org>.
After some experiments, I figured out the following:
1. Druid uses above 8gb of memory for testing. (building-druid.png)
2. With 8gb(physical)+4gb(swap) of memory, the test succeeds regardless of
maven version (3.3.9, 3.5.2, 3.5.4) or MAVEN_OPTS. However, with
8gb(physical)+2gb(swap) of memory[^1], some tests failed. The list of
failing tests differs between maven 3.3.9 and 3.5.2.
In short, retaining sufficient memory solved the problem - *It seems like
12gb of memory is a recommended setting for building druid.* (I guess lots
of you are working with the MacBook Pro with 16gb RAM, right? In that case,
you must not have encountered this problem.)
If you are okay, may I update the building documentation for the newbies
like me?
Thanks,
Dongjin
+1. While building Druid, I found another problem. But this issue should be
discussed in another thread.
[^1]: You know, the other processes also occupy the memory.
On Tue, Jul 24, 2018 at 3:07 AM Jihoon Son <ji...@apache.org> wrote:
> I'm also using Maven 3.5.2 and not using any special configurations for
> Maven, but I have never seen that error too.
> Most of our Travis jobs have been working with only 512 MB of direct
> memory. Only the 'strict compilation' Travis job requires 3 GB of memory.
>
> I think it's worthwhile to look into this more. Maybe we somehow use more
> memory when we run all tests by 'mvn install'. Maybe this relates to the
> frequent transient failures of 'processing module test', one of our Travis
> jobs.
>
> Jihoon
>
> On Mon, Jul 23, 2018 at 9:32 AM Gian Merlino <gi...@apache.org> wrote:
>
> > Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it has
> > been working for for me. I don't think I"m using any special Maven
> > overrides (at least, I don't see anything interesting in my ~/.m2
> directory
> > or in my environment variables). It might have to do with how much memory
> > our machines have? I do most of my builds on a Mac with 16GB RAM. Maybe
> try
> > checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn install
> > commands, which might be needed for more low memory environments.
> >
> > $ mvn --version
> > Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
> > 2017-10-18T00:58:13-07:00)
> > Maven home: /usr/local/Cellar/maven/3.5.2/libexec
> > Java version: 1.8.0_161, vendor: Oracle Corporation
> > Java home:
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "mac os x", version: "10.13.5", arch: "x86_64", family: "mac"
> >
> > On Mon, Jul 23, 2018 at 6:40 AM Dongjin Lee <do...@apache.org> wrote:
> >
> > > Finally, it seems like I found the reason. It was a composition of
> > several
> > > problems:
> > >
> > > - Druid should not be built with maven 3.5.x. With 3.5.2, Test suites
> > like
> > > `GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9
> which
> > is
> > > built in the latest version of IntelliJ, those errors disappeared. It
> > seems
> > > like maven 3.5.x is not stable yet - it applied a drastic change, and
> it
> > is
> > > also why they skipped 3.4.x.
> > > - It seems like Druid requires some MaxDirectMemorySize configuration
> for
> > > some test suites. With some JVM parameter like
> > `-XX:MaxDirectMemorySize=4g`
> > > some test suites were passed, but not all. I am now trying the other
> > > options with enlarged swap space.
> > >
> > > Question: How much MaxDirectMemorySize configuration are you using?
> > >
> > > Best,
> > > Dongjin
> > >
> > > On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <gh...@gmail.com> wrote:
> > >
> > > > Hi Dongjin,
> > > >
> > > > that is weird. It looks like the vm crashed because of out of memory
> > > while
> > > > testing.
> > > > It might be a real issue or not.
> > > > Have you set any memory configuration for your maven?
> > > >
> > > > Jihoon
> > > >
> > > > On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <do...@apache.org>
> > wrote:
> > > >
> > > > > Hi Jihoon,
> > > > >
> > > > > I ran `mvn clean package` following development/build
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
> > > > > >
> > > > > .
> > > > >
> > > > > Dongjin
> > > > >
> > > > > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <ji...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Hi Dongjin,
> > > > > >
> > > > > > what maven command did you run?
> > > > > >
> > > > > > Jihoon
> > > > > >
> > > > > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <dongjin@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > Hello. I am trying to build druid, but it fails. My environment
> > is
> > > > like
> > > > > > the
> > > > > > > following:
> > > > > > >
> > > > > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> > > > > > > - RAM: 7704 MB
> > > > > > > - OS: ubuntu 18.04
> > > > > > > - JDK: openjdk version "1.8.0_171" (default configuration, with
> > > > > > MaxHeapSize
> > > > > > > = 1928 MB)
> > > > > > > - Branch: master (commit: cd8ea3d)
> > > > > > >
> > > > > > > The error message I got is:
> > > > > > >
> > > > > > > [INFO]
> > > > > > > >
> > > > > >
> > > >
> > ------------------------------------------------------------------------
> > > > > > > > [INFO] Reactor Summary:
> > > > > > > > [INFO]
> > > > > > > > [INFO] io.druid:druid .....................................
> > > > SUCCESS [
> > > > > > > > 50.258 s]
> > > > > > > > [INFO] java-util ..........................................
> > > SUCCESS
> > > > > > > [03:57
> > > > > > > > min]
> > > > > > > > [INFO] druid-api ..........................................
> > > > SUCCESS [
> > > > > > > > 22.694 s]
> > > > > > > > [INFO] druid-common .......................................
> > > > SUCCESS [
> > > > > > > > 14.083 s]
> > > > > > > > [INFO] druid-hll ..........................................
> > > > SUCCESS [
> > > > > > > > 17.126 s]
> > > > > > > > [INFO] extendedset ........................................
> > > > SUCCESS [
> > > > > > > > 10.856 s]
> > > > > > > >
> > > > > > > > *[INFO] druid-processing ...................................
> > > > FAILURE
> > > > > > > > [04:36 min]*[INFO] druid-aws-common
> > > > > ...................................
> > > > > > > > SKIPPED
> > > > > > > > [INFO] druid-server .......................................
> > > SKIPPED
> > > > > > > > [INFO] druid-examples .....................................
> > > SKIPPED
> > > > > > > > ...
> > > > > > > > [INFO]
> > > > > > > >
> > > > > >
> > > >
> > ------------------------------------------------------------------------
> > > > > > > > [INFO] BUILD FAILURE
> > > > > > > > [INFO]
> > > > > > > >
> > > > > >
> > > >
> > ------------------------------------------------------------------------
> > > > > > > > [INFO] Total time: 10:29 min
> > > > > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> > > > > > > > [INFO] Final Memory: 88M/777M
> > > > > > > > [INFO]
> > > > > > > >
> > > > > >
> > > >
> > ------------------------------------------------------------------------
> > > > > > > >
> > > > > > > > *[ERROR] Failed to execute goal
> > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > > > > > (default-test)
> > > > > > > > on project druid-processing: Execution default-test of goal
> > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > > failed:
> > > > > The
> > > > > > > > forked VM terminated without properly saying goodbye. VM
> crash
> > or
> > > > > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
> > > > > > > > /home/djlee/workspace/java/druid/processing &&
> > > > > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> > > > > > > -Duser.language=en
> > > > > > > > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
> > > > > > > >
> > > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> > > > > > > > -Ddruid.indexing.doubleStorage=double -jar
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
> > > > > > >
> > > > > > >
> > > > > > > It seems like it fails when it runs tests on `druid-processing`
> > > > module
> > > > > > but
> > > > > > > I can't certain. Is there anyone who can give me some hints?
> > Thanks
> > > > in
> > > > > > > advance.
> > > > > > >
> > > > > > > Best,
> > > > > > > Dongjin
> > > > > > >
> > > > > > > --
> > > > > > > *Dongjin Lee*
> > > > > > >
> > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > >
> > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > > <http://github.com/dongjinleekr>linkedin:
> > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > > > > www.slideshare.net/dongjinleekr
> > > > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > *Dongjin Lee*
> > > > > >
> > > > > > *A hitchhiker in the mathematical world.*
> > > > > >
> > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > <http://github.com/dongjinleekr>linkedin:
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > > www.slideshare.net/dongjinleekr
> > > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > > >
> > > > >
> > > >
> > > > --
> > > > *Dongjin Lee*
> > > >
> > > > *A hitchhiker in the mathematical world.*
> > > >
> > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > <http://github.com/dongjinleekr>linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > www.slideshare.net/dongjinleekr
> > > > <http://www.slideshare.net/dongjinleekr>*
> > > >
> > >
> >
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github: <http://goog_969573159/>github.com/dongjinleekr
> <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <http://kr.linkedin.com/in/dongjinleekr>slideshare: www.slideshare.net/dongjinleekr
> <http://www.slideshare.net/dongjinleekr>*
>
Re: Build failure on 0.13.SNAPSHOT
Posted by Jihoon Son <ji...@apache.org>.
I'm also using Maven 3.5.2 and not using any special configurations for
Maven, but I have never seen that error too.
Most of our Travis jobs have been working with only 512 MB of direct
memory. Only the 'strict compilation' Travis job requires 3 GB of memory.
I think it's worthwhile to look into this more. Maybe we somehow use more
memory when we run all tests by 'mvn install'. Maybe this relates to the
frequent transient failures of 'processing module test', one of our Travis
jobs.
Jihoon
On Mon, Jul 23, 2018 at 9:32 AM Gian Merlino <gi...@apache.org> wrote:
> Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it has
> been working for for me. I don't think I"m using any special Maven
> overrides (at least, I don't see anything interesting in my ~/.m2 directory
> or in my environment variables). It might have to do with how much memory
> our machines have? I do most of my builds on a Mac with 16GB RAM. Maybe try
> checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn install
> commands, which might be needed for more low memory environments.
>
> $ mvn --version
> Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
> 2017-10-18T00:58:13-07:00)
> Maven home: /usr/local/Cellar/maven/3.5.2/libexec
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.5", arch: "x86_64", family: "mac"
>
> On Mon, Jul 23, 2018 at 6:40 AM Dongjin Lee <do...@apache.org> wrote:
>
> > Finally, it seems like I found the reason. It was a composition of
> several
> > problems:
> >
> > - Druid should not be built with maven 3.5.x. With 3.5.2, Test suites
> like
> > `GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9 which
> is
> > built in the latest version of IntelliJ, those errors disappeared. It
> seems
> > like maven 3.5.x is not stable yet - it applied a drastic change, and it
> is
> > also why they skipped 3.4.x.
> > - It seems like Druid requires some MaxDirectMemorySize configuration for
> > some test suites. With some JVM parameter like
> `-XX:MaxDirectMemorySize=4g`
> > some test suites were passed, but not all. I am now trying the other
> > options with enlarged swap space.
> >
> > Question: How much MaxDirectMemorySize configuration are you using?
> >
> > Best,
> > Dongjin
> >
> > On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <gh...@gmail.com> wrote:
> >
> > > Hi Dongjin,
> > >
> > > that is weird. It looks like the vm crashed because of out of memory
> > while
> > > testing.
> > > It might be a real issue or not.
> > > Have you set any memory configuration for your maven?
> > >
> > > Jihoon
> > >
> > > On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <do...@apache.org>
> wrote:
> > >
> > > > Hi Jihoon,
> > > >
> > > > I ran `mvn clean package` following development/build
> > > > <
> > > >
> > >
> >
> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
> > > > >
> > > > .
> > > >
> > > > Dongjin
> > > >
> > > > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <ji...@apache.org>
> > > wrote:
> > > >
> > > > > Hi Dongjin,
> > > > >
> > > > > what maven command did you run?
> > > > >
> > > > > Jihoon
> > > > >
> > > > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <do...@apache.org>
> > > wrote:
> > > > >
> > > > > > Hello. I am trying to build druid, but it fails. My environment
> is
> > > like
> > > > > the
> > > > > > following:
> > > > > >
> > > > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> > > > > > - RAM: 7704 MB
> > > > > > - OS: ubuntu 18.04
> > > > > > - JDK: openjdk version "1.8.0_171" (default configuration, with
> > > > > MaxHeapSize
> > > > > > = 1928 MB)
> > > > > > - Branch: master (commit: cd8ea3d)
> > > > > >
> > > > > > The error message I got is:
> > > > > >
> > > > > > [INFO]
> > > > > > >
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > > > [INFO] Reactor Summary:
> > > > > > > [INFO]
> > > > > > > [INFO] io.druid:druid .....................................
> > > SUCCESS [
> > > > > > > 50.258 s]
> > > > > > > [INFO] java-util ..........................................
> > SUCCESS
> > > > > > [03:57
> > > > > > > min]
> > > > > > > [INFO] druid-api ..........................................
> > > SUCCESS [
> > > > > > > 22.694 s]
> > > > > > > [INFO] druid-common .......................................
> > > SUCCESS [
> > > > > > > 14.083 s]
> > > > > > > [INFO] druid-hll ..........................................
> > > SUCCESS [
> > > > > > > 17.126 s]
> > > > > > > [INFO] extendedset ........................................
> > > SUCCESS [
> > > > > > > 10.856 s]
> > > > > > >
> > > > > > > *[INFO] druid-processing ...................................
> > > FAILURE
> > > > > > > [04:36 min]*[INFO] druid-aws-common
> > > > ...................................
> > > > > > > SKIPPED
> > > > > > > [INFO] druid-server .......................................
> > SKIPPED
> > > > > > > [INFO] druid-examples .....................................
> > SKIPPED
> > > > > > > ...
> > > > > > > [INFO]
> > > > > > >
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > > > [INFO] BUILD FAILURE
> > > > > > > [INFO]
> > > > > > >
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > > > [INFO] Total time: 10:29 min
> > > > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> > > > > > > [INFO] Final Memory: 88M/777M
> > > > > > > [INFO]
> > > > > > >
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > > >
> > > > > > > *[ERROR] Failed to execute goal
> > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > > > > (default-test)
> > > > > > > on project druid-processing: Execution default-test of goal
> > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > failed:
> > > > The
> > > > > > > forked VM terminated without properly saying goodbye. VM crash
> or
> > > > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
> > > > > > > /home/djlee/workspace/java/druid/processing &&
> > > > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> > > > > > -Duser.language=en
> > > > > > > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
> > > > > > >
> > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> > > > > > > -Ddruid.indexing.doubleStorage=double -jar
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
> > > > > >
> > > > > >
> > > > > > It seems like it fails when it runs tests on `druid-processing`
> > > module
> > > > > but
> > > > > > I can't certain. Is there anyone who can give me some hints?
> Thanks
> > > in
> > > > > > advance.
> > > > > >
> > > > > > Best,
> > > > > > Dongjin
> > > > > >
> > > > > > --
> > > > > > *Dongjin Lee*
> > > > > >
> > > > > > *A hitchhiker in the mathematical world.*
> > > > > >
> > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > <http://github.com/dongjinleekr>linkedin:
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > > > www.slideshare.net/dongjinleekr
> > > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > > >
> > > > >
> > > > > --
> > > > > *Dongjin Lee*
> > > > >
> > > > > *A hitchhiker in the mathematical world.*
> > > > >
> > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > <http://github.com/dongjinleekr>linkedin:
> > > > kr.linkedin.com/in/dongjinleekr
> > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > www.slideshare.net/dongjinleekr
> > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > >
> > > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > <http://github.com/dongjinleekr>linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > www.slideshare.net/dongjinleekr
> > > <http://www.slideshare.net/dongjinleekr>*
> > >
> >
>
Re: Build failure on 0.13.SNAPSHOT
Posted by Gian Merlino <gi...@apache.org>.
Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it has
been working for for me. I don't think I"m using any special Maven
overrides (at least, I don't see anything interesting in my ~/.m2 directory
or in my environment variables). It might have to do with how much memory
our machines have? I do most of my builds on a Mac with 16GB RAM. Maybe try
checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn install
commands, which might be needed for more low memory environments.
$ mvn --version
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
2017-10-18T00:58:13-07:00)
Maven home: /usr/local/Cellar/maven/3.5.2/libexec
Java version: 1.8.0_161, vendor: Oracle Corporation
Java home:
/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.5", arch: "x86_64", family: "mac"
On Mon, Jul 23, 2018 at 6:40 AM Dongjin Lee <do...@apache.org> wrote:
> Finally, it seems like I found the reason. It was a composition of several
> problems:
>
> - Druid should not be built with maven 3.5.x. With 3.5.2, Test suites like
> `GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9 which is
> built in the latest version of IntelliJ, those errors disappeared. It seems
> like maven 3.5.x is not stable yet - it applied a drastic change, and it is
> also why they skipped 3.4.x.
> - It seems like Druid requires some MaxDirectMemorySize configuration for
> some test suites. With some JVM parameter like `-XX:MaxDirectMemorySize=4g`
> some test suites were passed, but not all. I am now trying the other
> options with enlarged swap space.
>
> Question: How much MaxDirectMemorySize configuration are you using?
>
> Best,
> Dongjin
>
> On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <gh...@gmail.com> wrote:
>
> > Hi Dongjin,
> >
> > that is weird. It looks like the vm crashed because of out of memory
> while
> > testing.
> > It might be a real issue or not.
> > Have you set any memory configuration for your maven?
> >
> > Jihoon
> >
> > On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <do...@apache.org> wrote:
> >
> > > Hi Jihoon,
> > >
> > > I ran `mvn clean package` following development/build
> > > <
> > >
> >
> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
> > > >
> > > .
> > >
> > > Dongjin
> > >
> > > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <ji...@apache.org>
> > wrote:
> > >
> > > > Hi Dongjin,
> > > >
> > > > what maven command did you run?
> > > >
> > > > Jihoon
> > > >
> > > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <do...@apache.org>
> > wrote:
> > > >
> > > > > Hello. I am trying to build druid, but it fails. My environment is
> > like
> > > > the
> > > > > following:
> > > > >
> > > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> > > > > - RAM: 7704 MB
> > > > > - OS: ubuntu 18.04
> > > > > - JDK: openjdk version "1.8.0_171" (default configuration, with
> > > > MaxHeapSize
> > > > > = 1928 MB)
> > > > > - Branch: master (commit: cd8ea3d)
> > > > >
> > > > > The error message I got is:
> > > > >
> > > > > [INFO]
> > > > > >
> > > >
> > ------------------------------------------------------------------------
> > > > > > [INFO] Reactor Summary:
> > > > > > [INFO]
> > > > > > [INFO] io.druid:druid .....................................
> > SUCCESS [
> > > > > > 50.258 s]
> > > > > > [INFO] java-util ..........................................
> SUCCESS
> > > > > [03:57
> > > > > > min]
> > > > > > [INFO] druid-api ..........................................
> > SUCCESS [
> > > > > > 22.694 s]
> > > > > > [INFO] druid-common .......................................
> > SUCCESS [
> > > > > > 14.083 s]
> > > > > > [INFO] druid-hll ..........................................
> > SUCCESS [
> > > > > > 17.126 s]
> > > > > > [INFO] extendedset ........................................
> > SUCCESS [
> > > > > > 10.856 s]
> > > > > >
> > > > > > *[INFO] druid-processing ...................................
> > FAILURE
> > > > > > [04:36 min]*[INFO] druid-aws-common
> > > ...................................
> > > > > > SKIPPED
> > > > > > [INFO] druid-server .......................................
> SKIPPED
> > > > > > [INFO] druid-examples .....................................
> SKIPPED
> > > > > > ...
> > > > > > [INFO]
> > > > > >
> > > >
> > ------------------------------------------------------------------------
> > > > > > [INFO] BUILD FAILURE
> > > > > > [INFO]
> > > > > >
> > > >
> > ------------------------------------------------------------------------
> > > > > > [INFO] Total time: 10:29 min
> > > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> > > > > > [INFO] Final Memory: 88M/777M
> > > > > > [INFO]
> > > > > >
> > > >
> > ------------------------------------------------------------------------
> > > > > >
> > > > > > *[ERROR] Failed to execute goal
> > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > > > (default-test)
> > > > > > on project druid-processing: Execution default-test of goal
> > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> failed:
> > > The
> > > > > > forked VM terminated without properly saying goodbye. VM crash or
> > > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
> > > > > > /home/djlee/workspace/java/druid/processing &&
> > > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> > > > > -Duser.language=en
> > > > > > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
> > > > > >
> -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> > > > > > -Ddruid.indexing.doubleStorage=double -jar
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
> > > > >
> > > > >
> > > > > It seems like it fails when it runs tests on `druid-processing`
> > module
> > > > but
> > > > > I can't certain. Is there anyone who can give me some hints? Thanks
> > in
> > > > > advance.
> > > > >
> > > > > Best,
> > > > > Dongjin
> > > > >
> > > > > --
> > > > > *Dongjin Lee*
> > > > >
> > > > > *A hitchhiker in the mathematical world.*
> > > > >
> > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > <http://github.com/dongjinleekr>linkedin:
> > > > kr.linkedin.com/in/dongjinleekr
> > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > > www.slideshare.net/dongjinleekr
> > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > >
> > > >
> > > > --
> > > > *Dongjin Lee*
> > > >
> > > > *A hitchhiker in the mathematical world.*
> > > >
> > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > <http://github.com/dongjinleekr>linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > www.slideshare.net/dongjinleekr
> > > > <http://www.slideshare.net/dongjinleekr>*
> > > >
> > >
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> > *github: <http://goog_969573159/>github.com/dongjinleekr
> > <http://github.com/dongjinleekr>linkedin:
> kr.linkedin.com/in/dongjinleekr
> > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> www.slideshare.net/dongjinleekr
> > <http://www.slideshare.net/dongjinleekr>*
> >
>
Re: Build failure on 0.13.SNAPSHOT
Posted by Dongjin Lee <do...@apache.org>.
Finally, it seems like I found the reason. It was a composition of several
problems:
- Druid should not be built with maven 3.5.x. With 3.5.2, Test suites like
`GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9 which is
built in the latest version of IntelliJ, those errors disappeared. It seems
like maven 3.5.x is not stable yet - it applied a drastic change, and it is
also why they skipped 3.4.x.
- It seems like Druid requires some MaxDirectMemorySize configuration for
some test suites. With some JVM parameter like `-XX:MaxDirectMemorySize=4g`
some test suites were passed, but not all. I am now trying the other
options with enlarged swap space.
Question: How much MaxDirectMemorySize configuration are you using?
Best,
Dongjin
On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <gh...@gmail.com> wrote:
> Hi Dongjin,
>
> that is weird. It looks like the vm crashed because of out of memory while
> testing.
> It might be a real issue or not.
> Have you set any memory configuration for your maven?
>
> Jihoon
>
> On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <do...@apache.org> wrote:
>
> > Hi Jihoon,
> >
> > I ran `mvn clean package` following development/build
> > <
> >
> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
> > >
> > .
> >
> > Dongjin
> >
> > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <ji...@apache.org>
> wrote:
> >
> > > Hi Dongjin,
> > >
> > > what maven command did you run?
> > >
> > > Jihoon
> > >
> > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <do...@apache.org>
> wrote:
> > >
> > > > Hello. I am trying to build druid, but it fails. My environment is
> like
> > > the
> > > > following:
> > > >
> > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> > > > - RAM: 7704 MB
> > > > - OS: ubuntu 18.04
> > > > - JDK: openjdk version "1.8.0_171" (default configuration, with
> > > MaxHeapSize
> > > > = 1928 MB)
> > > > - Branch: master (commit: cd8ea3d)
> > > >
> > > > The error message I got is:
> > > >
> > > > [INFO]
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > [INFO] Reactor Summary:
> > > > > [INFO]
> > > > > [INFO] io.druid:druid .....................................
> SUCCESS [
> > > > > 50.258 s]
> > > > > [INFO] java-util .......................................... SUCCESS
> > > > [03:57
> > > > > min]
> > > > > [INFO] druid-api ..........................................
> SUCCESS [
> > > > > 22.694 s]
> > > > > [INFO] druid-common .......................................
> SUCCESS [
> > > > > 14.083 s]
> > > > > [INFO] druid-hll ..........................................
> SUCCESS [
> > > > > 17.126 s]
> > > > > [INFO] extendedset ........................................
> SUCCESS [
> > > > > 10.856 s]
> > > > >
> > > > > *[INFO] druid-processing ...................................
> FAILURE
> > > > > [04:36 min]*[INFO] druid-aws-common
> > ...................................
> > > > > SKIPPED
> > > > > [INFO] druid-server ....................................... SKIPPED
> > > > > [INFO] druid-examples ..................................... SKIPPED
> > > > > ...
> > > > > [INFO]
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > [INFO] BUILD FAILURE
> > > > > [INFO]
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > [INFO] Total time: 10:29 min
> > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> > > > > [INFO] Final Memory: 88M/777M
> > > > > [INFO]
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > >
> > > > > *[ERROR] Failed to execute goal
> > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > > (default-test)
> > > > > on project druid-processing: Execution default-test of goal
> > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test failed:
> > The
> > > > > forked VM terminated without properly saying goodbye. VM crash or
> > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
> > > > > /home/djlee/workspace/java/druid/processing &&
> > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> > > > -Duser.language=en
> > > > > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
> > > > > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> > > > > -Ddruid.indexing.doubleStorage=double -jar
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
> > > >
> > > >
> > > > It seems like it fails when it runs tests on `druid-processing`
> module
> > > but
> > > > I can't certain. Is there anyone who can give me some hints? Thanks
> in
> > > > advance.
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > --
> > > > *Dongjin Lee*
> > > >
> > > > *A hitchhiker in the mathematical world.*
> > > >
> > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > <http://github.com/dongjinleekr>linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > www.slideshare.net/dongjinleekr
> > > > <http://www.slideshare.net/dongjinleekr>*
> > > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > <http://github.com/dongjinleekr>linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > www.slideshare.net/dongjinleekr
> > > <http://www.slideshare.net/dongjinleekr>*
> > >
> >
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github: <http://goog_969573159/>github.com/dongjinleekr
> <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <http://kr.linkedin.com/in/dongjinleekr>slideshare: www.slideshare.net/dongjinleekr
> <http://www.slideshare.net/dongjinleekr>*
>
Re: Build failure on 0.13.SNAPSHOT
Posted by Jihoon Son <gh...@gmail.com>.
Hi Dongjin,
that is weird. It looks like the vm crashed because of out of memory while
testing.
It might be a real issue or not.
Have you set any memory configuration for your maven?
Jihoon
On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <do...@apache.org> wrote:
> Hi Jihoon,
>
> I ran `mvn clean package` following development/build
> <
> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
> >
> .
>
> Dongjin
>
> On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <ji...@apache.org> wrote:
>
> > Hi Dongjin,
> >
> > what maven command did you run?
> >
> > Jihoon
> >
> > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <do...@apache.org> wrote:
> >
> > > Hello. I am trying to build druid, but it fails. My environment is like
> > the
> > > following:
> > >
> > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> > > - RAM: 7704 MB
> > > - OS: ubuntu 18.04
> > > - JDK: openjdk version "1.8.0_171" (default configuration, with
> > MaxHeapSize
> > > = 1928 MB)
> > > - Branch: master (commit: cd8ea3d)
> > >
> > > The error message I got is:
> > >
> > > [INFO]
> > > >
> > ------------------------------------------------------------------------
> > > > [INFO] Reactor Summary:
> > > > [INFO]
> > > > [INFO] io.druid:druid ..................................... SUCCESS [
> > > > 50.258 s]
> > > > [INFO] java-util .......................................... SUCCESS
> > > [03:57
> > > > min]
> > > > [INFO] druid-api .......................................... SUCCESS [
> > > > 22.694 s]
> > > > [INFO] druid-common ....................................... SUCCESS [
> > > > 14.083 s]
> > > > [INFO] druid-hll .......................................... SUCCESS [
> > > > 17.126 s]
> > > > [INFO] extendedset ........................................ SUCCESS [
> > > > 10.856 s]
> > > >
> > > > *[INFO] druid-processing ................................... FAILURE
> > > > [04:36 min]*[INFO] druid-aws-common
> ...................................
> > > > SKIPPED
> > > > [INFO] druid-server ....................................... SKIPPED
> > > > [INFO] druid-examples ..................................... SKIPPED
> > > > ...
> > > > [INFO]
> > > >
> > ------------------------------------------------------------------------
> > > > [INFO] BUILD FAILURE
> > > > [INFO]
> > > >
> > ------------------------------------------------------------------------
> > > > [INFO] Total time: 10:29 min
> > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> > > > [INFO] Final Memory: 88M/777M
> > > > [INFO]
> > > >
> > ------------------------------------------------------------------------
> > > >
> > > > *[ERROR] Failed to execute goal
> > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > (default-test)
> > > > on project druid-processing: Execution default-test of goal
> > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test failed:
> The
> > > > forked VM terminated without properly saying goodbye. VM crash or
> > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
> > > > /home/djlee/workspace/java/druid/processing &&
> > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> > > -Duser.language=en
> > > > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
> > > > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> > > > -Ddruid.indexing.doubleStorage=double -jar
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
> > >
> > >
> > > It seems like it fails when it runs tests on `druid-processing` module
> > but
> > > I can't certain. Is there anyone who can give me some hints? Thanks in
> > > advance.
> > >
> > > Best,
> > > Dongjin
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > <http://github.com/dongjinleekr>linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > www.slideshare.net/dongjinleekr
> > > <http://www.slideshare.net/dongjinleekr>*
> > >
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> > *github: <http://goog_969573159/>github.com/dongjinleekr
> > <http://github.com/dongjinleekr>linkedin:
> kr.linkedin.com/in/dongjinleekr
> > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> www.slideshare.net/dongjinleekr
> > <http://www.slideshare.net/dongjinleekr>*
> >
>
Re: Build failure on 0.13.SNAPSHOT
Posted by Dongjin Lee <do...@apache.org>.
Hi Jihoon,
I ran `mvn clean package` following development/build
<https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md>
.
Dongjin
On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <ji...@apache.org> wrote:
> Hi Dongjin,
>
> what maven command did you run?
>
> Jihoon
>
> On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <do...@apache.org> wrote:
>
> > Hello. I am trying to build druid, but it fails. My environment is like
> the
> > following:
> >
> > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> > - RAM: 7704 MB
> > - OS: ubuntu 18.04
> > - JDK: openjdk version "1.8.0_171" (default configuration, with
> MaxHeapSize
> > = 1928 MB)
> > - Branch: master (commit: cd8ea3d)
> >
> > The error message I got is:
> >
> > [INFO]
> > >
> ------------------------------------------------------------------------
> > > [INFO] Reactor Summary:
> > > [INFO]
> > > [INFO] io.druid:druid ..................................... SUCCESS [
> > > 50.258 s]
> > > [INFO] java-util .......................................... SUCCESS
> > [03:57
> > > min]
> > > [INFO] druid-api .......................................... SUCCESS [
> > > 22.694 s]
> > > [INFO] druid-common ....................................... SUCCESS [
> > > 14.083 s]
> > > [INFO] druid-hll .......................................... SUCCESS [
> > > 17.126 s]
> > > [INFO] extendedset ........................................ SUCCESS [
> > > 10.856 s]
> > >
> > > *[INFO] druid-processing ................................... FAILURE
> > > [04:36 min]*[INFO] druid-aws-common ...................................
> > > SKIPPED
> > > [INFO] druid-server ....................................... SKIPPED
> > > [INFO] druid-examples ..................................... SKIPPED
> > > ...
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > > [INFO] BUILD FAILURE
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > > [INFO] Total time: 10:29 min
> > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> > > [INFO] Final Memory: 88M/777M
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > >
> > > *[ERROR] Failed to execute goal
> > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> (default-test)
> > > on project druid-processing: Execution default-test of goal
> > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test failed: The
> > > forked VM terminated without properly saying goodbye. VM crash or
> > > System.exit called?*[ERROR] Command was /bin/sh -c cd
> > > /home/djlee/workspace/java/druid/processing &&
> > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> > -Duser.language=en
> > > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
> > > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> > > -Ddruid.indexing.doubleStorage=double -jar
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
> >
> >
> > It seems like it fails when it runs tests on `druid-processing` module
> but
> > I can't certain. Is there anyone who can give me some hints? Thanks in
> > advance.
> >
> > Best,
> > Dongjin
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> > *github: <http://goog_969573159/>github.com/dongjinleekr
> > <http://github.com/dongjinleekr>linkedin:
> kr.linkedin.com/in/dongjinleekr
> > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > www.slideshare.net/dongjinleekr
> > <http://www.slideshare.net/dongjinleekr>*
> >
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github: <http://goog_969573159/>github.com/dongjinleekr
> <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <http://kr.linkedin.com/in/dongjinleekr>slideshare: www.slideshare.net/dongjinleekr
> <http://www.slideshare.net/dongjinleekr>*
>
Re: Build failure on 0.13.SNAPSHOT
Posted by Jihoon Son <ji...@apache.org>.
Hi Dongjin,
what maven command did you run?
Jihoon
On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <do...@apache.org> wrote:
> Hello. I am trying to build druid, but it fails. My environment is like the
> following:
>
> - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> - RAM: 7704 MB
> - OS: ubuntu 18.04
> - JDK: openjdk version "1.8.0_171" (default configuration, with MaxHeapSize
> = 1928 MB)
> - Branch: master (commit: cd8ea3d)
>
> The error message I got is:
>
> [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] io.druid:druid ..................................... SUCCESS [
> > 50.258 s]
> > [INFO] java-util .......................................... SUCCESS
> [03:57
> > min]
> > [INFO] druid-api .......................................... SUCCESS [
> > 22.694 s]
> > [INFO] druid-common ....................................... SUCCESS [
> > 14.083 s]
> > [INFO] druid-hll .......................................... SUCCESS [
> > 17.126 s]
> > [INFO] extendedset ........................................ SUCCESS [
> > 10.856 s]
> >
> > *[INFO] druid-processing ................................... FAILURE
> > [04:36 min]*[INFO] druid-aws-common ...................................
> > SKIPPED
> > [INFO] druid-server ....................................... SKIPPED
> > [INFO] druid-examples ..................................... SKIPPED
> > ...
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] BUILD FAILURE
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Total time: 10:29 min
> > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> > [INFO] Final Memory: 88M/777M
> > [INFO]
> > ------------------------------------------------------------------------
> >
> > *[ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test)
> > on project druid-processing: Execution default-test of goal
> > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test failed: The
> > forked VM terminated without properly saying goodbye. VM crash or
> > System.exit called?*[ERROR] Command was /bin/sh -c cd
> > /home/djlee/workspace/java/druid/processing &&
> > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> -Duser.language=en
> > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
> > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> > -Ddruid.indexing.doubleStorage=double -jar
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
>
>
> It seems like it fails when it runs tests on `druid-processing` module but
> I can't certain. Is there anyone who can give me some hints? Thanks in
> advance.
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github: <http://goog_969573159/>github.com/dongjinleekr
> <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> www.slideshare.net/dongjinleekr
> <http://www.slideshare.net/dongjinleekr>*
>