You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/12/17 16:39:31 UTC

[GitHub] [hadoop] amahussein opened a new pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

amahussein opened a new pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560


   This change to increase the memory limit from 20g to 22g.
   The last commit is needed to trigger the tests and increase the timeout. It should be skipped before merging.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] aajisaka commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-747825118


   I asked the infrastructure team how much memory we can use: https://issues.apache.org/jira/browse/INFRA-21207


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] amahussein edited a comment on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
amahussein edited a comment on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-757970031


   @aajisaka , @ericbadger 
   Any thoughts whether we should move fwd with this PR?
   The error still exists as per January 10th build https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/382/#showFailuresLink. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-747919101


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m  7s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 51s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  20m 44s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  20m  0s |  |  trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |  17m 29s |  |  trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  mvnsite  |  25m  5s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  15m 39s |  |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   7m 11s |  |  trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   7m 31s |  |  trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 32s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |  27m 15s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m 47s |  |  the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |  20m 47s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m 24s |  |  the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |  20m 24s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |  24m 45s |  |  the patch passed  |
   | +1 :green_heart: |  shellcheck  |   0m  0s |  |  There were no new shellcheck issues.  |
   | +1 :green_heart: |  shelldocs  |   0m 17s |  |  There were no new shelldocs issues.  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  xml  |   0m  3s |  |  The patch has no ill-formed XML file.  |
   | +1 :green_heart: |  shadedclient  |  19m 49s |  |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   9m 10s |  |  the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   9m 12s |  |  the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  | 617m 25s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2560/1/artifact/out/patch-unit-root.txt) |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 51s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 884m 20s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.hdfs.TestDecommissionWithStriped |
   |   | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
   |   | hadoop.hdfs.TestErasureCodingPolicies |
   |   | hadoop.yarn.applications.distributedshell.TestDistributedShell |
   |   | hadoop.tools.dynamometer.TestDynamometerInfra |
   |   | hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2560/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2560 |
   | Optional Tests | dupname asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit shadedclient xml |
   | uname | Linux 28251d6781c4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 4c033bafa02 |
   | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2560/1/testReport/ |
   | Max. process+thread count | 4052 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project hadoop-yarn-project . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2560/1/console |
   | versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] amahussein commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-750422564


   > The nodes have 98GB of memory and two executors. Therefore we can use about 40GB of memory for each job.
   Thanks @aajisaka !
   I suggest we increase the memory to 22GB.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] aajisaka commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-758439351


   Merged. Thank you @amahussein and @eric-badger 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] aajisaka commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-758437944


   > So I guess go for it. Though I still think this is addressing a symptom, not the root cause
   
   I have the same opinion.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] aajisaka merged pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
aajisaka merged pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] amahussein commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-752206607


   @aajisaka Do you think we should try merging that change (increasing of 2GB) until we find out what is causing the error of OOM?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] ericbadger commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
ericbadger commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-749148544


   I'm not sure how to profile this because I don't really know what access we have to the machines that run Jenkins, what else runs on there, or anything else. Logging memory over time would be a good first step to help narrow it down. Then once we know what module is suspect, we can either investigate on our own local machines or we can enable further debugging on that specific module.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] amahussein commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-748313031


   > I asked the infrastructure team how much memory we can use: https://issues.apache.org/jira/browse/INFRA-21207
   
   Thanks @aajisaka ! 
   
   > It would be nice to know what's tying up all of our memory. Because 20 GB is a lot for us to be using for unit tests
   
   @ericbadger . Definitely!
   It would be easier to narrow the scope of investigation if we had this error reported the earliest it started to happen.
   Unfortunately, it does not seem that it is the case.
   The other way is to profile the memory of the image during the execution. Do you have any suggestion of how to approach this?
   A straightforward way would be to dump the system men to the log at the beginning of each module.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] ericbadger commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
ericbadger commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-747662919


   Jeez, are we really running out of memory in a 20 GB container? That seems insane to me


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] eric-badger commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
eric-badger commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-758288185


   It doesn't really hurt anything to ask for more memory unless we tell it to touch all pages or unless we fill all of the memory. If we fill all of the memory, then I guess that means that we needed it. If we don't, it's not like the kernel can't use it elsewhere (unless we touched those pages). So I guess go for it. Though I still think this is addressing a symptom, not the root cause


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] amahussein commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-757970031


   @aajisaka , @ericbadger 
   Any thoughts whether we should move fwd with this PR?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] ericbadger commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
ericbadger commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-747663571


   It would be nice to know what's tying up all of our memory. Because 20 GB is a lot for us to be using for unit tests


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] amahussein edited a comment on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
amahussein edited a comment on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-757970031


   @aajisaka , @ericbadger 
   Any thoughts whether we should move fwd with this PR?
   The error still exists as per January 10th build https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/382/#showFailuresLink. 
   Furthermore, I see that there is a new module added to the build `hadoop-cloud-storage-project/hadoop-huaweicloud` which is likely going to increase the resources usage.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] aajisaka commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-749933007


   > I asked the infrastructure team how much memory we can use: https://issues.apache.org/jira/browse/INFRA-21207
   
   The nodes have 98GB of memory and two executors. Therefore we can use about 40GB of memory for each job.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2560: HADOOP-17438. Increase docker memory limit in Jenkins.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2560:
URL: https://github.com/apache/hadoop/pull/2560#issuecomment-750452358


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 30s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  shadedclient  |  23m 11s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  shellcheck  |   0m  0s |  |  There were no new shellcheck issues.  |
   | +1 :green_heart: |  shelldocs  |   0m 16s |  |  There were no new shelldocs issues.  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  15m 13s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 38s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  41m 40s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2560/2/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2560 |
   | Optional Tests | dupname asflicense shellcheck shelldocs |
   | uname | Linux b4c8a855c5a9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 16a20503cac |
   | Max. process+thread count | 538 (vs. ulimit of 5500) |
   | modules | C: . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2560/2/console |
   | versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org