You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/03/11 06:01:59 UTC
[GitHub] [hadoop] bilaharith opened a new pull request #1890: HADOOP-16854
Fix to prevent OutOfMemoryException
bilaharith opened a new pull request #1890: HADOOP-16854 Fix to prevent OutOfMemoryException
URL: https://github.com/apache/hadoop/pull/1890
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1890:
HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool
and bytebuffer pool common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#discussion_r394372894
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -63,20 +70,53 @@
private final int bufferSize;
private byte[] buffer;
private int bufferIndex;
- private final int maxConcurrentRequestCount;
+
+ private static int maxConcurrentRequestcount;
+ private static int maxBufferCount;
private ConcurrentLinkedDeque<WriteOperation> writeOperations;
- private final ThreadPoolExecutor threadExecutor;
- private final ExecutorCompletionService<Void> completionService;
+ private static final Object INIT_LOCK = new Object();
+ private static ThreadPoolExecutor threadExecutor;
+ private static ExecutorCompletionService<Void> completionService;
+
+ private static final int ONE_MB = 1024 * 1024;
+ private static final int HUNDRED_MB = 100 * ONE_MB;
+ private static final int MIN_MEMORY_THRESHOLD = HUNDRED_MB;
/**
* Queue storing buffers with the size of the Azure block ready for
* reuse. The pool allows reusing the blocks instead of allocating new
* blocks. After the data is sent to the service, the buffer is returned
* back to the queue
*/
- private final ElasticByteBufferPool byteBufferPool
- = new ElasticByteBufferPool();
+ private static final ElasticByteBufferPool BYTE_BUFFER_POOL
+ = new ElasticByteBufferPool();
+ private static AtomicInteger buffersToBeReturned = new AtomicInteger(0);
+
+ static {
+ if (threadExecutor == null) {
+ synchronized (INIT_LOCK) {
+ if (threadExecutor == null) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ maxConcurrentRequestcount = 4 * availableProcessors;
+ maxBufferCount = maxConcurrentRequestcount + availableProcessors + 1;
Review comment:
I don't like the hard coded assumptions about #of CPUs and amount of space which can be used for buffering.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
bilaharith commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-601364876
Please check the following PR
https://github.com/apache/hadoop/pull/1901
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-599513845
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 51s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 22s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 51s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 4s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| -1 :x: | findbugs | 0m 54s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 58m 54s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Possible doublecheck on org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.threadExecutor in org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.<static initializer for AbfsOutputStream>() At AbfsOutputStream.java:org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.<static initializer for AbfsOutputStream>() At AbfsOutputStream.java:[lines 97-99] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 598fdcaa2f6e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ea68863 |
| Default Java | 1.8.0_242 |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/6/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/6/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1890:
HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool
and bytebuffer pool common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#discussion_r393387356
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -285,14 +311,22 @@ private synchronized void writeCurrentBufferToService() throws IOException {
final byte[] bytes = buffer;
final int bytesLength = bufferIndex;
- buffer = byteBufferPool.getBuffer(false, bufferSize).array();
bufferIndex = 0;
final long offset = position;
position += bytesLength;
- if (threadExecutor.getQueue().size() >= maxConcurrentRequestCount * 2) {
+ if (Runtime.getRuntime().freeMemory() < MIN_MEMORY_THRESHOLD
+ && buffersToBeReturned.get() > 0) {
+ LOG.debug("Low memory");
Review comment:
what if` Runtime.getRuntime().freeMemory() < MIN_MEMORY_THRESHOLD` but `buffersToBeReturned.get()` is `0`?
Can we make the LOG more meaningful like "Low memory, less than 100 MB is available, the existing read tasks consumes 2048 MB, waiting for completion."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-599375514
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 51s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 0s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 51s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 25s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 23s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 23s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| -1 :x: | mvnsite | 0m 24s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 2s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed |
| -1 :x: | findbugs | 0m 27s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 27s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 56m 59s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c4ecc203ffa3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ea68863 |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/3/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/3/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/3/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/3/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/3/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/3/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/3/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith closed pull request #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
bilaharith closed pull request #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1890:
HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool
and bytebuffer pool common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#discussion_r394371435
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemFlush.java
##########
@@ -207,6 +210,70 @@ public Void call() throws Exception {
assertEquals((long) TEST_BUFFER_SIZE * FLUSH_TIMES, fileStatus.getLen());
}
+ @Test
+ public void testWriteWithMultipleOutputStreamAtTheSameTime()
+ throws IOException, InterruptedException, ExecutionException {
+ AzureBlobFileSystem fs = getFileSystem();
+ String testFilePath = methodName.getMethodName();
+ Path[] testPaths = new Path[CONCURRENT_STREAM_OBJS_TEST_OBJ_COUNT];
+ createNStreamsAndWriteDifferentSizesConcurrently(fs, testFilePath,
+ CONCURRENT_STREAM_OBJS_TEST_OBJ_COUNT, testPaths);
+ assertSuccessfulWritesOnAllStreams(fs,
+ CONCURRENT_STREAM_OBJS_TEST_OBJ_COUNT, testPaths);
+ }
+
+ private void assertSuccessfulWritesOnAllStreams(final FileSystem fs,
+ final int numConcurrentObjects, final Path[] testPaths)
+ throws IOException {
+ for (int i = 0; i < numConcurrentObjects; i++) {
+ FileStatus fileStatus = fs.getFileStatus(testPaths[i]);
+ int numWritesMadeOnStream = i + 1;
+ long expectedLength = TEST_BUFFER_SIZE * numWritesMadeOnStream;
+ assertThat(fileStatus.getLen(), is(equalTo(expectedLength)));
+ }
+ }
+
+ private void createNStreamsAndWriteDifferentSizesConcurrently(
+ final FileSystem fs, final String testFilePath,
+ final int numConcurrentObjects, final Path[] testPaths)
+ throws ExecutionException, InterruptedException {
+ final byte[] b = new byte[TEST_BUFFER_SIZE];
+ new Random().nextBytes(b);
+ final ExecutorService es = Executors.newFixedThreadPool(40);
+ final List<Future<Void>> futureTasks = new ArrayList<>();
+ for (int i = 0; i < numConcurrentObjects; i++) {
+ Path testPath = new Path(testFilePath + i);
+ testPaths[i] = testPath;
+ int numWritesToBeDone = i + 1;
+ futureTasks.add(es.submit(() -> {
+ try (FSDataOutputStream stream = fs.create(testPath)) {
+ makeNWritesToStream(stream, numWritesToBeDone, b, es);
+ }
+ return null;
+ }));
+ }
+ for (Future<Void> futureTask : futureTasks) {
+ futureTask.get();
+ }
+ es.shutdownNow();
+ }
+
+ private void makeNWritesToStream(final FSDataOutputStream stream,
+ final int numWrites, final byte[] b, final ExecutorService es)
+ throws IOException, ExecutionException, InterruptedException {
+ final List<Future<Void>> futureTasks = new ArrayList<>();
+ for (int i = 0; i < numWrites; i++) {
+ futureTasks.add(es.submit(() -> {
+ stream.write(b);
+ return null;
+ }));
+ }
+ for (Future<Void> futureTask : futureTasks) {
Review comment:
I'm sure there is a way to block for multiple futures. See also FutureIOSupport.awaitFuture
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-599396267
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 4s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 42s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 50s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 27s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 8s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| -1 :x: | findbugs | 0m 59s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 57m 29s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Possible doublecheck on org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.threadExecutor in new org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream(AbfsClient, String, long, int, boolean, boolean) At AbfsOutputStream.java:new org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream(AbfsClient, String, long, int, boolean, boolean) At AbfsOutputStream.java:[lines 112-114] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 51cd613b37fa 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ea68863 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/5/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/5/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/5/testReport/ |
| Max. process+thread count | 425 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-601348998
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 5s | https://github.com/apache/hadoop/pull/1890 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/13/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-600997678
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 5s | https://github.com/apache/hadoop/pull/1890 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/9/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1890:
HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool
and bytebuffer pool common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-597482104
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 34s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 50s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 50s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 5s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| -1 :x: | findbugs | 0m 54s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 24s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 58m 15s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Possible doublecheck on org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.threadExecutor in new org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream(AbfsClient, String, long, int, boolean, boolean) At AbfsOutputStream.java:new org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream(AbfsClient, String, long, int, boolean, boolean) At AbfsOutputStream.java:[lines 112-114] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0a6b74a2b28b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cf9cf83 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/1/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/1/testReport/ |
| Max. process+thread count | 415 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1890:
HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool
and bytebuffer pool common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#discussion_r394372966
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -63,20 +70,53 @@
private final int bufferSize;
private byte[] buffer;
private int bufferIndex;
- private final int maxConcurrentRequestCount;
+
+ private static int maxConcurrentRequestcount;
+ private static int maxBufferCount;
private ConcurrentLinkedDeque<WriteOperation> writeOperations;
- private final ThreadPoolExecutor threadExecutor;
- private final ExecutorCompletionService<Void> completionService;
+ private static final Object INIT_LOCK = new Object();
+ private static ThreadPoolExecutor threadExecutor;
+ private static ExecutorCompletionService<Void> completionService;
+
+ private static final int ONE_MB = 1024 * 1024;
+ private static final int HUNDRED_MB = 100 * ONE_MB;
+ private static final int MIN_MEMORY_THRESHOLD = HUNDRED_MB;
/**
* Queue storing buffers with the size of the Azure block ready for
* reuse. The pool allows reusing the blocks instead of allocating new
* blocks. After the data is sent to the service, the buffer is returned
* back to the queue
*/
- private final ElasticByteBufferPool byteBufferPool
- = new ElasticByteBufferPool();
+ private static final ElasticByteBufferPool BYTE_BUFFER_POOL
+ = new ElasticByteBufferPool();
+ private static AtomicInteger buffersToBeReturned = new AtomicInteger(0);
+
+ static {
+ if (threadExecutor == null) {
+ synchronized (INIT_LOCK) {
+ if (threadExecutor == null) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
Review comment:
4*core is the existing calculation. Making the same configurable in the next iteration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-601282759
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 5s | https://github.com/apache/hadoop/pull/1890 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/12/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-601107729
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 4s | https://github.com/apache/hadoop/pull/1890 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/11/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-599379560
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 31m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 13s | trunk passed |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 22s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 51s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 22s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 22s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| -1 :x: | mvnsite | 0m 23s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 22s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| -1 :x: | findbugs | 0m 25s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 24s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 92m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a8124915e0c1 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ea68863 |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/2/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/2/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/2/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/2/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/2/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1890:
HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool
and bytebuffer pool common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#discussion_r394373334
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -63,20 +70,53 @@
private final int bufferSize;
private byte[] buffer;
private int bufferIndex;
- private final int maxConcurrentRequestCount;
+
+ private static int maxConcurrentRequestcount;
+ private static int maxBufferCount;
private ConcurrentLinkedDeque<WriteOperation> writeOperations;
- private final ThreadPoolExecutor threadExecutor;
- private final ExecutorCompletionService<Void> completionService;
+ private static final Object INIT_LOCK = new Object();
+ private static ThreadPoolExecutor threadExecutor;
+ private static ExecutorCompletionService<Void> completionService;
+
+ private static final int ONE_MB = 1024 * 1024;
+ private static final int HUNDRED_MB = 100 * ONE_MB;
+ private static final int MIN_MEMORY_THRESHOLD = HUNDRED_MB;
/**
* Queue storing buffers with the size of the Azure block ready for
* reuse. The pool allows reusing the blocks instead of allocating new
* blocks. After the data is sent to the service, the buffer is returned
* back to the queue
*/
- private final ElasticByteBufferPool byteBufferPool
- = new ElasticByteBufferPool();
+ private static final ElasticByteBufferPool BYTE_BUFFER_POOL
Review comment:
ElasticByteBufferPool is is trouble because it never frees cached buffers
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-599388981
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 46s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 25s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 34s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 43s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| -1 :x: | findbugs | 0m 57s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 22s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 66m 49s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Possible doublecheck on org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.threadExecutor in new org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream(AbfsClient, String, long, int, boolean, boolean) At AbfsOutputStream.java:new org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream(AbfsClient, String, long, int, boolean, boolean) At AbfsOutputStream.java:[lines 112-114] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux aa72f0276124 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ea68863 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/4/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/4/testReport/ |
| Max. process+thread count | 457 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1890:
HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool
and bytebuffer pool common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#discussion_r394358730
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -63,20 +70,53 @@
private final int bufferSize;
private byte[] buffer;
private int bufferIndex;
- private final int maxConcurrentRequestCount;
+
+ private static int maxConcurrentRequestcount;
+ private static int maxBufferCount;
private ConcurrentLinkedDeque<WriteOperation> writeOperations;
- private final ThreadPoolExecutor threadExecutor;
- private final ExecutorCompletionService<Void> completionService;
+ private static final Object INIT_LOCK = new Object();
+ private static ThreadPoolExecutor threadExecutor;
+ private static ExecutorCompletionService<Void> completionService;
+
+ private static final int ONE_MB = 1024 * 1024;
+ private static final int HUNDRED_MB = 100 * ONE_MB;
+ private static final int MIN_MEMORY_THRESHOLD = HUNDRED_MB;
/**
* Queue storing buffers with the size of the Azure block ready for
* reuse. The pool allows reusing the blocks instead of allocating new
* blocks. After the data is sent to the service, the buffer is returned
* back to the queue
*/
- private final ElasticByteBufferPool byteBufferPool
- = new ElasticByteBufferPool();
+ private static final ElasticByteBufferPool BYTE_BUFFER_POOL
+ = new ElasticByteBufferPool();
+ private static AtomicInteger buffersToBeReturned = new AtomicInteger(0);
+
+ static {
+ if (threadExecutor == null) {
+ synchronized (INIT_LOCK) {
+ if (threadExecutor == null) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
Review comment:
this is making some big assumptions about exclusive access to CPUs. Why the specific choice of 4*core?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-597482104
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 34s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 50s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 50s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 5s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| -1 :x: | findbugs | 0m 54s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 24s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 58m 15s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Possible doublecheck on org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.threadExecutor in new org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream(AbfsClient, String, long, int, boolean, boolean) At AbfsOutputStream.java:new org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream(AbfsClient, String, long, int, boolean, boolean) At AbfsOutputStream.java:[lines 112-114] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0a6b74a2b28b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cf9cf83 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/1/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/1/testReport/ |
| Max. process+thread count | 415 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-599531863
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 25m 33s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 46s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 49s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| -1 :x: | findbugs | 0m 55s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 22s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 69m 14s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Possible doublecheck on org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.threadExecutor in org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.<static initializer for AbfsOutputStream>() At AbfsOutputStream.java:org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.<static initializer for AbfsOutputStream>() At AbfsOutputStream.java:[lines 97-99] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5bc63547926f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ea68863 |
| Default Java | 1.8.0_242 |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/7/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/7/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/7/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-600821237
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 5s | https://github.com/apache/hadoop/pull/1890 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/8/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1890:
HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool
and bytebuffer pool common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#discussion_r394372525
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -63,20 +70,53 @@
private final int bufferSize;
private byte[] buffer;
private int bufferIndex;
- private final int maxConcurrentRequestCount;
+
+ private static int maxConcurrentRequestcount;
+ private static int maxBufferCount;
private ConcurrentLinkedDeque<WriteOperation> writeOperations;
- private final ThreadPoolExecutor threadExecutor;
- private final ExecutorCompletionService<Void> completionService;
+ private static final Object INIT_LOCK = new Object();
+ private static ThreadPoolExecutor threadExecutor;
+ private static ExecutorCompletionService<Void> completionService;
+
+ private static final int ONE_MB = 1024 * 1024;
+ private static final int HUNDRED_MB = 100 * ONE_MB;
+ private static final int MIN_MEMORY_THRESHOLD = HUNDRED_MB;
/**
* Queue storing buffers with the size of the Azure block ready for
* reuse. The pool allows reusing the blocks instead of allocating new
* blocks. After the data is sent to the service, the buffer is returned
* back to the queue
*/
- private final ElasticByteBufferPool byteBufferPool
- = new ElasticByteBufferPool();
+ private static final ElasticByteBufferPool BYTE_BUFFER_POOL
+ = new ElasticByteBufferPool();
+ private static AtomicInteger buffersToBeReturned = new AtomicInteger(0);
+
+ static {
+ if (threadExecutor == null) {
+ synchronized (INIT_LOCK) {
+ if (threadExecutor == null) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ maxConcurrentRequestcount = 4 * availableProcessors;
+ maxBufferCount = maxConcurrentRequestcount + availableProcessors + 1;
+ ThreadFactory deamonThreadFactory = new ThreadFactory() {
+ @Override
+ public Thread newThread(Runnable runnable) {
+ Thread deamonThread = Executors.defaultThreadFactory()
+ .newThread(runnable);
+ deamonThread.setDaemon(true);
+ return deamonThread;
+ }
+ };
+ threadExecutor = new ThreadPoolExecutor(maxConcurrentRequestcount,
Review comment:
use HadoopExecutors for executors if possible
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool
common across all AbfsOutputStream instances
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1890: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#issuecomment-601093951
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 4s | https://github.com/apache/hadoop/pull/1890 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/1890 |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1890/10/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org