You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/02/21 14:20:34 UTC
[GitHub] [hadoop-ozone] elek opened a new pull request #578: HDDS-3053.
Decrease the number of the chunk writer threads
elek opened a new pull request #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578
## What changes were proposed in this pull request?
As of now we create 60 threads (`dfs.container.ratis.num.write.chunk.threads`) to write chunk data to the disk. As the write is limited by the IO I can't see any benefit to have so many threads. High number of thread means a high context switch overhead, therefore it seems to be more reasonable to use only a limited number of threads.
For example 10 threads should be enough even with 5 external disk.
If you know any reason to keep the number 60, please let me know...
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3053
## How was this patch tested?
It tested with a custom Freon test (HDDS-3052). The detailed test results available from here: https://hackmd.io/eYNzRqPSRyiMfWQ6mAYSSg
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-597072663
> did u check the pending request queue in the leader?
No I didn't. Why is it interesting?
> how many mappers were used for the test?
92 (see the link for this and all the other parameters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek edited a comment on issue #578: HDDS-3053.
Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek edited a comment on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-607122419
> I feel with more no of disks per datanode for write, we may need more threads depending upon the load.
I agree, with 1 hard disk I would use 3-4 threads. I proposed to use 10 which seems to be fine with ~5 disks and seems to be a good, balanced default.
Should we use 20 instead of 10? Is there any reason to have exactly 60?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-615109509
/ready
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant edited a comment on issue #578:
HDDS-3053. Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
bshashikant edited a comment on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-606416137
> Any more comments / suggestions?
If i remember correctly, number of threads is currently for all pipelines a datanode has which may span over multiple disks. This is not a per disk configuration.
I feel with more no of disks per datanode for write, we may need more threads depending upon the load.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-607825319
Other proposal to have a consensus:
What about checking the number of the disks (number of configured locations) and use the current configuration number as **per disk** value. (We can use `disk * 10` or `disk * 8` thread).
Would it be more safe?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-607830043
/pending @arp "Let's hold off committing this until we can discuss it further."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
arp7 commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-607358494
Let's hold off committing this until we can discuss it further. 20 threads will be insufficient if the number of disks is large.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-605937013
Any more comments / suggestions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant commented on issue #578: HDDS-3053.
Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
bshashikant commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-607157966
> > I feel with more no of disks per datanode for write, we may need more threads depending upon the load.
>
> I agree, with 1 hard disk I would use 3-4 threads. I proposed to use 10 which seems to be fine with ~5 disks and seems to be a good, balanced default.
>
> Should we use 20 instead of 10? Is there any reason to have exactly 60?
There is no logic to use exactly 60. I am ok to make to 20 for now.
We need to test with multiple disks and varying load to see how it really impacts the performance. We can even decide to make it a factor of number of data disks available per datanode but this can be done in follow up jira.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant commented on issue #578: HDDS-3053.
Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
bshashikant commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-606416137
> Any more comments / suggestions?
If i remember correctly, number of threads is currently for all pipelines a datanode has which may span over multiple disks. This is not a per disk configuration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on issue #578:
HDDS-3053. Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-600791255
I want to add one point here.
The same chunkExecutor threads are used for readStateMachineData. This will be used by the leader when sending append entry logs to followers. So, reducing this will make request also processing slow
```
CompletableFuture.supplyAsync(() -> {
try {
future.complete(
getCachedStateMachineData(entry.getIndex(), entry.getTerm(),
requestProto));
} catch (IOException e) {
metrics.incNumReadStateMachineFails();
future.completeExceptionally(e);
}
return future;
}, getChunkExecutor(requestProto.getWriteChunk()));
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant commented on issue #578: HDDS-3053.
Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
bshashikant commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-597591486
> > did u check the pending request queue in the leader?
>
> No I didn't. Why is it interesting?
>
> > how many mappers were used for the test?
>
> 92 (see the link for this and all the other parameters.
With lesser no of threads, if the no pending tasks keep on building on the chunk executor queue bcoz of lesser no of threads available for work, once it hits the max limit, the main thread itself starts executing the write call which degrades the performance as per the caller run policy set in the writeChunkExecutor as observed in our tests.
@elek , if you really think this seems to be causing issues in tests , we can change it to a lower value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-595668256
@bshashikant
I executed teragen and for me it was slightly faster:
https://hackmd.io/P3Hkf09fTLS78Ho4W82YbQ
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-607122419
> I feel with more no of disks per datanode for write, we may need more threads depending upon the load.
I agree, with 1 hard disk I would use 3-4 threads. I proposed to use 10 which seems to be fine with ~5 disks and seems to be a good, balanced default
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #578: HDDS-3053.
Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-600791255
I want to add one point here.
The same chunkExecutor threads are used for readStateMachineData. This will be used by leader when sending append entry logs to followers.
```
CompletableFuture.supplyAsync(() -> {
try {
future.complete(
getCachedStateMachineData(entry.getIndex(), entry.getTerm(),
requestProto));
} catch (IOException e) {
metrics.incNumReadStateMachineFails();
future.completeExceptionally(e);
}
return future;
}, getChunkExecutor(requestProto.getWriteChunk()));
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-603816449
> Can we make it a data driven decision? I looked at the linked document. I didn't understand the test methodology. E.g. when you say 60 thread / spinning disk, is that 60 chunk executor threads or 60 client threads?
There was no client thread in this test. The command was mentioned later in the doc:
```
60 thread / spinning disk
Command:
ozone freon --verbose cmdw -n1000000 -s 1024 -t60
```
This is using the ChunkManagerImpl fro 60 threads.
If you are interested about an e2e test, please check the teragen results (added in a comment).
> Can we make it a data driven decision?
100% agree. If you need any more date, just let me know. I think we should have 60 threads only if we have data as more threads mean more overhead. But if we have data to keep it, I am fine with it.
Based on the existing test, 10 threads should be enough as a default (IMHO)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-592537925
@mukul1987 @arp7 @bshashikant @lokeshj1703
Let me know if you have any opinion or any suggestion for a test before merging this....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-603814439
> So, reducing this will make request also processing slow
Good point, and I think they should be separated anyway. But it seems that even just 10 threads is enough to write **and** read. At least this is what the data shows for us (teragen + load test).
If you have any idea how to reproduce any related problem, just let me know and I will try it out. But I can't see any problem during the tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek edited a comment on issue #578: HDDS-3053.
Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek edited a comment on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-607772502
> Let's hold off committing this until we can discuss it further
Let's discuss it further ;-) (Marked the the PR with red flag to avoid unexpected merge without reading the thread)
I am happy with **any** numbers (even with 60), if you can explain how the number is chosen.
As you wrote earlier:
> Can we make it a data driven decision?
(yes)
My questions:
1. What is the expected number of the disks (for default configuration, something like an average)?
2. What is the number of the threads per disks which are required to have the best performance?
3. What is the performance penalty of using too much thread (optimizing to big pool of disks)
My answers:
1. 5 disk
2. 4 threads per disk (I am convinced to use 20 instead of 10 ;-) )
3. 10% percent slowness (measured data, see my first link: using 10 threads was ~10% faster than using 60 threads)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on issue #578: HDDS-3053.
Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-600776996
Agree with @elek chunkexecutor threads are mostly I/O bounded, adding excessive number of threads won't help much but could introduce expensive context switch overhead. I'm +1 on this change. We may later add some documents to set it properly based on the number of physical disks and number of CPU cores.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
arp7 commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-600788099
Can we make it a data driven decision? I looked at the linked document. I didn't understand the test methodology. E.g. when you say _60 thread / spinning disk_, is that 60 chunk executor threads or 60 client threads?
If we don't have full asynchronous IO - and I don't think we do - then we do need multiple threads per volume to keep more IOs in flight.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
arp7 commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-609987748
> What about checking the number of the disks (number of configured locations) and use the current configuration number as per disk value. (We can use disk * 10 or disk * 8 thread).
I like this idea, let's do this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant commented on issue #578: HDDS-3053.
Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
bshashikant commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-595739852
> @bshashikant
>
> I executed teragen and for me it was slightly faster:
>
> https://hackmd.io/P3Hkf09fTLS78Ho4W82YbQ
@marton, did u check the pending request queue in the leader? how many mappers were used for the test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-600757015
> @elek , if you really think this seems to be causing issues in tests , we can change it to a lower value.
I think it cause performance degradation if we use significant higher number of thread than the number of the cores. AFAIK the context switch / thread handling in Java is quite expensive (compared to the Go for example...). And it makes harder to debug the problems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-607772502
> Let's hold off committing this until we can discuss it further
Let's discuss it further ;-)
I am happy with **any** numbers (even with 60), if you can explain how the number is chosen.
As you wrote earlier:
> Can we make it a data driven decision?
(yes)
My questions:
1. What is the expected number of the disks (for default configuration, something like an average)?
2. What is the number of the threads per disks which are required to have the best performance?
3. What is the performance penalty of using too much thread (optimizing to big pool of disks)
My answers:
1. 5 disk
2. 4 threads per disk (I am convinced to use 20 instead of 10 ;-) )
3. 10% percent slowness (measured data, see my first link: using 10 threads was ~10% faster than using 60 threads)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-594649275
> but i would still prefer to do some similar tests before changing the default here
You mean a 100G teragen with and without this change? I can do it, if this is the suggestion...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #578: HDDS-3053. Decrease
the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
elek commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-594648760
> Also, it was observed that the queue used to hold the pending task in the write chunk executor used to fill up fast
Fix me if I am wrong, but if the disk is slow, the queue will be filled up very fast and it's independent from the number of the threads.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant commented on issue #578: HDDS-3053.
Decrease the number of the chunk writer threads
Posted by GitBox <gi...@apache.org>.
bshashikant commented on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-593769045
> @mukul1987 @arp7 @bshashikant @lokeshj1703
> Let me know if you have any opinion or any suggestion for a test before merging this....
@elek , whatever tests we conducted(using teragen for runs > 100GB), we never changed this limit. Also, it was observed that the queue used to hold the pending task in the write chunk executor used to fill up fast thereby indicating all the threads being busy actually doing write chunk. Though 60 may not be the ideal value, but i would still prefer to do some similar tests before changing the default here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org