You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "lwllvyb (via GitHub)" <gi...@apache.org> on 2024/10/21 12:26:18 UTC
[PR] [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler [incubator-uniffle]
lwllvyb opened a new pull request, #2216:
URL: https://github.com/apache/incubator-uniffle/pull/2216
### What changes were proposed in this pull request?
Support config buffer size for HadoopShuffleWriteHandler
### Why are the changes needed?
Fix: #2209
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Locally test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #2216:
URL: https://github.com/apache/incubator-uniffle/pull/2216
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #2216:
URL: https://github.com/apache/incubator-uniffle/pull/2216#issuecomment-2428170926
> > Should this config option be same with the local? Maybe they should have different default value for better performance.
>
> Thanks for your suggestion. I am very agree with you that they will get better performance with different values for different storages. I will add different configs (`rss.storage.localfile.write.dataBufferSize`, `rss.storage.hadoop.write.indexBufferSize`, `rss.storage.hadoop.write.dataBufferSize`, `rss.storage.hadoop.write.indexBufferSize`) for LOCALFILE and HADOOP.
>
> If we can set the default values to be the same (e.g. 8K ?), we can config different values to cope with different real cluster scenarios.
>
> Do you have any better suggestions on default values? Thanks again for taking the time to review.
Ok for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler [incubator-uniffle]
Posted by "lwllvyb (via GitHub)" <gi...@apache.org>.
lwllvyb commented on PR #2216:
URL: https://github.com/apache/incubator-uniffle/pull/2216#issuecomment-2428401120
> > Should this config option be same with the local? Maybe they should have different default value for better performance.
>
> Thanks for your suggestion. I am very agree with you that they will get better performance with different values for different storages. I will add different configs (`rss.storage.localfile.write.dataBufferSize`, `rss.storage.hadoop.write.indexBufferSize`, `rss.storage.hadoop.write.dataBufferSize`, `rss.storage.hadoop.write.indexBufferSize`) for LOCALFILE and HADOOP.
>
> If we can set the default values to be the same (e.g. 8K ?), we can config different values to cope with different real cluster scenarios.
>
> Do you have any better suggestions on default values? Thanks again for taking the time to review.
To be consistent with ```rss.storage.type```'s values, I update the configs from `rss.storage.hadoop.write.dataBufferSize`, `rss.storage.hadoop.write.indexBufferSize` to `rss.storage.hdfs.write.dataBufferSize`, `rss.storage.hdfs.write.indexBufferSize`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2216:
URL: https://github.com/apache/incubator-uniffle/pull/2216#issuecomment-2426625837
## Test Results
2 926 files +22 2 926 suites +22 6h 22m 41s :stopwatch: + 21m 30s
1 049 tests ± 0 1 047 :white_check_mark: + 1 2 :zzz: ±0 0 :x: - 1
13 045 runs +51 13 015 :white_check_mark: +52 30 :zzz: ±0 0 :x: - 1
Results for commit 995fd916. ± Comparison against base commit 3a35b0f5.
[test-results]:data:application/gzip;base64,H4sIALhRFmcC/02MSw6DMAwFr4Ky7sIOxMW9TAUhkaLyqUKyQr17wy+wnLHfLMK63sziVUiW9CjEHF24cxd9E9w0rkYyYVLpHtYPhIpPfM9R6909L/dx33WXhW1cnwRkYbyf/GF8HLdqCZU6MFdLwMvt1fLc3Ksb36N6GgYXEghmZTtGksrUWLMBZpAtoSECQkJLte6wteL3B0axj3QUAQAA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler [incubator-uniffle]
Posted by "lwllvyb (via GitHub)" <gi...@apache.org>.
lwllvyb commented on code in PR #2216:
URL: https://github.com/apache/incubator-uniffle/pull/2216#discussion_r1810401133
##########
common/src/main/java/org/apache/uniffle/common/config/RssBaseConf.java:
##########
@@ -294,17 +294,29 @@ public class RssBaseConf extends RssConf {
+ " first combining the username and the password with a colon (uniffle:uniffle123)"
+ ", and then by encoding the resulting string in base64 (dW5pZmZsZTp1bmlmZmxlMTIz).");
- public static final ConfigOption<String> RSS_STORAGE_WRITE_DATA_BUFFER_SIZE =
- ConfigOptions.key("rss.storage.write.dataBufferSize")
+ public static final ConfigOption<String> RSS_STORAGE_LOCALFILE_WRITE_DATA_BUFFER_SIZE =
+ ConfigOptions.key("rss.storage.localfile.write.dataBufferSize")
.stringType()
.defaultValue("8k")
- .withDescription("The buffer size to cache the write data content.");
+ .withDescription("The buffer size to cache the write data content for LOCALFILE.");
- public static final ConfigOption<String> RSS_STORAGE_WRITE_INDEX_BUFFER_SIZE =
- ConfigOptions.key("rss.storage.write.indexBufferSize")
+ public static final ConfigOption<String> RSS_STORAGE_LOCALFILE_WRITE_INDEX_BUFFER_SIZE =
+ ConfigOptions.key("rss.storage.localfile.write.indexBufferSize")
.stringType()
.defaultValue("8k")
- .withDescription("The buffer size to cache the write index content.");
+ .withDescription("The buffer size to cache the write index content for LOCALFILE.");
+
+ public static final ConfigOption<String> RSS_STORAGE_HDFS_WRITE_DATA_BUFFER_SIZE =
+ ConfigOptions.key("rss.storage.hdfs.write.dataBufferSize")
+ .stringType()
+ .defaultValue("8k")
Review Comment:
<img width="986" alt="image" src="https://github.com/user-attachments/assets/05f43826-c21f-4366-9ecb-7aec7177bd6f">
<img width="761" alt="image" src="https://github.com/user-attachments/assets/9fd916e1-1a0a-4cc0-a63f-fa33eca9d7c6">
I have checked the source code of hadoop. The `io.file.buffer.size` default value is 4096.
So i think `8k` as the default value is ok for HDFS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler [incubator-uniffle]
Posted by "lwllvyb (via GitHub)" <gi...@apache.org>.
lwllvyb commented on PR #2216:
URL: https://github.com/apache/incubator-uniffle/pull/2216#issuecomment-2428143604
> Should this config option be same with the local? Maybe they should have different default value for better performance.
Thanks for your suggestion. I am very agree with you that they will get better performance with different values for different storages.
I will add different configs (```rss.storage.localfile.write.dataBufferSize```, ```rss.storage.hadoop.write.indexBufferSize```, ```rss.storage.hadoop.write.dataBufferSize```, ```rss.storage.hadoop.write.indexBufferSize```) for LOCALFILE and HADOOP.
If we can set the default values to be the same (e.g. 8K ?), we can config different values to cope with different real cluster scenarios.
Do you have any better suggestions on default values?
Thanks again for taking the time to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #2216:
URL: https://github.com/apache/incubator-uniffle/pull/2216#discussion_r1810532934
##########
common/src/main/java/org/apache/uniffle/common/config/RssBaseConf.java:
##########
@@ -294,17 +294,29 @@ public class RssBaseConf extends RssConf {
+ " first combining the username and the password with a colon (uniffle:uniffle123)"
+ ", and then by encoding the resulting string in base64 (dW5pZmZsZTp1bmlmZmxlMTIz).");
- public static final ConfigOption<String> RSS_STORAGE_WRITE_DATA_BUFFER_SIZE =
- ConfigOptions.key("rss.storage.write.dataBufferSize")
+ public static final ConfigOption<String> RSS_STORAGE_LOCALFILE_WRITE_DATA_BUFFER_SIZE =
+ ConfigOptions.key("rss.storage.localfile.write.dataBufferSize")
.stringType()
.defaultValue("8k")
- .withDescription("The buffer size to cache the write data content.");
+ .withDescription("The buffer size to cache the write data content for LOCALFILE.");
- public static final ConfigOption<String> RSS_STORAGE_WRITE_INDEX_BUFFER_SIZE =
- ConfigOptions.key("rss.storage.write.indexBufferSize")
+ public static final ConfigOption<String> RSS_STORAGE_LOCALFILE_WRITE_INDEX_BUFFER_SIZE =
+ ConfigOptions.key("rss.storage.localfile.write.indexBufferSize")
.stringType()
.defaultValue("8k")
- .withDescription("The buffer size to cache the write index content.");
+ .withDescription("The buffer size to cache the write index content for LOCALFILE.");
+
+ public static final ConfigOption<String> RSS_STORAGE_HDFS_WRITE_DATA_BUFFER_SIZE =
+ ConfigOptions.key("rss.storage.hdfs.write.dataBufferSize")
+ .stringType()
+ .defaultValue("8k")
Review Comment:
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler [incubator-uniffle]
Posted by "lwllvyb (via GitHub)" <gi...@apache.org>.
lwllvyb closed pull request #2216: [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler
URL: https://github.com/apache/incubator-uniffle/pull/2216
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2209] [FOLLOWUP]feat(server) Support config buffer size for HadoopShuffleWriteHandler [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #2216:
URL: https://github.com/apache/incubator-uniffle/pull/2216#discussion_r1810151726
##########
common/src/main/java/org/apache/uniffle/common/config/RssBaseConf.java:
##########
@@ -294,17 +294,29 @@ public class RssBaseConf extends RssConf {
+ " first combining the username and the password with a colon (uniffle:uniffle123)"
+ ", and then by encoding the resulting string in base64 (dW5pZmZsZTp1bmlmZmxlMTIz).");
- public static final ConfigOption<String> RSS_STORAGE_WRITE_DATA_BUFFER_SIZE =
- ConfigOptions.key("rss.storage.write.dataBufferSize")
+ public static final ConfigOption<String> RSS_STORAGE_LOCALFILE_WRITE_DATA_BUFFER_SIZE =
+ ConfigOptions.key("rss.storage.localfile.write.dataBufferSize")
.stringType()
.defaultValue("8k")
- .withDescription("The buffer size to cache the write data content.");
+ .withDescription("The buffer size to cache the write data content for LOCALFILE.");
- public static final ConfigOption<String> RSS_STORAGE_WRITE_INDEX_BUFFER_SIZE =
- ConfigOptions.key("rss.storage.write.indexBufferSize")
+ public static final ConfigOption<String> RSS_STORAGE_LOCALFILE_WRITE_INDEX_BUFFER_SIZE =
+ ConfigOptions.key("rss.storage.localfile.write.indexBufferSize")
.stringType()
.defaultValue("8k")
- .withDescription("The buffer size to cache the write index content.");
+ .withDescription("The buffer size to cache the write index content for LOCALFILE.");
+
+ public static final ConfigOption<String> RSS_STORAGE_HDFS_WRITE_DATA_BUFFER_SIZE =
+ ConfigOptions.key("rss.storage.hdfs.write.dataBufferSize")
+ .stringType()
+ .defaultValue("8k")
Review Comment:
Is `8k` ok for HDFS? Maybe we would better use Hadoop default config option before we have a benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org