You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by heary-cao <gi...@git.apache.org> on 2017/06/01 09:18:50 UTC
[GitHub] spark pull request #18174: Improve Serializerbuffersize and filebuffersize c...
GitHub user heary-cao opened a pull request:
https://github.com/apache/spark/pull/18174
Improve Serializerbuffersize and filebuffersize configurable
## What changes were proposed in this pull request?
This PR Improvement in three:
1.With `spark.shuffle.file.buffer` configure fileBufferSizeBytes of UnsafeExternalSorter .
2.With `spark.shuffle.sort.initialSerBufferSize` configure SerializerBufferSize of UnsafeShuffleWriter.
3.Remove outputBufferSizeInBytes and inputBufferSizeInBytes to initialize in mergeSpillsWithFileStream function.
## How was this patch tested?
The unit test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/heary-cao/spark buffersize
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18174.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18174
----
commit ed248072db3611609e502b3082085c5905a83868
Author: caoxuewen <ca...@zte.com.cn>
Date: 2017-06-01T09:16:07Z
Improve Serializerbuffersize and filebuffersize configurable
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve diskWriteBufferSize configura...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18174
There's no description of your test or what the numbers mean. I assume they're times. Why would a smaller buffer be faster?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]Improve diskWriteBufferSize co...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r123891763
--- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -123,6 +126,8 @@
this.inMemSorter = new ShuffleInMemorySorter(
this, initialSize, conf.getBoolean("spark.shuffle.sort.useRadixSort", true));
this.peakMemoryUsedBytes = getMemoryUsage();
+ this.diskWriteBufferSize =
+ conf.getInt("spark.shuffle.spill.diskWriteBufferSize", DISK_WRITE_BUFFER_SIZE);
--- End diff --
please move these new configs to `org.apache.spark.internal.config`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125548280
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,29 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle writer.")
--- End diff --
OK,
I have modify it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #79219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79219/testReport)** for PR 18174 at commit [`dc3997a`](https://github.com/apache/spark/commit/dc3997ab6d6546fac35d0b0f4044d351f7d03847).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #79219 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79219/testReport)** for PR 18174 at commit [`dc3997a`](https://github.com/apache/spark/commit/dc3997ab6d6546fac35d0b0f4044d351f7d03847).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #78967 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78967/testReport)** for PR 18174 at commit [`62ccd4e`](https://github.com/apache/spark/commit/62ccd4e401c576e1b281f08e240c6c8a70e91994).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/18174
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #79195 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79195/testReport)** for PR 18174 at commit [`dc3997a`](https://github.com/apache/spark/commit/dc3997ab6d6546fac35d0b0f4044d351f7d03847).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79219/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79123/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve Serializerbuffersize and file...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/18174
Performance tuning, we need to set up the size.
Such as:
Filebuffersize is set to `64K` or `16K`
Serializerbuffersize is set to `512K` or `2M`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78967/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve diskWriteBufferSize configura...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/18174
@srowen
thanks for review it.
In our performance tuning, find the row of record the size of more than 2M. so need to initialSerBufferSize configurable. but Change `initialSerBufferSize ` is not good for performance tuning.
However, change spill `diskWriteBufferSize ` is good for performance tuning.
So I did a little experiment, change the size of the diskWriteBufferSize to test.
set diskWriteBufferSize to 1M, 512K, 256k, 256K, 128K, 64K,etc.
diskWriteBufferSize `1M 512K 256K 128K 64K 32K 16K 8K 4K`
RecordSize:2.5M `742 722 694 686 667 668 671 669 683`
RecordSize:1M `294 293 292 287 283 285 281 279 285`
In order to eliminate the interference of other factors, these results are tested take the average of 10 times.
please review code again.
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125468331
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,26 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle writer.")
+ .bytesConf(ByteUnit.KiB)
+ .checkValues(Set(4, 8, 16, 32, 64, 128, 256, 512, 1024))
--- End diff --
we should use `checkValue(v => v >= 0 && v <= Int.MaxValue / 1024, ...)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125204174
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,20 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .intConf
+ .createWithDefault(32 * 1024)
+
+ private[spark] val SHUFFLE_UNSAFE_FILE_OUTPUT_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.unsafe.file.output.buffer")
--- End diff --
OK,
thank you for review it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve Serializerbuffersize and file...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18174
These are already configurable, right? they're intentionally connected to other configs. Do you have a case where this is significantly suboptimal?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125548173
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,29 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle writer.")
+ .bytesConf(ByteUnit.KiB)
+ .checkValue(v => v > 0 && v <= Int.MaxValue / 1024,
+ s"The file buffer size must be greater than 0 to ${Int.MaxValue / 1024}.")
--- End diff --
thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18174
thanks, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/18174
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve diskWriteBufferSize configura...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/18174
@srowen
yes, you're right, It's time, and their unit is MS.
the numbers Is the average time of 10 times running` forceSorterToSpill`.
I assume big buffer copies time consuming longer than small buffer. Although the small buffer has been copied many times. or local file systems write big buffer that time consuming longer than small buffer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125546508
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,29 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle writer.")
+ .bytesConf(ByteUnit.KiB)
+ .checkValue(v => v > 0 && v <= Int.MaxValue / 1024,
+ s"The file buffer size must be greater than 0 to ${Int.MaxValue / 1024}.")
+ .createWithDefaultString("32k")
+
+ private[spark] val SHUFFLE_UNSAFE_FILE_OUTPUT_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.unsafe.file.output.buffer")
+ .doc("The file system for this buffer size after each partition " +
+ "is written in unsafe shuffle writer.")
+ .bytesConf(ByteUnit.KiB)
+ .checkValue(v => v > 0 && v <= Int.MaxValue / 1024,
+ s"The buffer size must be greater than 0 to ${Int.MaxValue / 1024}.")
+ .createWithDefaultString("32k")
+
+ private[spark] val SHUFFLE_DISK_WRITE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.spill.diskWriteBufferSize")
+ .doc("The buffer size to use when writing the sorted records to an on-disk file.")
+ .intConf
--- End diff --
use `bytesConf` please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125547523
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,29 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle writer.")
+ .bytesConf(ByteUnit.KiB)
+ .checkValue(v => v > 0 && v <= Int.MaxValue / 1024,
+ s"The file buffer size must be greater than 0 to ${Int.MaxValue / 1024}.")
+ .createWithDefaultString("32k")
+
+ private[spark] val SHUFFLE_UNSAFE_FILE_OUTPUT_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.unsafe.file.output.buffer")
+ .doc("The file system for this buffer size after each partition " +
+ "is written in unsafe shuffle writer.")
+ .bytesConf(ByteUnit.KiB)
+ .checkValue(v => v > 0 && v <= Int.MaxValue / 1024,
+ s"The buffer size must be greater than 0 to ${Int.MaxValue / 1024}.")
+ .createWithDefaultString("32k")
+
+ private[spark] val SHUFFLE_DISK_WRITE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.spill.diskWriteBufferSize")
+ .doc("The buffer size to use when writing the sorted records to an on-disk file.")
+ .intConf
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125106803
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,20 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
--- End diff --
Also please add `.checkValue` for all these three
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r124214950
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java ---
@@ -61,6 +65,9 @@ public UnsafeSorterSpillWriter(
this.file = spilledFileInfo._2();
this.blockId = spilledFileInfo._1();
this.numRecordsToWrite = numRecordsToWrite;
+ this.diskWriteBufferSize =
--- End diff --
I don't have special interest over this, but at least we should keep the same behavior for both `ShuffleExternalSorter` class and `UnsafeSorterSpillWriter` class, otherwise we are introducing unnecessary complexity. Could you please update this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r123924134
--- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -123,6 +126,8 @@
this.inMemSorter = new ShuffleInMemorySorter(
this, initialSize, conf.getBoolean("spark.shuffle.sort.useRadixSort", true));
this.peakMemoryUsedBytes = getMemoryUsage();
+ this.diskWriteBufferSize =
+ conf.getInt("spark.shuffle.spill.diskWriteBufferSize", DISK_WRITE_BUFFER_SIZE);
--- End diff --
@cloud-fan
thanks for review it.
Thank you for your advice. I tried to fix it.
However, org.apache.spark.internal.config cannot be imported in the ShuffleExternalSorter.java class
This modification will change the org.apache.spark.internal.config and affect other code changes,
so I suggest using the PR to modify it. Do you agree?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r124041199
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java ---
@@ -61,6 +65,9 @@ public UnsafeSorterSpillWriter(
this.file = spilledFileInfo._2();
this.blockId = spilledFileInfo._1();
this.numRecordsToWrite = numRecordsToWrite;
+ this.diskWriteBufferSize =
--- End diff --
Why are we doing this in a different way?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #79074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79074/testReport)** for PR 18174 at commit [`f6d895c`](https://github.com/apache/spark/commit/f6d895c944c514b7e51db19388ef00016671dddb).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79195/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve diskWriteBufferSize configura...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/18174
@gatorsmile
Can you review this code.
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/18174
@cloud-fan
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18174
LGTM, pending test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125106509
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,20 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
--- End diff --
The same here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #78981 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78981/testReport)** for PR 18174 at commit [`62ccd4e`](https://github.com/apache/spark/commit/62ccd4e401c576e1b281f08e240c6c8a70e91994).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: Improve Serializerbuffersize and filebuffersize configur...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve Serializerbuffersize configur...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18174
I don't think that addresses my question? when would you set this separately?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78981/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r124239231
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java ---
@@ -61,6 +65,9 @@ public UnsafeSorterSpillWriter(
this.file = spilledFileInfo._2();
this.blockId = spilledFileInfo._1();
this.numRecordsToWrite = numRecordsToWrite;
+ this.diskWriteBufferSize =
--- End diff --
ok, I have update it.
please review it again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve diskWriteBufferSize configura...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18174
It would be also great to update the explicit performance tuning result in PR description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]Improve diskWriteBufferSize co...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r123893797
--- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -82,6 +82,9 @@
/** The buffer size to use when writing spills using DiskBlockObjectWriter */
private final int fileBufferSizeBytes;
+ /** The buffer size to use when writes the sorted records to an on-disk file */
--- End diff --
nit: 'writes' -> 'writing'
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve Serializerbuffersize configur...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/18174
@srowen
sorry, the description of pr would be wrong. so I modify to this PR.
`spark.shuffle.file.buffer` is already configurable, in addition to other PR#18182 to describe it.
`spark.shuffle.sort.initialSerBufferSize`, It is a new configuration to configure SerializerBufferSize.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125421705
--- Diff: core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java ---
@@ -360,12 +368,10 @@ void forceSorterToSpill() throws IOException {
final OutputStream bos = new BufferedOutputStream(
new FileOutputStream(outputFile),
- (int) sparkConf.getSizeAsKb("spark.shuffle.unsafe.file.output.buffer", "32k") * 1024);
+ outputBufferSizeInBytes);
--- End diff --
thanks for explain it.
The explanation is very clear.
I study it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125546440
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,29 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle writer.")
--- End diff --
can we copy the document from http://spark.apache.org/docs/latest/configuration.html ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #79123 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79123/testReport)** for PR 18174 at commit [`3efc743`](https://github.com/apache/spark/commit/3efc7433802155c957e78d23abf4847cde8e0d07).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve diskWriteBufferSize configura...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18174
the change looks reasonable, but you really need to update your PR description. You are not improving some config, but introducing a new config which is hard coded before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by manku-timma <gi...@git.apache.org>.
Github user manku-timma commented on the issue:
https://github.com/apache/spark/pull/18174
Just to understand what is happening.
1. Shuffle records are written to a serialisation buffer (1M) after serialisation
2. The serialised buffer is written to in-memory-sorter’s buffer
3. once in-memory sorter’s buffer is full, the data is copied to sorter’s disk buffer (1M)
4. the sorter’s disk buffer is written out to a buffered output stream (buffer = 32k)
I am guessing reducing the sorter’s disk buffer (in step 3) is helping because it triggers fewer writes at the step 4.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125106486
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,20 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .intConf
+ .createWithDefault(32 * 1024)
+
+ private[spark] val SHUFFLE_UNSAFE_FILE_OUTPUT_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.unsafe.file.output.buffer")
--- End diff --
No description?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r123923859
--- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -82,6 +82,9 @@
/** The buffer size to use when writing spills using DiskBlockObjectWriter */
private final int fileBufferSizeBytes;
+ /** The buffer size to use when writes the sorted records to an on-disk file */
--- End diff --
@jiangxb1987
thanks for review it,
The UnsafeSorterSpillWriter changes were updated.
please review it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125204165
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,20 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
--- End diff --
OK,
thank you for review it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #78981 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78981/testReport)** for PR 18174 at commit [`62ccd4e`](https://github.com/apache/spark/commit/62ccd4e401c576e1b281f08e240c6c8a70e91994).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by manku-timma <gi...@git.apache.org>.
Github user manku-timma commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125386319
--- Diff: core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java ---
@@ -360,12 +368,10 @@ void forceSorterToSpill() throws IOException {
final OutputStream bos = new BufferedOutputStream(
new FileOutputStream(outputFile),
- (int) sparkConf.getSizeAsKb("spark.shuffle.unsafe.file.output.buffer", "32k") * 1024);
+ outputBufferSizeInBytes);
--- End diff --
Just to understand what is happening.
1. Shuffle records are written to a serialisation buffer (1M) after serialisation
2. The serialised buffer is written to in-memory-sorter’s buffer
3. once in-memory sorter’s buffer is full, the data is copied to sorter’s disk buffer (1M)
4. the sorter’s disk buffer is written out to a buffered output stream (buffer = 32k)
I am guessing reducing the sorter’s disk buffer (in step 3) is helping because it triggers fewer writes/allocations in a single call at step 4 (and allowing more parallelism in writing back to disk and copying of data).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125468128
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,26 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle writer.")
+ .bytesConf(ByteUnit.KiB)
+ .checkValues(Set(4, 8, 16, 32, 64, 128, 256, 512, 1024))
--- End diff --
do we have this requirement before? I think the only requirement is, the value should be `>= 0` and `<= Int.MaxValue / 1024`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r124182713
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java ---
@@ -61,6 +65,9 @@ public UnsafeSorterSpillWriter(
this.file = spilledFileInfo._2();
this.blockId = spilledFileInfo._1();
this.numRecordsToWrite = numRecordsToWrite;
+ this.diskWriteBufferSize =
--- End diff --
Do you want to initialize as a member variable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79074/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]Improve diskWriteBufferSize configura...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18174
cc @jiangxb1987 @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125546465
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,29 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle writer.")
+ .bytesConf(ByteUnit.KiB)
+ .checkValue(v => v > 0 && v <= Int.MaxValue / 1024,
+ s"The file buffer size must be greater than 0 to ${Int.MaxValue / 1024}.")
--- End diff --
`must be greater than 0 and less than ${Int.MaxValue / 1024}`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]Improve diskWriteBufferSize co...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r123891837
--- Diff: core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java ---
@@ -140,6 +143,9 @@ public UnsafeShuffleWriter(
this.transferToEnabled = sparkConf.getBoolean("spark.file.transferTo", true);
this.initialSortBufferSize = sparkConf.getInt("spark.shuffle.sort.initialBufferSize",
DEFAULT_INITIAL_SORT_BUFFER_SIZE);
+ this.inputBufferSizeInBytes = (int) sparkConf.getSizeAsKb("spark.shuffle.file.buffer", "32k") * 1024;
+ this.outputBufferSizeInBytes = (int) sparkConf.getSizeAsKb("spark.shuffle.unsafe.file.output.buffer",
--- End diff --
as you are modifying it, can you also move these 2 configs into `org.apache.spark.internal.config`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125545875
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,26 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle writer.")
+ .bytesConf(ByteUnit.KiB)
+ .checkValues(Set(4, 8, 16, 32, 64, 128, 256, 512, 1024))
--- End diff --
I have modify it.
For buffer size, I think 0 is meaningless,
so I have modify for
`checkValue(v => v > 0 && v <= Int.MaxValue / 1024, ...)`
do you agree it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #79195 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79195/testReport)** for PR 18174 at commit [`dc3997a`](https://github.com/apache/spark/commit/dc3997ab6d6546fac35d0b0f4044d351f7d03847).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by heary-cao <gi...@git.apache.org>.
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r124265367
--- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -123,6 +126,8 @@
this.inMemSorter = new ShuffleInMemorySorter(
this, initialSize, conf.getBoolean("spark.shuffle.sort.useRadixSort", true));
this.peakMemoryUsedBytes = getMemoryUsage();
+ this.diskWriteBufferSize =
+ conf.getInt("spark.shuffle.spill.diskWriteBufferSize", DISK_WRITE_BUFFER_SIZE);
--- End diff --
Thank you for your guidance.
please review it again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #78967 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78967/testReport)** for PR 18174 at commit [`62ccd4e`](https://github.com/apache/spark/commit/62ccd4e401c576e1b281f08e240c6c8a70e91994).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r123932606
--- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -123,6 +126,8 @@
this.inMemSorter = new ShuffleInMemorySorter(
this, initialSize, conf.getBoolean("spark.shuffle.sort.useRadixSort", true));
this.peakMemoryUsedBytes = getMemoryUsage();
+ this.diskWriteBufferSize =
+ conf.getInt("spark.shuffle.spill.diskWriteBufferSize", DISK_WRITE_BUFFER_SIZE);
--- End diff --
oh, in java, you need to
```
import org.apache.spark.internal.config.package$;
...
package$.MODULE$.XXX()
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #79074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79074/testReport)** for PR 18174 at commit [`f6d895c`](https://github.com/apache/spark/commit/f6d895c944c514b7e51db19388ef00016671dddb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18174
**[Test build #79123 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79123/testReport)** for PR 18174 at commit [`3efc743`](https://github.com/apache/spark/commit/3efc7433802155c957e78d23abf4847cde8e0d07).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18174
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18174
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org