You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JoshRosen <gi...@git.apache.org> on 2015/04/22 01:20:33 UTC
[GitHub] spark pull request: [WIP] [SPARK-7041] Avoid writing empty files i...
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/5622
[WIP] [SPARK-7041] Avoid writing empty files in ExternalSorter
In ExternalSorter, we may end up opening disk writers files for empty partitions; this occurs because we manually call `open()` after creating the writer, causing serialization and compression input streams to be created; these streams may write headers to the output stream, resulting in non-zero-length files being created for partitions that contain no records. This is unnecessary, though, since the disk object writer will automatically open itself when the first write is performed. Removing this eager open() call and rewriting the consumers to cope with the non-existence of empty files results in a large performance benefit for certain sparse workloads when using sort-based shuffle.
This patch is marked as [WIP] because it incorporates code from another one of my PRs (#5606). Submitting now so Jenkins tests it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark file-handle-optimizations
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5622.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5622
----
commit aeb680e986474e9e16bb61b8a3e165d1be1c4c2e
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-04-21T07:22:55Z
[SPARK-3386] Reuse SerializerInstance in shuffle code paths
commit 64f83982439db4f2ac4e6814dcc6a6ecdea82074
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-04-21T08:07:34Z
Use ThreadLocal for serializer instance in CoarseGrainedExecutorBackend
commit f661ce7f26cf1a6131bebaaa051ed25542b774a2
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-04-21T17:14:39Z
Remove thread local; add comment instead
commit a21a5836b1b1b536419eb39368e1a91f37e58eb6
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-04-21T18:55:11Z
Avoid IO operations on empty files in BlockObjectWriter.
commit f81918bf23ee1e0ce72878c5f818288c039afd24
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-04-21T19:10:46Z
Do not create empty files at all.
commit b650ab2703fdde17f8d463dbc1257e70f3999fc8
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-04-21T20:30:00Z
Reduce scope of FileOutputStream in ExternalSorter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-94974567
[Test build #30710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30710/consoleFull) for PR 5622 at commit [`3c9c944`](https://github.com/apache/spark/commit/3c9c9447d4d4e8ddeb036167390073e3b67fb621).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-94981578
[Test build #30709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30709/consoleFull) for PR 5622 at commit [`b650ab2`](https://github.com/apache/spark/commit/b650ab2703fdde17f8d463dbc1257e70f3999fc8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Bypa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-155807725
**[Test build #45642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45642/consoleFull)** for PR 5622 at commit [`966cc15`](https://github.com/apache/spark/commit/966cc15dfeb43e3431677ec7bbef9ee027ef860c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5622#discussion_r28836513
--- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala ---
@@ -732,19 +730,24 @@ private[spark] class ExternalSorter[K, V, C](
// this simple we spill out the current in-memory collection so that everything is in files.
spillToPartitionFiles(if (aggregator.isDefined) map else buffer)
partitionWriters.foreach(_.commitAndClose())
- val out = new FileOutputStream(outputFile, true)
val writeStartTime = System.nanoTime
util.Utils.tryWithSafeFinally {
for (i <- 0 until numPartitions) {
- val in = new FileInputStream(partitionWriters(i).fileSegment().file)
- util.Utils.tryWithSafeFinally {
- lengths(i) = org.apache.spark.util.Utils.copyStream(in, out, false, transferToEnabled)
- } {
- in.close()
+ val file = partitionWriters(i).fileSegment().file
+ if (!file.exists()) {
+ lengths(i) = 0
+ } else {
+ val out = new FileOutputStream(outputFile, true)
--- End diff --
Yep, good catch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-94983869
[Test build #30710 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30710/consoleFull) for PR 5622 at commit [`3c9c944`](https://github.com/apache/spark/commit/3c9c9447d4d4e8ddeb036167390073e3b67fb621).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Explode(child: Expression)`
* `protected[spark] abstract class NativeType extends DataType `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-95031060
[Test build #30722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30722/consoleFull) for PR 5622 at commit [`ad13424`](https://github.com/apache/spark/commit/ad13424c044d7c1c71bb802a5fb08fdf73effc4d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Explode(child: Expression)`
* `protected[spark] abstract class NativeType extends DataType `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-95008798
[Test build #30722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30722/consoleFull) for PR 5622 at commit [`ad13424`](https://github.com/apache/spark/commit/ad13424c044d7c1c71bb802a5fb08fdf73effc4d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Bypa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-155844652
**[Test build #45642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45642/consoleFull)** for PR 5622 at commit [`966cc15`](https://github.com/apache/spark/commit/966cc15dfeb43e3431677ec7bbef9ee027ef860c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5622#discussion_r28989398
--- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala ---
@@ -736,11 +734,16 @@ private[spark] class ExternalSorter[K, V, C](
val writeStartTime = System.nanoTime
util.Utils.tryWithSafeFinally {
for (i <- 0 until numPartitions) {
- val in = new FileInputStream(partitionWriters(i).fileSegment().file)
- util.Utils.tryWithSafeFinally {
- lengths(i) = org.apache.spark.util.Utils.copyStream(in, out, false, transferToEnabled)
- } {
- in.close()
+ val file = partitionWriters(i).fileSegment().file
+ if (!file.exists()) {
+ lengths(i) = 0
+ } else {
+ val in = new FileInputStream(file)
+ util.Utils.tryWithSafeFinally {
--- End diff --
This was a carryover from the old code, but I can fix this up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Bypa...
Posted by JoshRosen <gi...@git.apache.org>.
GitHub user JoshRosen reopened a pull request:
https://github.com/apache/spark/pull/5622
[SPARK-7041] Avoid writing empty files in BypassMergeSortShuffleWriter
In BypassMergeSortShuffleWriter, we may end up opening disk writers files for empty partitions; this occurs because we manually call `open()` after creating the writer, causing serialization and compression input streams to be created; these streams may write headers to the output stream, resulting in non-zero-length files being created for partitions that contain no records. This is unnecessary, though, since the disk object writer will automatically open itself when the first write is performed. Removing this eager open() call and rewriting the consumers to cope with the non-existence of empty files results in a large performance benefit for certain sparse workloads when using sort-based shuffle. This has an impact for small-scale Spark SQL jobs in unit tests and `spark-shell`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark file-handle-optimizations
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5622.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5622
----
commit 00bcf8a893c021fa4a949c5ac077a34881870ace
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-04-21T18:55:11Z
Avoid IO operations on empty files in BlockObjectWriter.
commit 8fd89b47efbef6325e0bc45bad0b74bf8ead4a6d
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-04-21T19:10:46Z
Do not create empty files at all.
commit 0db87c341686e7b24e760583bcc9fe9054d3095a
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-04-21T20:30:00Z
Reduce scope of FileOutputStream in ExternalSorter
commit 7e2340d05721d6374e78069baa5870e87cd0cfb1
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-04-22T00:45:45Z
Revert "Reduce scope of FileOutputStream in ExternalSorter"
This reverts commit 3c9c9447d4d4e8ddeb036167390073e3b67fb621.
commit 54cd5ceb025f635552a14e6241d43e9858fb095d
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-06-05T21:38:38Z
Merge remote-tracking branch 'origin/master' into file-handle-optimizations
commit 5c777cf40ee1f70092639a8abfe8b9598d6d3636
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-06-05T21:35:53Z
Rework SPARK-7041 for BypassMergeSort split
commit c7caa5c6b54c86895a2f57ba448b1dd626ce5cf4
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-06-06T05:21:20Z
Merge remote-tracking branch 'origin/master' into file-handle-optimizations
commit aaa51bf58f286f0c1dbb0a38afb514e7a38b1183
Author: Josh Rosen <jo...@databricks.com>
Date: 2015-06-09T17:50:35Z
Actually avoid calling open() :)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5622#discussion_r28834809
--- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala ---
@@ -732,19 +730,24 @@ private[spark] class ExternalSorter[K, V, C](
// this simple we spill out the current in-memory collection so that everything is in files.
spillToPartitionFiles(if (aggregator.isDefined) map else buffer)
partitionWriters.foreach(_.commitAndClose())
- val out = new FileOutputStream(outputFile, true)
val writeStartTime = System.nanoTime
util.Utils.tryWithSafeFinally {
for (i <- 0 until numPartitions) {
- val in = new FileInputStream(partitionWriters(i).fileSegment().file)
- util.Utils.tryWithSafeFinally {
- lengths(i) = org.apache.spark.util.Utils.copyStream(in, out, false, transferToEnabled)
- } {
- in.close()
+ val file = partitionWriters(i).fileSegment().file
+ if (!file.exists()) {
--- End diff --
I guess I'm implicitly assuming that the only reason that this file would not exist is because no values were written to it. If we can think of cases where this assumption might be violated, such as someone deleting the file from underneath us, then maybe we should use some other data structure to track whether we should expect a file or not.
We could try just `touch`ing the file to ensure that it always exists, but I found that this had a measurable performance impact compared to not generating the file in the first place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5622#discussion_r28834707
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -189,13 +189,12 @@ private[spark] class DiskBlockObjectWriter(
objOut.flush()
bs.flush()
close()
- }
-
- val truncateStream = new FileOutputStream(file, true)
- try {
- truncateStream.getChannel.truncate(initialPosition)
- } finally {
- truncateStream.close()
+ val truncateStream = new FileOutputStream(file, true)
--- End diff --
I moved this into the `if (initialized)` block because there should be no need to truncate unless we've written to the file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-7041] Avoid writing empty files i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-94971843
[Test build #30709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30709/consoleFull) for PR 5622 at commit [`b650ab2`](https://github.com/apache/spark/commit/b650ab2703fdde17f8d463dbc1257e70f3999fc8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Bypa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-155844838
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-95031073
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30722/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5622#discussion_r28836457
--- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala ---
@@ -732,19 +730,24 @@ private[spark] class ExternalSorter[K, V, C](
// this simple we spill out the current in-memory collection so that everything is in files.
spillToPartitionFiles(if (aggregator.isDefined) map else buffer)
partitionWriters.foreach(_.commitAndClose())
- val out = new FileOutputStream(outputFile, true)
val writeStartTime = System.nanoTime
util.Utils.tryWithSafeFinally {
for (i <- 0 until numPartitions) {
- val in = new FileInputStream(partitionWriters(i).fileSegment().file)
- util.Utils.tryWithSafeFinally {
- lengths(i) = org.apache.spark.util.Utils.copyStream(in, out, false, transferToEnabled)
- } {
- in.close()
+ val file = partitionWriters(i).fileSegment().file
+ if (!file.exists()) {
+ lengths(i) = 0
+ } else {
+ val out = new FileOutputStream(outputFile, true)
--- End diff --
this should be opened only once not once per partition
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Bypa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-155844842
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45642/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Bypa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-155806597
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/5622#discussion_r28986044
--- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala ---
@@ -736,11 +734,16 @@ private[spark] class ExternalSorter[K, V, C](
val writeStartTime = System.nanoTime
util.Utils.tryWithSafeFinally {
for (i <- 0 until numPartitions) {
- val in = new FileInputStream(partitionWriters(i).fileSegment().file)
- util.Utils.tryWithSafeFinally {
- lengths(i) = org.apache.spark.util.Utils.copyStream(in, out, false, transferToEnabled)
- } {
- in.close()
+ val file = partitionWriters(i).fileSegment().file
+ if (!file.exists()) {
+ lengths(i) = 0
+ } else {
+ val in = new FileInputStream(file)
+ util.Utils.tryWithSafeFinally {
--- End diff --
Shouldn't we avoid using partial package names like this? Just import spark.util.Utils.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-94983878
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30710/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Bypa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-155806567
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-94981605
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30709/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7041] Avoid writing empty files in Exte...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/5622#issuecomment-94978516
Doh, looks like this legitimately failed tests in ExternalSorterSuite due to the empty files not being present. I'll investigate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org