You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by davies <gi...@git.apache.org> on 2014/10/06 22:47:32 UTC
[GitHub] spark pull request: embed small object in broadcast to avoid RPC
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/2681
embed small object in broadcast to avoid RPC
For most of tasks, the serialized data will small, such as less than 4k or 8k, we can avoid the RPC at all if the data was embedded in the Broadcast object it self.
With this patch, The size of task will be similar to that before we use broadcast for them, no RPC (but still cached, only one deserialization per executor)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark embed
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2681.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2681
----
commit 3fd051d4bc9a305375f6d5abeb64258beacaa3d9
Author: Davies Liu <da...@gmail.com>
Date: 2014-10-03T00:07:27Z
embed small object in broadcast to avoid RPC
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-64299267
[Test build #23808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23808/consoleFull) for PR 2681 at commit [`1ffd763`](https://github.com/apache/spark/commit/1ffd763c8c8c18b356d86513383b35fb847e8569).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-61345908
[Test build #500 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/500/consoleFull) for PR 2681 at commit [`247b53b`](https://github.com/apache/spark/commit/247b53b4de2f34eaaeab9f757462a022ffc784d3).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60452847
Even if we merge #2933, I still would like to have this, because people could use broadcast for small dataset (such as in MLlib), this patch can improve these cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-59633500
Based on some offline discussions, we think that we may have identified the cause of the Snappy issues that we've seen with TorrentBroadcast. I'm going to merge this PR and then work on some refactoring that may fix the underlying bug (or at least add additional log information to aid in debugging). I also have some thoughts on how to improve the test coverage of our broadcast implementation, but I'l address this in my PR.
Thanks @davies!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60692969
[Test build #481 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/481/consoleFull) for PR 2681 at commit [`823302c`](https://github.com/apache/spark/commit/823302c0407c12ba26559c5117a4e7f84deaf6ac).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60446011
**[Test build #22151 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22151/consoleFull)** for PR 2681 at commit [`823302c`](https://github.com/apache/spark/commit/823302c0407c12ba26559c5117a4e7f84deaf6ac) after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60723279
[Test build #483 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/483/consoleFull) for PR 2681 at commit [`823302c`](https://github.com/apache/spark/commit/823302c0407c12ba26559c5117a4e7f84deaf6ac).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58116679
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21349/Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60822025
[Test build #22367 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22367/consoleFull) for PR 2681 at commit [`247b53b`](https://github.com/apache/spark/commit/247b53b4de2f34eaaeab9f757462a022ffc784d3).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2681#discussion_r18534899
--- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -161,6 +178,10 @@ private[spark] class TorrentBroadcast[T: ClassTag](
_value = x.asInstanceOf[T]
case None =>
+ if (numBlocks == 0) {
--- End diff --
when will this ever happen?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58272706
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21405/consoleFull) for PR 2681 at commit [`fc9d2c4`](https://github.com/apache/spark/commit/fc9d2c4c7982a519525d66527ba42d008e31cf39).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58096752
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21341/consoleFull) for PR 2681 at commit [`3fd051d`](https://github.com/apache/spark/commit/3fd051d4bc9a305375f6d5abeb64258beacaa3d9).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/2681#discussion_r18537422
--- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -161,6 +178,10 @@ private[spark] class TorrentBroadcast[T: ClassTag](
_value = x.asInstanceOf[T]
case None =>
+ if (numBlocks == 0) {
--- End diff --
In the case of having multiple sparkenv in different threads, then this will happen, also blockmanager will return an invalid results.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60814705
[Test build #487 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/487/consoleFull) for PR 2681 at commit [`247b53b`](https://github.com/apache/spark/commit/247b53b4de2f34eaaeab9f757462a022ffc784d3).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58104999
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21341/consoleFull) for PR 2681 at commit [`3fd051d`](https://github.com/apache/spark/commit/3fd051d4bc9a305375f6d5abeb64258beacaa3d9).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2681#discussion_r18548586
--- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -161,6 +178,10 @@ private[spark] class TorrentBroadcast[T: ClassTag](
_value = x.asInstanceOf[T]
case None =>
+ if (numBlocks == 0) {
--- End diff --
can u add some inline comment explaining this? thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60452848
[Test build #419 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/419/consoleFull) for PR 2681 at commit [`823302c`](https://github.com/apache/spark/commit/823302c0407c12ba26559c5117a4e7f84deaf6ac).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60445861
PR #2933 will so similar things as this one, that also works for HttpBroadcast.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60825071
[Test build #487 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/487/consoleFull) for PR 2681 at commit [`247b53b`](https://github.com/apache/spark/commit/247b53b4de2f34eaaeab9f757462a022ffc784d3).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class VectorTransformer(object):`
* `class Normalizer(VectorTransformer):`
* `class JavaModelWrapper(VectorTransformer):`
* `class StandardScalerModel(JavaModelWrapper):`
* `class StandardScaler(object):`
* `class HashingTF(object):`
* `class IDFModel(JavaModelWrapper):`
* `class IDF(object):`
* `class Word2VecModel(JavaModelWrapper):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60472515
[Test build #429 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/429/consoleFull) for PR 2681 at commit [`823302c`](https://github.com/apache/spark/commit/823302c0407c12ba26559c5117a4e7f84deaf6ac).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-62980143
@JoshRosen @pwendell @rxin This is an optimization for TorrentBroadcast (avoid all RPC for small objects), which can reduce latency for streaming (not benchmarked).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-64305586
[Test build #23808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23808/consoleFull) for PR 2681 at commit [`1ffd763`](https://github.com/apache/spark/commit/1ffd763c8c8c18b356d86513383b35fb847e8569).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class CompressedSerializer(FramedSerializer):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2681#discussion_r18534919
--- Diff: core/src/test/scala/org/apache/spark/broadcast/BroadcastSuite.scala ---
@@ -257,7 +257,7 @@ class BroadcastSuite extends FunSuite with LocalSparkContext {
new SparkContext("local", "test", broadcastConf)
}
val blockManagerMaster = sc.env.blockManager.master
- val list = List[Int](1, 2, 3, 4)
+ val list = (1 to 4096).toList
--- End diff --
can u make sure we have unit tests for both cases? i.e. small broadcast and large ones.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-64305591
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23808/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58105010
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21341/Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-59464471
This error was not happened in tests of this PR, it happened in tests of our product, which have similar pattern as streaming, the job was submitted via py4j.
The PR also check the number of blocks in readBlocks(), will throw a meaningful exception in case of fail to get cached object in local mode.
TorrentBroadcast is so complicated (including several RPC) that it's not as stable as HTTPBroadcast or w/o broadcast (we had saw some cases reported by users in maillist), the motivation of this PR is to remove the complicity for most cases (serialized task is small), then it will be more stable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60718068
[Test build #483 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/483/consoleFull) for PR 2681 at commit [`823302c`](https://github.com/apache/spark/commit/823302c0407c12ba26559c5117a4e7f84deaf6ac).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-59481089
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/393/consoleFull) for PR 2681 at commit [`732949f`](https://github.com/apache/spark/commit/732949fc8dda3a4caacc04b0979b67729af4c975).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-59488832
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/393/consoleFull) for PR 2681 at commit [`732949f`](https://github.com/apache/spark/commit/732949fc8dda3a4caacc04b0979b67729af4c975).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60698386
[Test build #481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/481/consoleFull) for PR 2681 at commit [`823302c`](https://github.com/apache/spark/commit/823302c0407c12ba26559c5117a4e7f84deaf6ac).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58231163
Should we change the default EMBEDDED_SIZE to 8k ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60446020
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22151/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-59123858
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21740/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/2681#discussion_r18537479
--- Diff: core/src/test/scala/org/apache/spark/broadcast/BroadcastSuite.scala ---
@@ -257,7 +257,7 @@ class BroadcastSuite extends FunSuite with LocalSparkContext {
new SparkContext("local", "test", broadcastConf)
}
val blockManagerMaster = sc.env.blockManager.master
- val list = List[Int](1, 2, 3, 4)
+ val list = (1 to 4096).toList
--- End diff --
I check this, the previous cases had covered small broadcasts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58263525
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21405/consoleFull) for PR 2681 at commit [`fc9d2c4`](https://github.com/apache/spark/commit/fc9d2c4c7982a519525d66527ba42d008e31cf39).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-63188181
I've pushed a commit to `spark-perf` which allows us to benchmark task launching throughput using different sized closures: https://github.com/databricks/spark-perf/commit/79e615a6c2ea881fc6abe1e7d20368419711b18c
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60430910
[Test build #22151 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22151/consoleFull) for PR 2681 at commit [`823302c`](https://github.com/apache/spark/commit/823302c0407c12ba26559c5117a4e7f84deaf6ac).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60457710
[Test build #419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/419/consoleFull) for PR 2681 at commit [`823302c`](https://github.com/apache/spark/commit/823302c0407c12ba26559c5117a4e7f84deaf6ac).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58717552
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/345/consoleFull) for PR 2681 at commit [`fc9d2c4`](https://github.com/apache/spark/commit/fc9d2c4c7982a519525d66527ba42d008e31cf39).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-59120483
Sometimes hit this bug during pyspark testing
```
Py4JJavaError: An error occurred while calling o55.collect. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 1 times, most recent failure: Lost task 0.0 in stage 5.0 (TID 8, localhost): java.io.IOException: PARSING_ERROR(2)
[info] org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:84)
[info] org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
[info] org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:594)
[info] org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
[info] org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
[info] org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
[info] org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
[info] org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:216)
[info] org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:170)
[info] sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source)
[info] sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[info] java.lang.reflect.Method.invoke(Method.java:606)
[info] java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
[info] java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
[info] java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
[info] java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
[info] java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
[info] java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
[info] java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
[info] java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
[info] java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
[info] org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
[info] org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
[info] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:164)
[info] java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[info] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[info] java.lang.Thread.run(Thread.java:745)
[info] Driver stacktrace:
[info] at
```
The test is really simple:
```
sc.parallelize(range(10)).count()
```
The serialized PythonRDD is about 4.1k, so I would like to increase the EMBED_SIZE to 8k, then most of the simple PythonRDD will be embedded, to make tests stable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60811162
[Test build #22367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22367/consoleFull) for PR 2681 at commit [`247b53b`](https://github.com/apache/spark/commit/247b53b4de2f34eaaeab9f757462a022ffc784d3).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58110881
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21349/consoleFull) for PR 2681 at commit [`55a40fa`](https://github.com/apache/spark/commit/55a40fa5c59c89bb5acda6d9b94bef2852c31457).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58116672
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21349/consoleFull) for PR 2681 at commit [`55a40fa`](https://github.com/apache/spark/commit/55a40fa5c59c89bb5acda6d9b94bef2852c31457).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60822038
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22367/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-59634598
I hope that we can have this in 1.1, some people see regression in 1.1 because of TorrentBroadcast, this patch will help for those.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-61350901
[Test build #500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/500/consoleFull) for PR 2681 at commit [`247b53b`](https://github.com/apache/spark/commit/247b53b4de2f34eaaeab9f757462a022ffc784d3).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60474748
**[Test build #429 timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/429/consoleFull)** for PR 2681 at commit [`823302c`](https://github.com/apache/spark/commit/823302c0407c12ba26559c5117a4e7f84deaf6ac) after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60431415
@JoshRosen Could you look at this again? I had rebased it on your changes. Hope this could make the tests more stable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-60812450
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22366/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-59633558
Actually, I changed my mind: I'm going to hold off on merging this because I don't want to backport the embedding of small objects into `branch-1.1` but I _do_ want to backport the fix for the TorrentBroadcast Snappy issue. I'll try to merge my patch in first, then rebase this on top of it. Sorry in advance for the merge conflict.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58272713
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21405/Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-59458452
@davies: that exception that you hit is _very_ helpful; it looks like you've been able to reproduce [SPARK-3958](https://issues.apache.org/jira/browse/SPARK-3958) in local tests. That JIRA describes some reports of cases where we've seen TorrentBroadcast throw similar errors, but I've had a really hard time debugging that issue because I've had trouble reliably reproducing this bug. Why do you think `EMBED_SIZE` seems to affect the reliability of TorrentBroadcast? If it does, that could be a valuable debugging clue for this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-58710262
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/345/consoleFull) for PR 2681 at commit [`fc9d2c4`](https://github.com/apache/spark/commit/fc9d2c4c7982a519525d66527ba42d008e31cf39).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-61435336
ping dashboard
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org