You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by witgo <gi...@git.apache.org> on 2014/09/13 04:35:31 UTC
[GitHub] spark pull request: [SPARK-3517]mapPartitions is not correct clear...
GitHub user witgo opened a pull request:
https://github.com/apache/spark/pull/2376
[SPARK-3517]mapPartitions is not correct clearing up the closure
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/witgo/spark SPARK-3517
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2376.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2376
----
commit 9d841bb3cc228f1c5b7a120d959b19c3b929d274
Author: GuoQiang Li <wi...@qq.com>
Date: 2014-09-13T02:34:20Z
mapPartitions is not correct clearing up the closure
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2376#issuecomment-55480695
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20248/consoleFull) for PR 2376 at commit [`9d841bb`](https://github.com/apache/spark/commit/9d841bb3cc228f1c5b7a120d959b19c3b929d274).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` class Dummy(object):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3517]mapPartitions is not correct clear...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2376#issuecomment-55479429
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20248/consoleFull) for PR 2376 at commit [`9d841bb`](https://github.com/apache/spark/commit/9d841bb3cc228f1c5b7a120d959b19c3b929d274).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...
Posted by witgo <gi...@git.apache.org>.
Github user witgo closed the pull request at:
https://github.com/apache/spark/pull/2376
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2376#issuecomment-55482040
What is the problem here? Can you give an example or test case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2376#issuecomment-55509430
It seems like the issue here is that unnecessary objects are being included in the closure, since presumably this bug would also manifest itself through serialization errors if the extra objects aren't serializable
Maybe we can test this directly with a mock object and a check that its `writeObject` method isn't called.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:
https://github.com/apache/spark/pull/2376#issuecomment-57078975
I temporarily can not reproduce it, and close this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:
https://github.com/apache/spark/pull/2376#issuecomment-55482720
@rxin Code like this:
```
val topicModel = "Big object"
val broadcastModel = data.context.broadcast(topicModel)
corpus = corpus.mapPartitions { docs =>
val topicModel = broadcastModel.value
.....
}
```
The serialized corpus RDD and serialized topicModel broadcast almost as big.
` cat spark.log | grep 'stored as values in memory'` =>
```
14/09/13 00:49:21 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 197.5 MB, free 2.6 GB)
14/09/13 00:49:24 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 197.7 MB, free 2.3 GB)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:
https://github.com/apache/spark/pull/2376#issuecomment-55913971
I tried to reproduce it with the way the test case, but was unsuccessful.But like the code below does appear the problem, do not know why
```scala
def runGibbsSampling(
data: RDD[Document], initModel: TopicModel,
totalIter: Int, burnInIter: Int
): (TopicModel, RDD[Document]) = {
require(totalIter > burnInIter, "totalIter is less than burnInIter")
require(totalIter > 0, "totalIter is less than 0")
require(burnInIter > 0, "burnInIter is less than 0")
val (numTopics, numTerms, alpha, beta) = (initModel.topicCounts_.size,
initModel.topicTermCounts_.head.size,
initModel.alpha, initModel.beta)
val probModel = TopicModel(numTopics, numTerms, alpha, beta)
logInfo("Start initialization")
var (topicModel, corpus) = sampleTermAssignment(data, initModel)
for (iter <- 1 to totalIter) {
logInfo("Start Gibbs sampling (Iteration %d/%d)".format(iter, totalIter))
val broadcastModel = data.context.broadcast(topicModel)
val previousCorpus = corpus
corpus = corpus.mapPartitions { docs =>
val rand = new Random
val topicModel = broadcastModel.value
val topicThisTerm = BDV.zeros[Double](numTopics)
docs.map { doc =>
val content = doc.content
val topics = doc.topics
val topicsDist = doc.topicsDist
for (i <- 0 until content.length) {
val term = content(i)
val topic = topics(i)
val chosenTopic = topicModel.dropOneDistSampler(topicsDist, topicThisTerm,
rand, term, topic)
if (topic != chosenTopic) {
topics(i) = chosenTopic
topicsDist(topic) += -1
topicsDist(chosenTopic) += 1
topicModel.update(term, topic, -1)
topicModel.update(term, chosenTopic, 1)
}
}
doc
}
}.setName(s"LDA-$iter").persist(StorageLevel.MEMORY_AND_DISK)
if (iter % 5 == 0 && data.context.getCheckpointDir.isDefined) {
corpus.checkpoint()
}
topicModel = collectTopicCounters(corpus, numTerms, numTopics)
if (iter > burnInIter) {
probModel.merge(topicModel)
}
previousCorpus.unpersist()
broadcastModel.unpersist()
}
val burnIn = (totalIter - burnInIter).toDouble
probModel.topicCounts_ :/= burnIn
probModel.topicTermCounts_.foreach(_ :/= burnIn)
(probModel, corpus)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2376#issuecomment-55482915
Can you add a unit test? You can just create a RDD and serialize it to test the size.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org