You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tmyklebu <gi...@git.apache.org> on 2014/04/30 01:56:51 UTC
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
GitHub user tmyklebu opened a pull request:
https://github.com/apache/spark/pull/593
[SPARK-1672][WIP] Separate partitioning in ALS
This is a work-in-progress. At present, the numbers of user and product blocks are not exposed to the user and no tests are present. As such, this should probably not be pulled as is.
This pull request changes the per-instance partitioner to two per-instance partitioners, one for users and one for products.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tmyklebu/spark alsseppar
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/593.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #593
----
commit c774d7d4bff91c9387d059d1189799fa0ff1f4b0
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-14T22:01:18Z
Make the partitioner a member variable and use it instead of modding directly.
commit c90b6d8e91f86cf89adf28de6f9185647c87e5c8
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-14T22:10:30Z
Scramble user and product ids before bucketing.
commit df27697649de50d364c42c76aebaebb34cbe87e2
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-15T19:47:17Z
Support custom partitioners. Currently we use the same partitioner for users and products.
commit d872b098d41c4fc088e579e8fe199aca149bca64
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-16T12:19:48Z
Add negative id ALS test.
commit 36a0f43519a1e8ea800b960157f8c8b050139105
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-16T16:42:31Z
Make the partitioner private.
commit 5ec9e6cd237c4ac7c1b597614c880ae75bacceee
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-16T17:00:39Z
Clean a couple of things up using 'map'.
commit f8413451c807282100a9be506ca2c992abb81918
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-16T17:12:47Z
Fix daft bug creating 'pairs', also for -> foreach.
commit 40edc235e59aab56d6f65c73ffe98859c78a889b
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-16T18:14:38Z
Fix missing space.
commit 674933abb7a373dc1c913467d668bad9045e560f
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-19T23:36:52Z
Fix style.
commit 495784f2a172957ab490e0f77ea504c0179ab798
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-19T23:41:23Z
Merge branch 'master' of https://github.com/apache/spark
commit 23d6f91b52c88b7006ec78496f777b72b1881bb4
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-21T00:06:19Z
Stop making the partitioner configurable.
commit dcf583ac4001c6da8d6b85e45e88043861a351d8
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-21T19:56:48Z
Remove the partitioner member variable; instead, thread that needle everywhere it needs to go.
commit 021f54b5d3d5177a5185442aaf3ddcde51a354a2
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-29T05:47:41Z
Separate user and product blocks.
commit bd8a75c53e1b84b4dfb493581bc1945eb93d624b
Author: Tor Myklebust <tm...@gmail.com>
Date: 2014-04-29T23:21:26Z
Merge branch 'master' of github.com:apache/spark into alsseppar
Conflicts:
mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/593#issuecomment-41751186
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12191731
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -163,14 +165,27 @@ class ALS private (
def run(ratings: RDD[Rating]): MatrixFactorizationModel = {
val sc = ratings.context
- val numBlocks = if (this.numBlocks == -1) {
- math.max(sc.defaultParallelism, ratings.partitions.size / 2)
+ val numUserBlocks = if (this.numUserBlocks == -1) {
+ math.max(sc.defaultParallelism, ratings.partitions.size / 2) + 1
--- End diff --
Why `+ 1`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12192379
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -381,14 +399,15 @@ class ALS private (
* the users (or (blockId, (p, u, r)) for the products). We create these simultaneously to avoid
* having to shuffle the (blockId, (u, p, r)) RDD twice, or to cache it.
*/
- private def makeLinkRDDs(numBlocks: Int, ratings: RDD[(Int, Rating)], partitioner: Partitioner)
+ private def makeLinkRDDs(numUserBlocks: Int, numProductBlocks: Int, ratings: RDD[(Int, Rating)],
+ userPartitioner: Partitioner, productPartitioner: Partitioner)
--- End diff --
Is `userPartitioner` referenced in the method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/593#issuecomment-41746643
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14572/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/593#issuecomment-41746562
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by tmyklebu <gi...@git.apache.org>.
Github user tmyklebu commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12195324
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -381,14 +399,15 @@ class ALS private (
* the users (or (blockId, (p, u, r)) for the products). We create these simultaneously to avoid
* having to shuffle the (blockId, (u, p, r)) RDD twice, or to cache it.
*/
- private def makeLinkRDDs(numBlocks: Int, ratings: RDD[(Int, Rating)], partitioner: Partitioner)
+ private def makeLinkRDDs(numUserBlocks: Int, numProductBlocks: Int, ratings: RDD[(Int, Rating)],
+ userPartitioner: Partitioner, productPartitioner: Partitioner)
--- End diff --
Nope! Nice catch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by tmyklebu <gi...@git.apache.org>.
Github user tmyklebu commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12195151
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -163,14 +165,27 @@ class ALS private (
def run(ratings: RDD[Rating]): MatrixFactorizationModel = {
val sc = ratings.context
- val numBlocks = if (this.numBlocks == -1) {
- math.max(sc.defaultParallelism, ratings.partitions.size / 2)
+ val numUserBlocks = if (this.numUserBlocks == -1) {
+ math.max(sc.defaultParallelism, ratings.partitions.size / 2) + 1
--- End diff --
Testing hack accidentally left in. Will remove.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/593#issuecomment-41932527
@tmyklebu I think the inconsistency in partitioners created more stages in the computation and larger shuffle size. We should subclass `Partitioner` and override `equals`, so Spark knows two RDDs use the same partitioner and then group/join operations can take advantage of it. Hope it can help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/593#issuecomment-41746642
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/593#issuecomment-41749027
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by tmyklebu <gi...@git.apache.org>.
Github user tmyklebu commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12214297
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -381,14 +399,15 @@ class ALS private (
* the users (or (blockId, (p, u, r)) for the products). We create these simultaneously to avoid
* having to shuffle the (blockId, (u, p, r)) RDD twice, or to cache it.
*/
- private def makeLinkRDDs(numBlocks: Int, ratings: RDD[(Int, Rating)], partitioner: Partitioner)
+ private def makeLinkRDDs(numUserBlocks: Int, numProductBlocks: Int, ratings: RDD[(Int, Rating)],
+ userPartitioner: Partitioner, productPartitioner: Partitioner)
: (RDD[(Int, InLinkBlock)], RDD[(Int, OutLinkBlock)]) =
{
- val grouped = ratings.partitionBy(new HashPartitioner(numBlocks))
+ val grouped = ratings.partitionBy(new HashPartitioner(numUserBlocks))
--- End diff --
Not sure I understand. That `HashPartitioner` serves as the identity map on the `blockId`s. The `blockId` in the `mapPartitionsWithValues` on the next line coincides with the `blockId`s of all of the ratings in `elements`.
`makeLinkRDDs` returns two RDDs that are both partitioned by that `HashPartitioner`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/593#issuecomment-41746556
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12191879
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -163,14 +165,27 @@ class ALS private (
def run(ratings: RDD[Rating]): MatrixFactorizationModel = {
val sc = ratings.context
- val numBlocks = if (this.numBlocks == -1) {
- math.max(sc.defaultParallelism, ratings.partitions.size / 2)
+ val numUserBlocks = if (this.numUserBlocks == -1) {
--- End diff --
It is error-prone to reuse the name `numUserBlocks`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12191764
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -163,14 +165,27 @@ class ALS private (
def run(ratings: RDD[Rating]): MatrixFactorizationModel = {
val sc = ratings.context
- val numBlocks = if (this.numBlocks == -1) {
- math.max(sc.defaultParallelism, ratings.partitions.size / 2)
+ val numUserBlocks = if (this.numUserBlocks == -1) {
+ math.max(sc.defaultParallelism, ratings.partitions.size / 2) + 1
} else {
- this.numBlocks
+ this.numUserBlocks
+ }
+ val numProductBlocks = if (this.numUserBlocks == -1) {
+ math.max(sc.defaultParallelism, ratings.partitions.size / 2) + 1
--- End diff --
Same question.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/593
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12191694
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -105,14 +106,15 @@ class ALS private (
* Constructs an ALS instance with default parameters: {numBlocks: -1, rank: 10, iterations: 10,
* lambda: 0.01, implicitPrefs: false, alpha: 1.0}.
*/
- def this() = this(-1, 10, 10, 0.01, false, 1.0)
+ def this() = this(-1, -1, 10, 10, 0.01, false, 1.0)
/**
* Set the number of blocks to parallelize the computation into; pass -1 for an auto-configured
* number of blocks. Default: -1.
--- End diff --
Need more doc mentioning this overwrites both `numUserBlocks` and `numProductBlocks`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12191807
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -163,14 +165,27 @@ class ALS private (
def run(ratings: RDD[Rating]): MatrixFactorizationModel = {
val sc = ratings.context
- val numBlocks = if (this.numBlocks == -1) {
- math.max(sc.defaultParallelism, ratings.partitions.size / 2)
+ val numUserBlocks = if (this.numUserBlocks == -1) {
+ math.max(sc.defaultParallelism, ratings.partitions.size / 2) + 1
} else {
- this.numBlocks
+ this.numUserBlocks
+ }
+ val numProductBlocks = if (this.numUserBlocks == -1) {
+ math.max(sc.defaultParallelism, ratings.partitions.size / 2) + 1
+ } else {
+ this.numProductBlocks
+ }
+
+ val userPartitioner = new Partitioner {
--- End diff --
Create a private method `newPartitioner(numBlocks: Int)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/593#issuecomment-41751187
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14576/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12196075
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -163,14 +165,27 @@ class ALS private (
def run(ratings: RDD[Rating]): MatrixFactorizationModel = {
val sc = ratings.context
- val numBlocks = if (this.numBlocks == -1) {
- math.max(sc.defaultParallelism, ratings.partitions.size / 2)
+ val numUserBlocks = if (this.numUserBlocks == -1) {
+ math.max(sc.defaultParallelism, ratings.partitions.size / 2) + 1
} else {
- this.numBlocks
+ this.numUserBlocks
+ }
+ val numProductBlocks = if (this.numUserBlocks == -1) {
+ math.max(sc.defaultParallelism, ratings.partitions.size / 2) + 1
+ } else {
+ this.numProductBlocks
+ }
+
+ val userPartitioner = new Partitioner {
--- End diff --
We need subclass `Partitioner` and implement `equals`. This is important for `groupBy` and `join`, because Spark compares the partitioner to determine the shuffle dependency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12196231
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -381,14 +399,15 @@ class ALS private (
* the users (or (blockId, (p, u, r)) for the products). We create these simultaneously to avoid
* having to shuffle the (blockId, (u, p, r)) RDD twice, or to cache it.
*/
- private def makeLinkRDDs(numBlocks: Int, ratings: RDD[(Int, Rating)], partitioner: Partitioner)
+ private def makeLinkRDDs(numUserBlocks: Int, numProductBlocks: Int, ratings: RDD[(Int, Rating)],
+ userPartitioner: Partitioner, productPartitioner: Partitioner)
: (RDD[(Int, InLinkBlock)], RDD[(Int, OutLinkBlock)]) =
{
- val grouped = ratings.partitionBy(new HashPartitioner(numBlocks))
+ val grouped = ratings.partitionBy(new HashPartitioner(numUserBlocks))
--- End diff --
Instead of using `HashPartitioner`, we should use the same partitioner we used for creating in/out link blocks. This helps determine shuffle dependency. See my previous comment on subclassing partitioner.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/593#issuecomment-41749024
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1672][WIP] Separate partitioning in ALS
Posted by tmyklebu <gi...@git.apache.org>.
Github user tmyklebu commented on a diff in the pull request:
https://github.com/apache/spark/pull/593#discussion_r12195265
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -163,14 +165,27 @@ class ALS private (
def run(ratings: RDD[Rating]): MatrixFactorizationModel = {
val sc = ratings.context
- val numBlocks = if (this.numBlocks == -1) {
- math.max(sc.defaultParallelism, ratings.partitions.size / 2)
+ val numUserBlocks = if (this.numUserBlocks == -1) {
--- End diff --
@mengxr It's error-prone to reuse the name `numBlocks`, too, but that didn't stop the previous author. I can remove the shadowing problem everywhere if you prefer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---