You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mateiz <gi...@git.apache.org> on 2014/06/08 08:07:16 UTC

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

GitHub user mateiz opened a pull request:

    https://github.com/apache/spark/pull/1009

    [SPARK-2044] Pluggable interface for shuffles

    This is a first cut at moving shuffle logic behind a pluggable interface, as described at https://issues.apache.org/jira/browse/SPARK-2044, to let us more easily experiment with new shuffle implementations. It moves the existing shuffle code to a class HashShuffleManager behind a general ShuffleManager interface.
    
    Two things are still missing to make this complete:
    * MapOutputTracker needs to be hidden behind the ShuffleManager interface; this will also require adding methods to ShuffleManager that will let the DAGScheduler interact with it as it does with the MapOutputTracker today
    * The code to do map-sides and reduce-side combine in ShuffledRDD, PairRDDFunctions, etc needs to be moved into the ShuffleManager's readers and writers
    
    However, some of these may also be done later after we merge the current interface.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mateiz/spark pluggable-shuffle

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/1009.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1009
    
----
commit 75cc0446d5dc3b605961c0c41d036fab4ee8e9e8
Author: Matei Zaharia <ma...@databricks.com>
Date:   2014-06-06T05:09:37Z

    Partial work to move hash shuffle in

commit 55c77175dc307030285a541ca2c47f31c05a207a
Author: Matei Zaharia <ma...@databricks.com>
Date:   2014-06-06T21:19:33Z

    Changed RDD code to use ShuffleReader

commit f6f011d7a76c7ef2c6de20afd9bdeb8007ca56bb
Author: Matei Zaharia <ma...@databricks.com>
Date:   2014-06-06T21:56:54Z

    Move hash shuffle reader behind ShuffleManager interface

commit 4f681ba7ebee7e8ad0b8cd5f8d692f2aaa910992
Author: Matei Zaharia <ma...@databricks.com>
Date:   2014-06-08T05:41:36Z

    Move write part of ShuffleMapTask to ShuffleManager

commit ac56831d24e87d8d445eaed30297e4cf3fb73deb
Author: Matei Zaharia <ma...@databricks.com>
Date:   2014-06-08T05:53:18Z

    Bug fix and better error message

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13523026
  
    --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala ---
    @@ -0,0 +1,42 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.shuffle.hash
    +
    +import org.apache.spark.serializer.Serializer
    +import org.apache.spark.shuffle.{BaseShuffleHandle, ShuffleReader}
    +import org.apache.spark.TaskContext
    +
    +class HashShuffleReader[K, C](
    +    handle: BaseShuffleHandle[K, _, C],
    +    startPartition: Int,
    +    endPartition: Int,
    +    context: TaskContext)
    +  extends ShuffleReader[K, C]
    +{
    +  require(endPartition == startPartition + 1,
    --- End diff --
    
    Who enforces this? I thought the API allows users to create a ShuffleReader with an arbitrary partition range, and it's up to the ShuffleReader to implement it correctly (perhaps by chunking itself up).



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13688732
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
    @@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
         part: Partitioner)
       extends RDD[P](prev.context, Nil) {
     
    -  private var serializer: Serializer = null
    +  private var serializer: Option[Serializer] = None
     
    +  /** Set a serializer for this RDD's shuffle, or null to use the default (spark.serializer) */
       def setSerializer(serializer: Serializer): ShuffledRDD[K, V, P] = {
    -    this.serializer = serializer
    +    this.serializer = Option(serializer)
    --- End diff --
    
    Not if serializer is null


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13529575
  
    --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleManager.scala ---
    @@ -0,0 +1,59 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.shuffle.hash
    +
    +import org.apache.spark._
    +import org.apache.spark.shuffle._
    +
    +/**
    + * A ShuffleManager using the hash-based implementation available up to and including Spark 1.0.
    --- End diff --
    
    May want to change the wording. "available up to and including" seems to suggest that we are going to remove this post 1.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45429378
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45662407
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by colorant <gi...@git.apache.org>.
Github user colorant commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13531291
  
    --- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala ---
    @@ -0,0 +1,57 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.shuffle
    +
    +import org.apache.spark.{TaskContext, ShuffleDependency}
    +
    +/**
    + * Pluggable interface for shuffle systems. A ShuffleManager is created in SparkEnv on both the
    + * driver and executors, based on the spark.shuffle.manager setting. The driver registers shuffles
    + * with it, and executors (or tasks running locally in the driver) can ask to read and write data.
    + *
    + * NOTE: this will be instantiated by SparkEnv so its constructor can take a SparkConf and
    + * boolean isDriver as parameters.
    + */
    +private[spark] trait ShuffleManager {
    +  /**
    +   * Register a shuffle with the manager and obtain a handle for it to pass to tasks.
    +   */
    +  def registerShuffle[K, V, C](
    +      shuffleId: Int,
    +      numMaps: Int,
    --- End diff --
    
    what is the target usage of this numMaps? seems not used in current code. And seems could be retrieved by other method?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13525201
  
    --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala ---
    @@ -0,0 +1,42 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.shuffle.hash
    +
    +import org.apache.spark.serializer.Serializer
    +import org.apache.spark.shuffle.{BaseShuffleHandle, ShuffleReader}
    +import org.apache.spark.TaskContext
    +
    +class HashShuffleReader[K, C](
    +    handle: BaseShuffleHandle[K, _, C],
    +    startPartition: Int,
    +    endPartition: Int,
    +    context: TaskContext)
    +  extends ShuffleReader[K, C]
    +{
    +  require(endPartition == startPartition + 1,
    --- End diff --
    
    (The reason is that for this to work we need to move aggregation into the ShuffleReader.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45829206
  
    Thanks Reynold. I've now moved the other TODOs into sub-tasks on https://issues.apache.org/jira/browse/SPARK-2044.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13690317
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
    @@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
         part: Partitioner)
       extends RDD[P](prev.context, Nil) {
     
    -  private var serializer: Serializer = null
    +  private var serializer: Option[Serializer] = None
     
    +  /** Set a serializer for this RDD's shuffle, or null to use the default (spark.serializer) */
       def setSerializer(serializer: Serializer): ShuffledRDD[K, V, P] = {
    -    this.serializer = serializer
    +    this.serializer = Option(serializer)
    --- End diff --
    
    ```
    scala> var serializer: Option[java.lang.Integer] = null
    serializer: Option[Integer] = null
    
    scala> serializer = Some(1)
    serializer: Option[Integer] = Some(1)
    
    scala> serializer.get
    res0: Integer = 1
    
    scala> serializer = Some(null)
    serializer: Option[Integer] = Some(null)
    
    scala> serializer.get
    res1: Integer = null
    
    scala> serializer.isDefined
    res2: Boolean = true
    
    scala> serializer = Option(null)
    serializer: Option[Integer] = None
    
    scala> serializer.isDefined
    res3: Boolean = false
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45819540
  
    Thanks Andrew for the comments. @pwendell @rxin I think this is good to merge now if it passes tests because other people want to contribute some other things (e.g. moving aggregation into the shuffle implementation) and I'd like to let them do that in parallel. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45429416
  
    Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45429375
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45429424
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13608317
  
    --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala ---
    @@ -15,26 +15,26 @@
      * limitations under the License.
      */
     
    -package org.apache.spark
    +package org.apache.spark.shuffle.hash
     
     import scala.collection.mutable.ArrayBuffer
     import scala.collection.mutable.HashMap
     
     import org.apache.spark.executor.ShuffleReadMetrics
     import org.apache.spark.serializer.Serializer
    -import org.apache.spark.storage.{BlockId, BlockManagerId, ShuffleBlockId}
    +import org.apache.spark.storage.{BlockId, BlockManagerId}
    --- End diff --
    
    nit: I don't think you intended to separate out this import


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/1009


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45819661
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45656917
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45819651
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13612262
  
    --- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala ---
    @@ -0,0 +1,57 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.shuffle
    +
    +import org.apache.spark.{TaskContext, ShuffleDependency}
    +
    +/**
    + * Pluggable interface for shuffle systems. A ShuffleManager is created in SparkEnv on both the
    + * driver and executors, based on the spark.shuffle.manager setting. The driver registers shuffles
    + * with it, and executors (or tasks running locally in the driver) can ask to read and write data.
    + *
    + * NOTE: this will be instantiated by SparkEnv so its constructor can take a SparkConf and
    + * boolean isDriver as parameters.
    + */
    +private[spark] trait ShuffleManager {
    +  /**
    +   * Register a shuffle with the manager and obtain a handle for it to pass to tasks.
    +   */
    +  def registerShuffle[K, V, C](
    +      shuffleId: Int,
    +      numMaps: Int,
    --- End diff --
    
    It will be used to register the shuffle with the map output tracker and things like that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45821959
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45821961
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15699/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45827229
  
    Ok merging this in master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13525195
  
    --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala ---
    @@ -0,0 +1,42 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.shuffle.hash
    +
    +import org.apache.spark.serializer.Serializer
    +import org.apache.spark.shuffle.{BaseShuffleHandle, ShuffleReader}
    +import org.apache.spark.TaskContext
    +
    +class HashShuffleReader[K, C](
    +    handle: BaseShuffleHandle[K, _, C],
    +    startPartition: Int,
    +    endPartition: Int,
    +    context: TaskContext)
    +  extends ShuffleReader[K, C]
    +{
    +  require(endPartition == startPartition + 1,
    --- End diff --
    
    Yup, this is just not yet supported in the first pass.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45429981
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15537/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13573936
  
    --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
    @@ -163,13 +164,20 @@ object SparkEnv extends Logging {
         def instantiateClass[T](propertyName: String, defaultClassName: String): T = {
           val name = conf.get(propertyName,  defaultClassName)
           val cls = Class.forName(name, true, Utils.getContextOrSparkClassLoader)
    -      // First try with the constructor that takes SparkConf. If we can't find one,
    -      // use a no-arg constructor instead.
    +      // Look for a constructor taking a SparkConf and a boolean isDriver, than one taking just
    --- End diff --
    
    than -> then


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45429386
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13628104
  
    --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala ---
    @@ -0,0 +1,113 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.shuffle.hash
    +
    +import org.apache.spark.shuffle.{BaseShuffleHandle, ShuffleWriter}
    +import org.apache.spark.{Logging, MapOutputTracker, SparkEnv, TaskContext}
    +import org.apache.spark.storage.{BlockObjectWriter, ShuffleWriterGroup}
    +import org.apache.spark.serializer.Serializer
    +import org.apache.spark.executor.ShuffleWriteMetrics
    +import org.apache.spark.scheduler.MapStatus
    +
    +class HashShuffleWriter[K, V](
    +    handle: BaseShuffleHandle[K, V, _],
    +    mapId: Int,
    +    context: TaskContext)
    +  extends ShuffleWriter[K, V] with Logging {
    +
    +  private val dep = handle.dependency
    +  private val numOutputSplits = dep.partitioner.numPartitions
    +  private val metrics = context.taskMetrics
    +  private var success = false
    +  private var stopping = false
    +
    +  private val blockManager = SparkEnv.get.blockManager
    +  private val shuffleBlockManager = blockManager.shuffleBlockManager
    +  private val ser = Serializer.getSerializer(dep.serializer.getOrElse(null))
    +  private val shuffle = shuffleBlockManager.forMapTask(dep.shuffleId, mapId, numOutputSplits, ser)
    +
    +  /** Write a record to this task's output */
    +  override def write(record: Product2[K, V]): Unit = {
    +    val pair = record.asInstanceOf[Product2[Any, Any]]
    +    val bucketId = dep.partitioner.getPartition(pair._1)
    +    shuffle.writers(bucketId).write(pair)
    +  }
    +
    +  /** Close this writer, passing along whether the map completed */
    +  override def stop(success: Boolean): Option[MapStatus] = {
    +    try {
    +      if (stopping) {
    +        return None
    +      }
    +      stopping = true
    +      if (success) {
    +        try {
    +          return Some(commitWritesAndBuildStatus())
    +        } catch {
    +          case e: Exception =>
    +            revertWrites()
    +            throw e
    +        }
    +      } else {
    +        revertWrites()
    +        return None
    +      }
    +    } finally {
    +      // Release the writers back to the shuffle block manager.
    +      if (shuffle != null && shuffle.writers != null) {
    +        try {
    +          shuffle.releaseWriters(success)
    +        } catch {
    +          case e: Exception => logError("Failed to release shuffle writers", e)
    +        }
    +      }
    +    }
    +  }
    +
    +  private def commitWritesAndBuildStatus(): MapStatus = {
    +    // Commit the writes. Get the size of each bucket block (total block size).
    +    var totalBytes = 0L
    +    var totalTime = 0L
    +    val compressedSizes = shuffle.writers.map { writer: BlockObjectWriter =>
    +      writer.commit()
    +      writer.close()
    +      val size = writer.fileSegment().length
    +      totalBytes += size
    +      totalTime += writer.timeWriting()
    +      MapOutputTracker.compressSize(size)
    +    }
    +
    +    // Update shuffle metrics.
    +    val shuffleMetrics = new ShuffleWriteMetrics
    +    shuffleMetrics.shuffleBytesWritten = totalBytes
    +    shuffleMetrics.shuffleWriteTime = totalTime
    +    metrics.shuffleWriteMetrics = Some(shuffleMetrics)
    +
    +    success = true
    --- End diff --
    
    I don't think this is used anywhere?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45429980
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by colorant <gi...@git.apache.org>.
Github user colorant commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13690254
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
    @@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
         part: Partitioner)
       extends RDD[P](prev.context, Nil) {
     
    -  private var serializer: Serializer = null
    +  private var serializer: Option[Serializer] = None
     
    +  /** Set a serializer for this RDD's shuffle, or null to use the default (spark.serializer) */
       def setSerializer(serializer: Serializer): ShuffledRDD[K, V, P] = {
    -    this.serializer = serializer
    +    this.serializer = Option(serializer)
    --- End diff --
    
    for null it will be None, isn't that the expect result?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45662409
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15631/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45430013
  
    Hi Matei, since we already have `ShuffleWriter` and `ShuffleManager`, do we still need to keep `ShuffleBlockManager`, I think two functionalities of `ShuffleBlockManager` as getting shuffle writer and managing shuffle block location can separately move into `HashShuffleWriter` and `HashShuffleManager` or some others. I think it will be more clear and loose-couple for the shuffle module and `BlockManager`.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13629580
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala ---
    @@ -88,7 +90,7 @@ class CoGroupedRDD[K](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]], part:
             new OneToOneDependency(rdd)
           } else {
             logDebug("Adding shuffle dependency with " + rdd)
    -        new ShuffleDependency[Any, Any](rdd, part, serializer)
    +        new ShuffleDependency[K, Any, Seq[Seq[_]]](rdd, part, serializer)
    --- End diff --
    
    `Seq[Seq[_]]` could also be `CoGroupCombiner`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45656932
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45429427
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13683531
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
    @@ -60,9 +61,10 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
       }
     
       override def compute(split: Partition, context: TaskContext): Iterator[P] = {
    -    val shuffledId = dependencies.head.asInstanceOf[ShuffleDependency[K, V]].shuffleId
    -    val ser = Serializer.getSerializer(serializer)
    -    SparkEnv.get.shuffleFetcher.fetch[P](shuffledId, split.index, context, ser)
    +    val dep = dependencies.head.asInstanceOf[ShuffleDependency[K, V, V]]
    --- End diff --
    
    Because there's no combining, the return type is (K, V), same as the input type (K, V).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13688708
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
    @@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
         part: Partitioner)
       extends RDD[P](prev.context, Nil) {
     
    -  private var serializer: Serializer = null
    +  private var serializer: Option[Serializer] = None
     
    +  /** Set a serializer for this RDD's shuffle, or null to use the default (spark.serializer) */
       def setSerializer(serializer: Serializer): ShuffledRDD[K, V, P] = {
    -    this.serializer = serializer
    +    this.serializer = Option(serializer)
    --- End diff --
    
    should this be `Some(serializer)`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1009#issuecomment-45429387
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15536/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13628183
  
    --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala ---
    @@ -0,0 +1,113 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.shuffle.hash
    +
    +import org.apache.spark.shuffle.{BaseShuffleHandle, ShuffleWriter}
    +import org.apache.spark.{Logging, MapOutputTracker, SparkEnv, TaskContext}
    +import org.apache.spark.storage.{BlockObjectWriter, ShuffleWriterGroup}
    --- End diff --
    
    minor: ShuffleWriterGroup not used


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1009#discussion_r13629622
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
    @@ -60,9 +61,10 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
       }
     
       override def compute(split: Partition, context: TaskContext): Iterator[P] = {
    -    val shuffledId = dependencies.head.asInstanceOf[ShuffleDependency[K, V]].shuffleId
    -    val ser = Serializer.getSerializer(serializer)
    -    SparkEnv.get.shuffleFetcher.fetch[P](shuffledId, split.index, context, ser)
    +    val dep = dependencies.head.asInstanceOf[ShuffleDependency[K, V, V]]
    --- End diff --
    
    Why is the combiner here also V?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---