You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by eyalfa <gi...@git.apache.org> on 2017/08/05 21:21:52 UTC

[GitHub] spark pull request #18855: [Spark 3151][Block Manager] DiskStore.getBytes fa...

GitHub user eyalfa opened a pull request:

    https://github.com/apache/spark/pull/18855

    [Spark 3151][Block Manager] DiskStore.getBytes fails for files larger than 2GB

    ## What changes were proposed in this pull request?
    introduced `DiskBlockData`, a new implementation of `BlockData` representing a whole file.
    this is somehow related to [SPARK-6236](https://issues.apache.org/jira/browse/SPARK-6236) as well
    
    This class follows the implementation of `EncryptedBlockData` just without the encryption. hence:
    * it uses FileOutputStream (todo: encrypted version actually uses `Channels.newInputStream`, not sure if it's the right choice for this)
    * `toNetty` is implemented in terms of `io.netty.channel.DefaultFileRegion#DefaultFileRegion`
    * `toByteBuffer` fails for files larger than 2GB (same behavior of the original code, just postponed a bit), it also respects the same configuration keys defined by the original code to choose between memory mapping and simple file read.
    (Please fill in changes proposed in this fix)
    
    ## How was this patch tested?
    added test to DiskStoreSuite and MemoryManagerSuite

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/eyalfa/spark SPARK-3151

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18855.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18855
    
----
commit fc3f1d78e14a30dd2f71fc65ec59a2def5c1a0d4
Author: Eyal Farago <ey...@nrgene.com>
Date:   2017-07-05T13:20:16Z

    SPARK-6235__take1: introduce a failing test.

commit 84687380026a6a3bcded27be517094d3f690c3bb
Author: Eyal Farago <ey...@nrgene.com>
Date:   2017-07-30T20:06:05Z

    SPARK-6235__add_failing_tests: add failing tests for block manager suite.

commit 15804497a477b8f97c08adfad5f0519504dc82f2
Author: Eyal Farago <ey...@nrgene.com>
Date:   2017-08-01T17:34:26Z

    SPARK-6235__add_failing_tests: introduce a new BlockData implementation to represent a disk backed block data.

commit c5028f50698c4fe48a06f5dd683dbee42f7e6b2b
Author: Eyal Farago <ey...@nrgene.com>
Date:   2017-08-05T19:57:41Z

    SPARK-6235__add_failing_tests: styling

commit 908c7860688534d0bb77bcbebbd2e006a161fb74
Author: Eyal Farago <ey...@nrgene.com>
Date:   2017-08-05T19:58:52Z

    SPARK-6235__add_failing_tests: adapt DiskStoreSuite to the modifications in the tested class.

commit 67f4259ca16c3ca7c904c9ccc5de9acbc25d2271
Author: Eyal Farago <ey...@nrgene.com>
Date:   2017-08-05T20:57:58Z

    SPARK-6235__add_failing_tests: try to reduce actual memory footprint of the >2gb tests.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80728/testReport)** for PR 18855 at commit [`732073c`](https://github.com/apache/spark/commit/732073c5c73d4c12cc1059314c25f1ae94fc4469).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131890077
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel : StorageLevel) {
    +    store = makeBlockManager(6L * 1024 * 1024 * 1024, "exec1")
    +    def mkBlobs() = {
    +      val rng = new java.util.Random(42)
    +      val buff = new Array[Byte](1024 * 1024)
    +      rng.nextBytes(buff)
    +      Iterator.fill(2 * 1024 + 1) {
    +        buff
    +      }
    +    }
    +    val res1 = store.getOrElseUpdate(
    +      RDDBlockId(42, 0),
    +      storageLevel,
    +      implicitly[ClassTag[Array[Byte]]],
    +      mkBlobs _
    +    )
    +    withClue(res1) {
    +      assert(res1.isLeft)
    +      assert(res1.left.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    --- End diff --
    
    just `a === b`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80716/testReport)** for PR 18855 at commit [`732073c`](https://github.com/apache/spark/commit/732073c5c73d4c12cc1059314c25f1ae94fc4469).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131912438
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel : StorageLevel) {
    +    store = makeBlockManager(6L * 1024 * 1024 * 1024, "exec1")
    +    def mkBlobs() = {
    +      val rng = new java.util.Random(42)
    +      val buff = new Array[Byte](1024 * 1024)
    +      rng.nextBytes(buff)
    +      Iterator.fill(2 * 1024 + 1) {
    +        buff
    +      }
    +    }
    +    val res1 = store.getOrElseUpdate(
    +      RDDBlockId(42, 0),
    +      storageLevel,
    +      implicitly[ClassTag[Array[Byte]]],
    +      mkBlobs _
    +    )
    +    withClue(res1) {
    --- End diff --
    
    I think it'd print an Either where left side is a case class with members: iterator (prints as empty/non empty iterator), an enum and number of bytes. 
    right side is an iterator, again this'd print an empty/not-empty iterator.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80716/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fails for...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Looks reasonable, can you explain which end-to-end cases this patch fixed? and also which end-to-end cases remain problematic.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80534 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80534/testReport)** for PR 18855 at commit [`6cbe8d0`](https://github.com/apache/spark/commit/6cbe8d07d001b04e67c77e191bef2bc6cce6b0ea).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    since this PR only focus on `DiskStore`, shall we remove the new tests in `BlockManagerSuite`? Seems the OOM only happens in `BlockManagerSuite`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133384946
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +149,64 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    minMemoryMapBytes : Long,
    +    maxMemoryMapBytes : Long,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, maxMemoryMapBytes)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    +    // I chose to leave to original error message here
    +    // since users are unfamiliar with the configureation key
    +    // controling maxMemoryMapBytes for tests
    +    require(blockSize < maxMemoryMapBytes,
    +      s"can't create a byte buffer of size $blockSize" +
    +      s" since it exceeds Int.MaxValue ${Int.MaxValue}.")
    --- End diff --
    
    `since it exceeds $maxMemoryMapBytes` is more accurate.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133382511
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -47,6 +47,8 @@ private[spark] class DiskStore(
         securityManager: SecurityManager) extends Logging {
     
       private val minMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
    +  private val maxMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapLimitForTests",
    +    s"${Int.MaxValue}b")
    --- End diff --
    
    nit: just `Int.MaxValue.toString`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    @cloud-fan , can you please give the 'ok to test'?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r132964212
  
    --- Diff: project/SparkBuild.scala ---
    @@ -790,7 +790,7 @@ object TestSettings {
         javaOptions in Test ++= System.getProperties.asScala.filter(_._1.startsWith("spark"))
           .map { case (k,v) => s"-D$k=$v" }.toSeq,
         javaOptions in Test += "-ea",
    -    javaOptions in Test ++= "-Xmx3g -Xss4096k"
    +    javaOptions in Test ++= "-Xmx6g -Xss4096k"
    --- End diff --
    
    I'm a little worried about this change. Since the change to `BlockManagerSuite` is not very related to this PR, can we revert and revisit it in follow-up PR? Then we can unblock this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r132978316
  
    --- Diff: project/SparkBuild.scala ---
    @@ -790,7 +790,7 @@ object TestSettings {
         javaOptions in Test ++= System.getProperties.asScala.filter(_._1.startsWith("spark"))
           .map { case (k,v) => s"-D$k=$v" }.toSeq,
         javaOptions in Test += "-ea",
    -    javaOptions in Test ++= "-Xmx3g -Xss4096k"
    +    javaOptions in Test ++= "-Xmx6g -Xss4096k"
    --- End diff --
    
    @cloud-fan , let's wait few hours and see what the other guys CCed for this (the last ones to edit the build) have to say about this. if they are also worried or do not comment I'll revert this.
    
    I must say I'm reluctant to revert these tests as I personally believe that lack of such tests contributed to spark's 2GB issues, including this one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80397/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133317789
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
    @@ -92,6 +92,45 @@ class DiskStoreSuite extends SparkFunSuite {
         assert(diskStore.getSize(blockId) === 0L)
       }
     
    +  test("blocks larger than 2gb") {
    +    val conf = new SparkConf()
    +      .set("spark.storage.memoryMapLimitForTests", "10k" )
    +    val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop = true)
    +    val diskStore = new DiskStore(conf, diskBlockManager, new SecurityManager(conf))
    +
    +
    +    val blockId = BlockId("rdd_1_2")
    +    diskStore.put(blockId) { chan =>
    +      val arr = new Array[Byte](1024)
    +      for {
    +        _ <- 0 until 20
    +      } {
    +        val buf = ByteBuffer.wrap(arr)
    +        while (buf.hasRemaining()) {
    +          chan.write(buf)
    +        }
    +      }
    +    }
    +
    +    val blockData = diskStore.getBytes(blockId)
    +    assert(blockData.size == 20 * 1024)
    +
    +    val chunkedByteBuffer = blockData.toChunkedByteBuffer(ByteBuffer.allocate)
    +    val chunks = chunkedByteBuffer.chunks
    +    assert(chunks.size === 2)
    +    for( chunk <- chunks ) {
    --- End diff --
    
    nit: `for (chunk...) {`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r132982799
  
    --- Diff: project/SparkBuild.scala ---
    @@ -790,7 +790,7 @@ object TestSettings {
         javaOptions in Test ++= System.getProperties.asScala.filter(_._1.startsWith("spark"))
           .map { case (k,v) => s"-D$k=$v" }.toSeq,
         javaOptions in Test += "-ea",
    -    javaOptions in Test ++= "-Xmx3g -Xss4096k"
    +    javaOptions in Test ++= "-Xmx6g -Xss4096k"
    --- End diff --
    
    I am +1 for separating it if this can be. Let's get some changes we are sure of into the code base first.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131709108
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +147,62 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    conf: SparkConf,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  private val minMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, Int.MaxValue)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    --- End diff --
    
    we will still hit the 2g limitation here, I'm wondering which end-to-end use cases are affected by it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133383513
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +149,64 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    minMemoryMapBytes : Long,
    --- End diff --
    
    nit: no space before `"`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131549141
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +147,62 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    conf: SparkConf,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  private val minMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, Int.MaxValue)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    +    require( size < Int.MaxValue
    --- End diff --
    
    Sorry for confusing you. I mean the last line in your comment. I know the `size` method has the same value as `blockSize` since I saw the `size` method.
    Other places in `toByteBuffer` uses `blockSize` instead of `size`. For ease of code reading, it would be good to use `blockSize` instead of `size` here.
    What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fails for...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Yea, please refer http://apache-spark-developers-list.1001551.n3.nabble.com/Some-PRs-not-automatically-linked-to-JIRAs-td22067.html Looks some problems related with it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Funny enough, that's the approach I've chosen.
    
    On Aug 15, 2017 19:17, "Marcelo Vanzin" <no...@github.com> wrote:
    
    > *@vanzin* commented on this pull request.
    > ------------------------------
    >
    > In core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala
    > <https://github.com/apache/spark/pull/18855#discussion_r133232857>:
    >
    > > @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
    >        super.fetchBlockSync(host, port, execId, blockId)
    >      }
    >    }
    > +
    > +  def testGetOrElseUpdateForLargeBlock(storageLevel: StorageLevel) {
    >
    > It would be probably easier to propagate the chunk size as a SparkConf
    > entry that is not documented. But up to you guys.
    >
    > —
    > You are receiving this because you authored the thread.
    > Reply to this email directly, view it on GitHub
    > <https://github.com/apache/spark/pull/18855#discussion_r133232857>, or mute
    > the thread
    > <https://github.com/notifications/unsubscribe-auth/ABFFOZXLge3svlsrRWDJnt_xImYb6cTjks5sYcSVgaJpZM4OukZQ>
    > .
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133317809
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
    @@ -92,6 +92,45 @@ class DiskStoreSuite extends SparkFunSuite {
         assert(diskStore.getSize(blockId) === 0L)
       }
     
    +  test("blocks larger than 2gb") {
    +    val conf = new SparkConf()
    +      .set("spark.storage.memoryMapLimitForTests", "10k" )
    +    val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop = true)
    +    val diskStore = new DiskStore(conf, diskBlockManager, new SecurityManager(conf))
    +
    --- End diff --
    
    nit: remove this empty line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131544351
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
    @@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite {
         assert(diskStore.getSize(blockId) === 0L)
       }
     
    +  test("blocks larger than 2gb") {
    +    val conf = new SparkConf()
    +    val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop = true)
    +    val diskStore = new DiskStore(conf, diskBlockManager, new SecurityManager(conf))
    +
    +    val mb = 1024*1024
    --- End diff --
    
    nit: `1024 * 1024`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [Spark 3151][Block Manager] DiskStore.getBytes fails for...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80397/testReport)** for PR 18855 at commit [`5a5c344`](https://github.com/apache/spark/commit/5a5c3448f87b4bc2dddec26310e952dd2eadb98b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131544320
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
    @@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite {
         assert(diskStore.getSize(blockId) === 0L)
       }
     
    +  test("blocks larger than 2gb") {
    +    val conf = new SparkConf()
    +    val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop = true)
    +    val diskStore = new DiskStore(conf, diskBlockManager, new SecurityManager(conf))
    +
    +    val mb = 1024*1024
    +    val gb = 1024L*mb
    +
    +    val blockId = BlockId("rdd_1_2")
    +    diskStore.put(blockId) { chan =>
    +      val arr = new Array[Byte](mb)
    +      for{
    --- End diff --
    
    nit: `for (`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133141988
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel: StorageLevel) {
    --- End diff --
    
    @cloud-fan  I know, 
    it even gets worse when using the `===` operator.
    
    I'm currently exploring the second direction pointed by @vanzin , introducing a test-only configuration key to configure the max page size


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131890565
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel : StorageLevel) {
    +    store = makeBlockManager(6L * 1024 * 1024 * 1024, "exec1")
    +    def mkBlobs() = {
    +      val rng = new java.util.Random(42)
    +      val buff = new Array[Byte](1024 * 1024)
    +      rng.nextBytes(buff)
    +      Iterator.fill(2 * 1024 + 1) {
    +        buff
    +      }
    +    }
    +    val res1 = store.getOrElseUpdate(
    +      RDDBlockId(42, 0),
    +      storageLevel,
    +      implicitly[ClassTag[Array[Byte]]],
    +      mkBlobs _
    +    )
    +    withClue(res1) {
    +      assert(res1.isLeft)
    +      assert(res1.left.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    +          a != null &&
    +            b != null &&
    +            a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +      })
    +    }
    +    val getResult = store.get(RDDBlockId(42, 0))
    +    withClue(getResult) {
    +      assert(getResult.isDefined)
    +      assert(getResult.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    +          a != null &&
    +            b != null &&
    +            a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +      })
    +    }
    +    val getBlockRes = store.getBlockData(RDDBlockId(42, 0))
    +    withClue(getBlockRes) {
    +      try {
    +        assert(getBlockRes.size() >= 2 * 1024 * 1024 * 1024)
    +        Utils.tryWithResource(getBlockRes.createInputStream()) { inpStrm =>
    +          val iter = store
    +            .serializerManager
    +            .dataDeserializeStream(RDDBlockId(42, 0)
    +              , inpStrm)(implicitly[ClassTag[Array[Byte]]])
    +          assert(iter.zipAll(mkBlobs(), null, null).forall {
    +            case (a, b) =>
    +              a != null &&
    +                b != null &&
    +                a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +          })
    +        }
    +      } finally {
    +        getBlockRes.release()
    +      }
    +    }
    +  }
    +
    +  test("getOrElseUpdate > 2gb, storage level = disk only") {
    --- End diff --
    
    oh we already have, then why we have these tests?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133135659
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel: StorageLevel) {
    --- End diff --
    
    @vanzin , 
    I've measured, test cases times range from 7-25 seconds on my laptop.
    point well taken :sunglasses: 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Agreed, though I can't really understand it...
    Ran SBT locally with --mem 12000 and provided the magical jvm flag that
    prints jvm provided CLI args, it seems SBT is indeed running with 12gb and
    the test is still failing - even when running this test only.
    
    Does SBT fork before executing tests? If so how does it configure the jvm
    running the tests?
    
    On Aug 9, 2017 08:29, "Wenchen Fan" <no...@github.com> wrote:
    
    > since this PR only focus on DiskStore, shall we remove the new tests in
    > BlockManagerSuite? Seems the OOM only happens in BlockManagerSuite
    >
    > —
    > You are receiving this because you authored the thread.
    > Reply to this email directly, view it on GitHub
    > <https://github.com/apache/spark/pull/18855#issuecomment-321156406>, or mute
    > the thread
    > <https://github.com/notifications/unsubscribe-auth/ABFFOZ8ahiQXd_dce4M6UG-Wib8ebaBTks5sWUOmgaJpZM4OukZQ>
    > .
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131543591
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +147,62 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    conf: SparkConf,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  private val minMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, Int.MaxValue)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    +    require( size < Int.MaxValue
    --- End diff --
    
    @kiszk , not sure I'm following your comment.
    this requirement results with an explicit errors when one tries to obtain a ByteBuffer larger than java.nio's limitations.
    original code used to fail in line 115 when calling `ByteBuffer.allocate` with block size larger than 2GB, newer code fails explicitly in this cae.
    
    ...or do you mean refer the `blockSize` val rather than the `size` method? can't really see a difference in that case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80716/testReport)** for PR 18855 at commit [`732073c`](https://github.com/apache/spark/commit/732073c5c73d4c12cc1059314c25f1ae94fc4469).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131912886
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel : StorageLevel) {
    +    store = makeBlockManager(6L * 1024 * 1024 * 1024, "exec1")
    +    def mkBlobs() = {
    +      val rng = new java.util.Random(42)
    +      val buff = new Array[Byte](1024 * 1024)
    +      rng.nextBytes(buff)
    +      Iterator.fill(2 * 1024 + 1) {
    +        buff
    +      }
    +    }
    +    val res1 = store.getOrElseUpdate(
    +      RDDBlockId(42, 0),
    +      storageLevel,
    +      implicitly[ClassTag[Array[Byte]]],
    +      mkBlobs _
    +    )
    +    withClue(res1) {
    +      assert(res1.isLeft)
    +      assert(res1.left.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    --- End diff --
    
    can't compare Arrays, you get identity equality which is usually not what you want. hence the `.seq` that forces it to be wrapped with a Seq


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133384047
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +149,64 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    minMemoryMapBytes : Long,
    +    maxMemoryMapBytes : Long,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, maxMemoryMapBytes)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    +    // I chose to leave to original error message here
    +    // since users are unfamiliar with the configureation key
    +    // controling maxMemoryMapBytes for tests
    +    require(blockSize < maxMemoryMapBytes,
    --- End diff --
    
    why we need this? I think we can see the original error message if we don't have this check and go to the memory map code path.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133232857
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel: StorageLevel) {
    --- End diff --
    
    It would be probably easier to propagate the chunk size as a `SparkConf` entry that is not documented. But up to you guys.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133166421
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel: StorageLevel) {
    --- End diff --
    
    shall we do them together in a follow-up PR? I think the test case in `DiskStoreSuite` is enough.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131890389
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel : StorageLevel) {
    +    store = makeBlockManager(6L * 1024 * 1024 * 1024, "exec1")
    +    def mkBlobs() = {
    +      val rng = new java.util.Random(42)
    +      val buff = new Array[Byte](1024 * 1024)
    +      rng.nextBytes(buff)
    +      Iterator.fill(2 * 1024 + 1) {
    +        buff
    +      }
    +    }
    +    val res1 = store.getOrElseUpdate(
    +      RDDBlockId(42, 0),
    +      storageLevel,
    +      implicitly[ClassTag[Array[Byte]]],
    +      mkBlobs _
    +    )
    +    withClue(res1) {
    +      assert(res1.isLeft)
    +      assert(res1.left.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    +          a != null &&
    +            b != null &&
    +            a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +      })
    +    }
    +    val getResult = store.get(RDDBlockId(42, 0))
    +    withClue(getResult) {
    +      assert(getResult.isDefined)
    +      assert(getResult.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    +          a != null &&
    +            b != null &&
    +            a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +      })
    +    }
    +    val getBlockRes = store.getBlockData(RDDBlockId(42, 0))
    +    withClue(getBlockRes) {
    +      try {
    +        assert(getBlockRes.size() >= 2 * 1024 * 1024 * 1024)
    +        Utils.tryWithResource(getBlockRes.createInputStream()) { inpStrm =>
    +          val iter = store
    +            .serializerManager
    +            .dataDeserializeStream(RDDBlockId(42, 0)
    +              , inpStrm)(implicitly[ClassTag[Array[Byte]]])
    +          assert(iter.zipAll(mkBlobs(), null, null).forall {
    +            case (a, b) =>
    +              a != null &&
    +                b != null &&
    +                a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +          })
    +        }
    +      } finally {
    +        getBlockRes.release()
    +      }
    +    }
    +  }
    +
    +  test("getOrElseUpdate > 2gb, storage level = disk only") {
    --- End diff --
    
    shall we just write a test in `DiskStoreSuite`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/18855


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133180495
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel: StorageLevel) {
    --- End diff --
    
    yes, currently working on:
    1.  parameterizing `DiskStore` and `DiskStoreSuite`
    2. revert the tests in `BlockManagerSuite` 
    3. revert the 6gb  change in sbt


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80534/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133144224
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel: StorageLevel) {
    --- End diff --
    
    @cloud-fan , @vanzin ,
    taking the 'parameterized approach', I'd remove most of the tests from `BlockManagerSuite` as they'd require propagating this parameter to too many subsystems.
    so, I'm going to modify `DiskStore` and `DiskStoreSuite` to use such a parameter, I'm not sure about leaving a test-case in `BlockManagerSuite` that tests `DISK_ONLY` persistence, what do you guys think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131890999
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
    @@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite {
         assert(diskStore.getSize(blockId) === 0L)
       }
     
    +  test("blocks larger than 2gb") {
    +    val conf = new SparkConf()
    +    val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop = true)
    +    val diskStore = new DiskStore(conf, diskBlockManager, new SecurityManager(conf))
    +
    +    val mb = 1024 * 1024
    +    val gb = 1024L * mb
    +
    +    val blockId = BlockId("rdd_1_2")
    +    diskStore.put(blockId) { chan =>
    +      val arr = new Array[Byte](mb)
    +      for {
    +        _ <- 0 until 2048
    +      } {
    +        val buf = ByteBuffer.wrap(arr)
    +        while (buf.hasRemaining()) {
    +          chan.write(buf)
    +        }
    +      }
    +    }
    +
    +    val blockData = diskStore.getBytes(blockId)
    +    assert(blockData.size == 2 * gb)
    --- End diff --
    
    test with 3gb to be more explicit that it's larger than 2gb?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80728/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133009803
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel: StorageLevel) {
    --- End diff --
    
    Have you measured how long these tests take? I've seen this tried before in other changes related to 2g limits, and this kind of test was always ridiculously slow.
    
    You can avoid this kind of test by making the chunk size configurable, e.g. in this line you're adding above:
    
        val chunkSize = math.min(remaining, Int.MaxValue)
    
    Then your test can run fast and not use a lot of memory. You just need to add extra checks that the data is being chunked properly, instead of relying on the JVM not throwing errors at you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131713763
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
    @@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite {
         assert(diskStore.getSize(blockId) === 0L)
       }
     
    +  test("blocks larger than 2gb") {
    +    val conf = new SparkConf()
    +    val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop = true)
    +    val diskStore = new DiskStore(conf, diskBlockManager, new SecurityManager(conf))
    +
    +    val mb = 1024 * 1024
    +    val gb = 1024L * mb
    +
    +    val blockId = BlockId("rdd_1_2")
    +    diskStore.put(blockId) { chan =>
    +      val arr = new Array[Byte](mb)
    +      for {
    +        _ <- 0 until 2048
    +      } {
    +        val buf = ByteBuffer.wrap(arr)
    +        while (buf.hasRemaining()) {
    +          chan.write(buf)
    +        }
    +      }
    +    }
    +
    +    val blockData = diskStore.getBytes(blockId)
    --- End diff --
    
    @kiszk, this is the test case I was referring to.
    I actually introduced it prior to actually (hopefully) fixing the bug in `DiskStore.getBytes`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    thanks, merging to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80697/testReport)** for PR 18855 at commit [`8be899f`](https://github.com/apache/spark/commit/8be899f93922ed3181fd67be0782bff8f47c6c37).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80747/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131889407
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +147,62 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    conf: SparkConf,
    --- End diff --
    
    we can pass in `minMemoryMapBytes` directly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131914883
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
    @@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite {
         assert(diskStore.getSize(blockId) === 0L)
       }
     
    +  test("blocks larger than 2gb") {
    +    val conf = new SparkConf()
    +    val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop = true)
    +    val diskStore = new DiskStore(conf, diskBlockManager, new SecurityManager(conf))
    +
    +    val mb = 1024 * 1024
    +    val gb = 1024L * mb
    +
    +    val blockId = BlockId("rdd_1_2")
    +    diskStore.put(blockId) { chan =>
    +      val arr = new Array[Byte](mb)
    +      for {
    +        _ <- 0 until 2048
    +      } {
    +        val buf = ByteBuffer.wrap(arr)
    +        while (buf.hasRemaining()) {
    +          chan.write(buf)
    +        }
    +      }
    +    }
    +
    +    val blockData = diskStore.getBytes(blockId)
    +    assert(blockData.size == 2 * gb)
    --- End diff --
    
    possible, will fix.
    I guess I aimed for the lowest possible failing value


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80728/testReport)** for PR 18855 at commit [`732073c`](https://github.com/apache/spark/commit/732073c5c73d4c12cc1059314c25f1ae94fc4469).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [Spark 3151][Block Manager] DiskStore.getBytes fails for...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    @rxin, @JoshRosen , @cloud-fan ,
    you seem to be the last guys to touch this class, can you please review?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133318267
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
    @@ -92,6 +92,45 @@ class DiskStoreSuite extends SparkFunSuite {
         assert(diskStore.getSize(blockId) === 0L)
       }
     
    +  test("blocks larger than 2gb") {
    +    val conf = new SparkConf()
    +      .set("spark.storage.memoryMapLimitForTests", "10k" )
    +    val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop = true)
    +    val diskStore = new DiskStore(conf, diskBlockManager, new SecurityManager(conf))
    +
    +
    +    val blockId = BlockId("rdd_1_2")
    +    diskStore.put(blockId) { chan =>
    +      val arr = new Array[Byte](1024)
    +      for {
    +        _ <- 0 until 20
    +      } {
    +        val buf = ByteBuffer.wrap(arr)
    +        while (buf.hasRemaining()) {
    +          chan.write(buf)
    +        }
    +      }
    +    }
    +
    +    val blockData = diskStore.getBytes(blockId)
    +    assert(blockData.size == 20 * 1024)
    +
    +    val chunkedByteBuffer = blockData.toChunkedByteBuffer(ByteBuffer.allocate)
    +    val chunks = chunkedByteBuffer.chunks
    +    assert(chunks.size === 2)
    +    for( chunk <- chunks ) {
    +      assert(chunk.limit === 10 * 1024)
    +    }
    +
    +    val e = intercept[IllegalArgumentException]{
    +      blockData.toByteBuffer()
    +    }
    +
    +    assert(e.getMessage ==
    --- End diff --
    
    nit: `===`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131758308
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +147,62 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    conf: SparkConf,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  private val minMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, Int.MaxValue)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    --- End diff --
    
    @cloud-fan 
    it took me roughly 4 hours, but I looked both at the shuffle cod path and at `BlockManager.getRemoteBytes`:
    it seems the first is robust to large blocks by using Netty's stream capabilities,
    the later seems to be broken as it's not using the Netty's streaming capabilities and actually tries to copy the result buffer into a heap based buffer. I think this deserves its own JIRA/PR.
    I think these two places plus the external shuffle server cover most of the relevant use cases (aside from local caching which i believe this PR completes in terms of being 2GB  proof).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133136857
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel: StorageLevel) {
    --- End diff --
    
    7-25 seconds is really a long time for a unit test...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133385095
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +149,64 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    minMemoryMapBytes : Long,
    +    maxMemoryMapBytes : Long,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, maxMemoryMapBytes)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    +    // I chose to leave to original error message here
    +    // since users are unfamiliar with the configureation key
    +    // controling maxMemoryMapBytes for tests
    +    require(blockSize < maxMemoryMapBytes,
    +      s"can't create a byte buffer of size $blockSize" +
    +      s" since it exceeds Int.MaxValue ${Int.MaxValue}.")
    --- End diff --
    
    or call `Utils.bytesToString` to make it more readable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131889556
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel : StorageLevel) {
    --- End diff --
    
    nit: `storageLevel: StorageLevel`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131712834
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +147,62 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    conf: SparkConf,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  private val minMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, Int.MaxValue)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    --- End diff --
    
    indeed.
    I chose to postpone the failure from `DiskStroe.getBytes` to this place as I believe it introduces no regression while still allowing the more common 'streamin' like use-case.
    
    further more, I think this plays well with the comment about future deprecation of `org.apache.spark.network.buffer.ManagedBuffer#nioByteBuffer` which seems to be the main reason for `BlockData` exposing the `toByteBuffer` method.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80747/testReport)** for PR 18855 at commit [`d0c98a1`](https://github.com/apache/spark/commit/d0c98a14ed3d9b4d70129c52af186c2c86f93f41).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80747/testReport)** for PR 18855 at commit [`d0c98a1`](https://github.com/apache/spark/commit/d0c98a14ed3d9b4d70129c52af186c2c86f93f41).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131889921
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel : StorageLevel) {
    +    store = makeBlockManager(6L * 1024 * 1024 * 1024, "exec1")
    +    def mkBlobs() = {
    +      val rng = new java.util.Random(42)
    +      val buff = new Array[Byte](1024 * 1024)
    +      rng.nextBytes(buff)
    +      Iterator.fill(2 * 1024 + 1) {
    +        buff
    +      }
    +    }
    +    val res1 = store.getOrElseUpdate(
    +      RDDBlockId(42, 0),
    +      storageLevel,
    +      implicitly[ClassTag[Array[Byte]]],
    +      mkBlobs _
    +    )
    +    withClue(res1) {
    --- End diff --
    
    does `res1` have a reasonable string representation?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    LGTM except some minor comments, thanks for working on it!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133005835
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +147,62 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    conf: SparkConf,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  private val minMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, Int.MaxValue)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    +    require( blockSize < Int.MaxValue
    --- End diff --
    
    no space after `(`; comma should be on this line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80397/testReport)** for PR 18855 at commit [`5a5c344`](https://github.com/apache/spark/commit/5a5c3448f87b4bc2dddec26310e952dd2eadb98b).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80534/testReport)** for PR 18855 at commit [`6cbe8d0`](https://github.com/apache/spark/commit/6cbe8d07d001b04e67c77e191bef2bc6cce6b0ea).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133007010
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel : StorageLevel) {
    +    store = makeBlockManager(6L * 1024 * 1024 * 1024, "exec1")
    +    def mkBlobs() = {
    +      val rng = new java.util.Random(42)
    +      val buff = new Array[Byte](1024 * 1024)
    +      rng.nextBytes(buff)
    +      Iterator.fill(2 * 1024 + 1) {
    +        buff
    +      }
    +    }
    +    val res1 = store.getOrElseUpdate(
    +      RDDBlockId(42, 0),
    +      storageLevel,
    +      implicitly[ClassTag[Array[Byte]]],
    +      mkBlobs _
    +    )
    +    withClue(res1) {
    +      assert(res1.isLeft)
    +      assert(res1.left.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    --- End diff --
    
    Even if `===` does not work, you have `Arrays.equal`, which is null-safe.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131543345
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +147,62 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    conf: SparkConf,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  private val minMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, Int.MaxValue)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    +    require( size < Int.MaxValue
    +      , s"can't create a byte buffer of size $blockSize"
    +        + s" since it exceeds Int.MaxValue ${Int.MaxValue}.")
    +    Utils.tryWithResource(open()) { channel =>
    +      if (blockSize < minMemoryMapBytes) {
    +        // For small files, directly read rather than memory map.
    +        val buf = ByteBuffer.allocate(blockSize.toInt)
    +        JavaUtils.readFully(channel, buf)
    +        buf.flip()
    +        buf
    +      } else {
    +        channel.map(MapMode.READ_ONLY, 0, file.length)
    +      }
    +    }
    +  }
    +
    +  override def size: Long = blockSize
    +
    +  override def dispose(): Unit = {}
    +
    +    private def open() = new FileInputStream(file).getChannel
    --- End diff --
    
    nit: remove 2 spaces for better indent.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131925431
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel : StorageLevel) {
    +    store = makeBlockManager(6L * 1024 * 1024 * 1024, "exec1")
    +    def mkBlobs() = {
    +      val rng = new java.util.Random(42)
    +      val buff = new Array[Byte](1024 * 1024)
    +      rng.nextBytes(buff)
    +      Iterator.fill(2 * 1024 + 1) {
    +        buff
    +      }
    +    }
    +    val res1 = store.getOrElseUpdate(
    +      RDDBlockId(42, 0),
    +      storageLevel,
    +      implicitly[ClassTag[Array[Byte]]],
    +      mkBlobs _
    +    )
    +    withClue(res1) {
    +      assert(res1.isLeft)
    +      assert(res1.left.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    --- End diff --
    
    `===` is a helper method in scala test and should be able to compare arrays


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    @cloud-fan I think I found the sbt setting that controlled max heap size for forked tests, I've increased it from 3g to 6g.
    cc: @srowen, @vanzin and @a-roberts  you guys seem to be the last ones to update this area in sbt, please review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    @cloud-fan, any idea how much memory is allocated for running sbt? currently one of my newly introduced tests fails on OOM during kryo serialization...
    
    it's actually a bit weird as this is the test that stores it in memory deserialized objects, notice that the test doesn't really generate a true 2GB large dataset as the iterator returns the same array on every `next()` call, so basically the underlying buffer(actually buffers as it used chunked byte buffer) used by kryo is the first materialization of such a large piece of data.
    sbt execution of this suite seems to stop on the first error so I can't tell if the 'memory deserialized' version of this test would have passed.
    
    please advice, is it possible to increase sbt's process heap size? or should I remove/comment/ignore these tests?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131913940
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel : StorageLevel) {
    +    store = makeBlockManager(6L * 1024 * 1024 * 1024, "exec1")
    +    def mkBlobs() = {
    +      val rng = new java.util.Random(42)
    +      val buff = new Array[Byte](1024 * 1024)
    +      rng.nextBytes(buff)
    +      Iterator.fill(2 * 1024 + 1) {
    +        buff
    +      }
    +    }
    +    val res1 = store.getOrElseUpdate(
    +      RDDBlockId(42, 0),
    +      storageLevel,
    +      implicitly[ClassTag[Array[Byte]]],
    +      mkBlobs _
    +    )
    +    withClue(res1) {
    +      assert(res1.isLeft)
    +      assert(res1.left.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    +          a != null &&
    +            b != null &&
    +            a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +      })
    +    }
    +    val getResult = store.get(RDDBlockId(42, 0))
    +    withClue(getResult) {
    +      assert(getResult.isDefined)
    +      assert(getResult.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    +          a != null &&
    +            b != null &&
    +            a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +      })
    +    }
    +    val getBlockRes = store.getBlockData(RDDBlockId(42, 0))
    +    withClue(getBlockRes) {
    +      try {
    +        assert(getBlockRes.size() >= 2 * 1024 * 1024 * 1024)
    +        Utils.tryWithResource(getBlockRes.createInputStream()) { inpStrm =>
    +          val iter = store
    +            .serializerManager
    +            .dataDeserializeStream(RDDBlockId(42, 0)
    +              , inpStrm)(implicitly[ClassTag[Array[Byte]]])
    +          assert(iter.zipAll(mkBlobs(), null, null).forall {
    +            case (a, b) =>
    +              a != null &&
    +                b != null &&
    +                a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +          })
    +        }
    +      } finally {
    +        getBlockRes.release()
    +      }
    +    }
    +  }
    +
    +  test("getOrElseUpdate > 2gb, storage level = disk only") {
    --- End diff --
    
    these tests cover more than just the `DiskOnly` storage level, they were crafted when I had bigger ambitions of solving the entire 2GB issue :sunglasses: , that was before seeing some ~100 files pull requests being abandoned or rejected.
    aside, these tests also test the entire orchestration done by `BlockManager` when an `RDD` requests a cached partition, notice that these tests intentionally makes two calls to the BlockManager in order to simulate both code paths (cache-hit, cache-miss).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133384556
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +149,64 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    minMemoryMapBytes : Long,
    +    maxMemoryMapBytes : Long,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, maxMemoryMapBytes)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    +    // I chose to leave to original error message here
    +    // since users are unfamiliar with the configureation key
    +    // controling maxMemoryMapBytes for tests
    +    require(blockSize < maxMemoryMapBytes,
    --- End diff --
    
    oh this is to verify the bug fix.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131543602
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +147,62 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    conf: SparkConf,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  private val minMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, Int.MaxValue)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    +    require( size < Int.MaxValue
    +      , s"can't create a byte buffer of size $blockSize"
    +        + s" since it exceeds Int.MaxValue ${Int.MaxValue}.")
    +    Utils.tryWithResource(open()) { channel =>
    +      if (blockSize < minMemoryMapBytes) {
    +        // For small files, directly read rather than memory map.
    +        val buf = ByteBuffer.allocate(blockSize.toInt)
    +        JavaUtils.readFully(channel, buf)
    +        buf.flip()
    +        buf
    +      } else {
    +        channel.map(MapMode.READ_ONLY, 0, file.length)
    +      }
    +    }
    +  }
    +
    +  override def size: Long = blockSize
    +
    +  override def dispose(): Unit = {}
    +
    +    private def open() = new FileInputStream(file).getChannel
    --- End diff --
    
    will do :sunglasses: 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131544332
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala ---
    @@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite {
         assert(diskStore.getSize(blockId) === 0L)
       }
     
    +  test("blocks larger than 2gb") {
    +    val conf = new SparkConf()
    +    val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop = true)
    +    val diskStore = new DiskStore(conf, diskBlockManager, new SecurityManager(conf))
    +
    +    val mb = 1024*1024
    +    val gb = 1024L*mb
    +
    +    val blockId = BlockId("rdd_1_2")
    +    diskStore.put(blockId) { chan =>
    +      val arr = new Array[Byte](mb)
    +      for{
    +        _ <- 0 until 2048
    +      }{
    --- End diff --
    
    nit: `} {`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80697/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    @cloud-fan , @vanzin ,
    any idea what happened to this build? seem environment issue after a successful build (0 failed tests, 'Build step 'Execute shell' marked build as failure')
    can one of you kindly ask jenkins to retest?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fails for...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    @kiszk , fixed styling+readability according to your comments.
    BTW, any idea why JIRA didn't associate this PR with SPARK-3151?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r133007549
  
    --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
    @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfterE
           super.fetchBlockSync(host, port, execId, blockId)
         }
       }
    +
    +  def testGetOrElseUpdateForLargeBlock(storageLevel: StorageLevel) {
    +    store = makeBlockManager(6L * 1024 * 1024 * 1024, "exec1")
    +    def mkBlobs() = {
    +      val rng = new java.util.Random(42)
    +      val buff = new Array[Byte](1024 * 1024)
    +      rng.nextBytes(buff)
    +      Iterator.fill(2 * 1024 + 1) {
    +        buff
    +      }
    +    }
    +    val res1 = store.getOrElseUpdate(
    +      RDDBlockId(42, 0),
    +      storageLevel,
    +      implicitly[ClassTag[Array[Byte]]],
    +      mkBlobs _
    +    )
    +    withClue(res1) {
    +      assert(res1.isLeft)
    +      assert(res1.left.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    +          a != null &&
    +            b != null &&
    +            a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +      })
    +    }
    +    val getResult = store.get(RDDBlockId(42, 0))
    +    withClue(getResult) {
    +      assert(getResult.isDefined)
    +      assert(getResult.get.data.zipAll(mkBlobs(), null, null).forall {
    +        case (a, b) =>
    +          a != null &&
    +            b != null &&
    +            a.asInstanceOf[Array[Byte]].seq == b.asInstanceOf[Array[Byte]].seq
    +      })
    +    }
    +    val getBlockRes = store.getBlockData(RDDBlockId(42, 0))
    +    withClue(getBlockRes) {
    +      try {
    +        assert(getBlockRes.size() >= 2 * 1024 * 1024 * 1024)
    +        Utils.tryWithResource(getBlockRes.createInputStream()) { inpStrm =>
    +          val iter = store
    +            .serializerManager
    +            .dataDeserializeStream(RDDBlockId(42, 0)
    +              , inpStrm)(implicitly[ClassTag[Array[Byte]]])
    --- End diff --
    
    Comma goes in the previous line. `inpStrm` is kind of an ugly variable name; pick one: `is`, `in`, `inputStream`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fails for...

Posted by eyalfa <gi...@git.apache.org>.
Github user eyalfa commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    @HyukjinKwon , interesting reading but I couldn't find a concrete reason or solution to the issue.
    @cloud-fan, I've encountered this bug when working with a disk persisted RDD, it turned out some of our partitions exceeded the 2GB limit and failed on `DiskStore.getBytes`.
    
    since we're using a specific commercial distro I couldn't test this patch on my use case (we got rid of the large partitions anyway by tuning number of partitions and tweaking the DAG altogether :sunglasses: ), so I did the next best thing: reproduce the issue in a test case, please see test "blocks larger than 2gb" in `DiskStoreSuite` inthis PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    **[Test build #80697 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80697/testReport)** for PR 18855 at commit [`8be899f`](https://github.com/apache/spark/commit/8be899f93922ed3181fd67be0782bff8f47c6c37).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18855#discussion_r131543084
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -165,6 +147,62 @@ private[spark] class DiskStore(
     
     }
     
    +private class DiskBlockData(
    +    conf: SparkConf,
    +    file: File,
    +    blockSize: Long) extends BlockData {
    +
    +  private val minMemoryMapBytes = conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
    +
    +  override def toInputStream(): InputStream = new FileInputStream(file)
    +
    +  /**
    +  * Returns a Netty-friendly wrapper for the block's data.
    +  *
    +  * Please see `ManagedBuffer.convertToNetty()` for more details.
    +  */
    +  override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
    +
    +  override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer): ChunkedByteBuffer = {
    +    Utils.tryWithResource(open()) { channel =>
    +      var remaining = blockSize
    +      val chunks = new ListBuffer[ByteBuffer]()
    +      while (remaining > 0) {
    +        val chunkSize = math.min(remaining, Int.MaxValue)
    +        val chunk = allocator(chunkSize.toInt)
    +        remaining -= chunkSize
    +        JavaUtils.readFully(channel, chunk)
    +        chunk.flip()
    +        chunks += chunk
    +      }
    +      new ChunkedByteBuffer(chunks.toArray)
    +    }
    +  }
    +
    +  override def toByteBuffer(): ByteBuffer = {
    +    require( size < Int.MaxValue
    --- End diff --
    
    Is it better to check `blockSize`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fails for...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/18855
  
    Sounds good to me except `BlockManagerSuite.scala`. I am not familiar whether this new test checks new code well or not.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org