You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by pwendell <gi...@git.apache.org> on 2014/02/28 07:45:32 UTC

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

GitHub user pwendell opened a pull request:

    https://github.com/apache/spark/pull/43

    SPARK-1145: Memory mapping with many small blocks can cause JVM allocation failures

    This includes some minor code clean-up as well. The main change is that small files are not memory mapped. There is a nicer way to write that code block using Scala's `Try` but to make it easy to back port and as simple as possible, I opted for the more explicit but less pretty format.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/pwendell/spark block-iter-logging

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/43.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #43
    
----
commit d238b885d6974283f3b96c26e26693074ff557e6
Author: Patrick Wendell <pw...@gmail.com>
Date:   2014-02-26T23:44:18Z

    Some logging and clean-up

commit 4e1514e666624d2285b190ff1419a9fae2ae929f
Author: Patrick Wendell <pw...@gmail.com>
Date:   2014-02-27T02:34:08Z

    Don't memory map for small files

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41138748
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14367/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10243564
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
    @@ -146,6 +146,12 @@ object BlockFetcherIterator {
         }
     
         protected def splitLocalRemoteBlocks(): ArrayBuffer[FetchRequest] = {
    +      // Make remote requests at most maxBytesInFlight / 5 in length; the reason to keep them
    +      // smaller than maxBytesInFlight is to allow multiple, parallel fetches from up to 5
    +      // nodes, rather than blocking on reading output from one node.
    +      val maxRequestSize = math.max(maxBytesInFlight / 5, 1L)
    --- End diff --
    
    Yeah I agree, note that the if statement below is `if (curRequestSize >= maxRequestSize)`. In general you might have some requests bigger than this of one map's output is just super large. This threshold is just used to batch together small map outputs so we can grab more than one at once.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41512199
  
    Looks good to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41504566
  
    Build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-36413557
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41512748
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14519/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10198184
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    --- End diff --
    
    Ah, so you're saying you might want to make the parameter higher than 8 KB? That makes sense, maybe it would be good to make it tunable. I thought you were saying that sometimes the patch here would create more IO load or use more memory.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10188851
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    --- End diff --
    
    We need to limit the number of random configuration parameters - just because something could possible have some minor performance gain if it's tweaked does not mean it should be exposed.
    
    This just needs to be set to equal a small number of page sizes. Since memory maps are aligned on page boundaries, then on average you expect to have 1/2 of a page of extra memory use. If the block size is 2 pages or less, then this overhead is really significant. Sure, maybe if we made it 3 or 5 instead of 2, but the range of useful values here is small and I don't think any user would ever want to tune this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41138747
  
    Build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-36413556
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41507066
  
     Build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-36326084
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-36327170
  
    One or more automated tests failed
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12929/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-36414230
  
    One or more automated tests failed
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12937/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r12031914
  
    --- Diff: dev/merge_spark_pr.py ---
    @@ -26,19 +26,33 @@
     
     import json
     import os
    +import re
    --- End diff --
    
    Gets displayed due to another merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10192379
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    --- End diff --
    
    There are performance implication between reading to vanilla buffer (note, this is not a direct buffer) and using memory mapped buffers - with added cost of some wastage for the 'last page'.
    In addition, you always leave the option open for zero copy enhancement in spark.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41512128
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10192236
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    --- End diff --
    
    Why would this have more impact than the current memory-mapping method? Both will lead to a bunch of seeks and filesystem operations, here when you read the blocks, and there when you try to send them over the network.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10226983
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
    @@ -146,6 +146,12 @@ object BlockFetcherIterator {
         }
     
         protected def splitLocalRemoteBlocks(): ArrayBuffer[FetchRequest] = {
    +      // Make remote requests at most maxBytesInFlight / 5 in length; the reason to keep them
    +      // smaller than maxBytesInFlight is to allow multiple, parallel fetches from up to 5
    +      // nodes, rather than blocking on reading output from one node.
    +      val maxRequestSize = math.max(maxBytesInFlight / 5, 1L)
    --- End diff --
    
    if you look at the code, this is used as a threshold where if the size gets larger than this it truncates the current request and creates a new one. I think the name was just wrong. Also even with my change it's *sort of* wrong because in fact the request size may be slightly larger than this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41512130
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10156309
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
    @@ -146,6 +146,12 @@ object BlockFetcherIterator {
         }
     
         protected def splitLocalRemoteBlocks(): ArrayBuffer[FetchRequest] = {
    +      // Make remote requests at most maxBytesInFlight / 5 in length; the reason to keep them
    +      // smaller than maxBytesInFlight is to allow multiple, parallel fetches from up to 5
    +      // nodes, rather than blocking on reading output from one node.
    +      val maxRequestSize = math.max(maxBytesInFlight / 5, 1L)
    +      logInfo("maxBytesInFlight: " + maxBytesInFlight + ", maxRequestSize: " + maxRequestSize)
    +
    --- End diff --
    
    Was always bothered by this ... not specifically related to this PR btw !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-36326074
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/43


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-36326073
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41244245
  
    Don't we want this in? I know a few cases on beefy machines jobs are failing because of this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-36413341
  
    Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10225873
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
    @@ -146,6 +146,12 @@ object BlockFetcherIterator {
         }
     
         protected def splitLocalRemoteBlocks(): ArrayBuffer[FetchRequest] = {
    +      // Make remote requests at most maxBytesInFlight / 5 in length; the reason to keep them
    +      // smaller than maxBytesInFlight is to allow multiple, parallel fetches from up to 5
    +      // nodes, rather than blocking on reading output from one node.
    +      val maxRequestSize = math.max(maxBytesInFlight / 5, 1L)
    --- End diff --
    
    Why'd you change this to be called maxRequestSize?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41136434
  
    Build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41507072
  
    Build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10156300
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
    @@ -146,6 +146,12 @@ object BlockFetcherIterator {
         }
     
         protected def splitLocalRemoteBlocks(): ArrayBuffer[FetchRequest] = {
    +      // Make remote requests at most maxBytesInFlight / 5 in length; the reason to keep them
    +      // smaller than maxBytesInFlight is to allow multiple, parallel fetches from up to 5
    +      // nodes, rather than blocking on reading output from one node.
    +      val maxRequestSize = math.max(maxBytesInFlight / 5, 1L)
    +      logInfo("maxBytesInFlight: " + maxBytesInFlight + ", maxRequestSize: " + maxRequestSize)
    +
    --- End diff --
    
    This assumes you have atleast 5 nodes - which need not be the general case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41136424
  
     Build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10156343
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
    @@ -146,6 +146,12 @@ object BlockFetcherIterator {
         }
     
         protected def splitLocalRemoteBlocks(): ArrayBuffer[FetchRequest] = {
    +      // Make remote requests at most maxBytesInFlight / 5 in length; the reason to keep them
    +      // smaller than maxBytesInFlight is to allow multiple, parallel fetches from up to 5
    +      // nodes, rather than blocking on reading output from one node.
    +      val maxRequestSize = math.max(maxBytesInFlight / 5, 1L)
    +      logInfo("maxBytesInFlight: " + maxBytesInFlight + ", maxRequestSize: " + maxRequestSize)
    +
    --- End diff --
    
    Btw, any reason to move this up ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r12031899
  
    --- Diff: make-distribution.sh ---
    @@ -43,7 +43,7 @@
     FWDIR="$(cd `dirname $0`; pwd)"
     DISTDIR="$FWDIR/dist"
     
    -VERSION=$(mvn help:evaluate -Dexpression=project.version |grep -v "INFO")
    +VERSION=$(mvn help:evaluate -Dexpression=project.version | grep -v "INFO" | tail -n 1)
    --- End diff --
    
    This is a github issue... not part of this patch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10226896
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
    @@ -146,6 +146,12 @@ object BlockFetcherIterator {
         }
     
         protected def splitLocalRemoteBlocks(): ArrayBuffer[FetchRequest] = {
    +      // Make remote requests at most maxBytesInFlight / 5 in length; the reason to keep them
    +      // smaller than maxBytesInFlight is to allow multiple, parallel fetches from up to 5
    +      // nodes, rather than blocking on reading output from one node.
    +      val maxRequestSize = math.max(maxBytesInFlight / 5, 1L)
    --- End diff --
    
    because it's the maximum allowable request size and not the minimum.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10174657
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    --- End diff --
    
    somewhat arbitrary, but related to the page size. If the segment is smaller than a couple of pages the fact that memory mapping aligns on page boundaries means a lot of wasted memory.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10174727
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
    @@ -146,6 +146,12 @@ object BlockFetcherIterator {
         }
     
         protected def splitLocalRemoteBlocks(): ArrayBuffer[FetchRequest] = {
    +      // Make remote requests at most maxBytesInFlight / 5 in length; the reason to keep them
    +      // smaller than maxBytesInFlight is to allow multiple, parallel fetches from up to 5
    +      // nodes, rather than blocking on reading output from one node.
    +      val maxRequestSize = math.max(maxBytesInFlight / 5, 1L)
    +      logInfo("maxBytesInFlight: " + maxBytesInFlight + ", maxRequestSize: " + maxRequestSize)
    +
    --- End diff --
    
    The only change here is to move this up. The reason is that there is no reason to calculate and log this in the inner loop, because its the same for all iterations of that loop. Logging it in the inner loop also means the same message is logged a bunch of times.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10200502
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    --- End diff --
    
    Actually both : higher or lower (turn it off with 0 for example).
    I am really not sure how this will interact with some of our experiments


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41508305
  
    @mateiz I think this addresses all your comments. Let me know!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10188872
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    --- End diff --
    
    Another way to think about it is this. This code path is really just to protect users who are doing something that already leads to very poor performance - shuffles where they use way too many partitions and the files are really tiny. The best thing a user can do there is to coalesce the number of mappers or reducers to get much better peformance. If they can figure that out, there is no reason to ever tune this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10225217
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    +        val buf = ByteBuffer.allocate(segment.length.toInt)
    +        try {
    +          channel.read(buf, segment.offset)
    +        }
    +        finally {
    +          channel.close()
    +        }
    +        Some(buf)
    +      } else {
    +        val buf = try {
    +          channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    +        } finally {
    +          channel.close()
    +        }
    +        Some(buf)
    --- End diff --
    
    nit: can you keep the finally outside of the if statement? just makes the code a little more concise and I think easier to read.  i.e.,
    
    val buf = try {
      if (blah)
      else (blah)
    } finally {
      channel.close()
    }


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10189405
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    --- End diff --
    
    I would rather have a hidden config param which can be tuned if required (however rarely), as opposed to have magic values which actually have no experimental basis.
    The impact of this PR is not just in memory usage, but also in the IO characteristics of spark.
    
    In particular, this is going to have significant performance impact if there are a large number of small files : which are fairly common - and in a lot of times, like in our experiments, aggressively aimed towards for the performance characterstics we are shooting for.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-36414229
  
    Merged build finished.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41513009
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41503665
  
     Build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10227330
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
    @@ -146,6 +146,12 @@ object BlockFetcherIterator {
         }
     
         protected def splitLocalRemoteBlocks(): ArrayBuffer[FetchRequest] = {
    +      // Make remote requests at most maxBytesInFlight / 5 in length; the reason to keep them
    +      // smaller than maxBytesInFlight is to allow multiple, parallel fetches from up to 5
    +      // nodes, rather than blocking on reading output from one node.
    +      val maxRequestSize = math.max(maxBytesInFlight / 5, 1L)
    --- End diff --
    
    Should we call this "targetRequestSize" or something?  I think min is more correct than max -- because it always waits until requests are at least this big before truncating them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10156359
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    --- End diff --
    
    How was this 8k limit arrived at ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-36327168
  
    Merged build finished.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r10180027
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
    @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: DiskBlockManage
       override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
         val segment = diskManager.getBlockLocation(blockId)
         val channel = new RandomAccessFile(segment.file, "r").getChannel()
    -    val buffer = try {
    -      channel.map(MapMode.READ_ONLY, segment.offset, segment.length)
    -    } finally {
    -      channel.close()
    +
    +    val buffer =
    +      // For small files, directly read rather than memory map
    +      if (segment.length < 2 * 4096) {
    --- End diff --
    
    I was hoping it was obtained in a more principled manner ... ah well
    in which case, atleast make it configurable - dont think anyone is going to muck around with it; but you never know.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41513005
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41503669
  
    Build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41512747
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41507906
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14518/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41504567
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14517/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:

    https://github.com/apache/spark/pull/43#discussion_r12030538
  
    --- Diff: docs/configuration.md ---
    @@ -123,6 +123,15 @@ Apart from these, the following properties are also available, and may be useful
       </td>
     </tr>
     <tr>
    +  <td>spark.storage.memoryMapThreshold</td>
    +  <td>2 * 4096</td>
    --- End diff --
    
    Probably just want to write 8192 here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41507905
  
    Build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41513991
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14520/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/43#issuecomment-41513990
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---