You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by peter-toth <gi...@git.apache.org> on 2018/10/01 19:35:49 UTC

[GitHub] spark pull request #22603: SPARK-25062: clean up BlockLocations in InMemoryF...

GitHub user peter-toth opened a pull request:

    https://github.com/apache/spark/pull/22603

    SPARK-25062: clean up BlockLocations in InMemoryFileIndex

    ## What changes were proposed in this pull request?
    
    `InMemoryFileIndex` caches `FileStatus` objects to paths. Each `FileStatus` object can contain several `BlockLocations`. Depending on the parallel discovery threshold (`spark.sql.sources.parallelPartitionDiscovery.threshold`) the file listing can happen on the driver or on the executors. If the listing happens on the executors the block location objects are converted to simple `BlockLocation` objects to ensure serialization requirements. If it happens on the driver then there is no conversion and depending on the file system a `BlockLocation` object can be a subclass like `HdfsBlockLocation` and consume more memory. This PR adds the conversion to the latter case and decreases memory consumption.
    
    ## How was this patch tested?
    
    Added unit test.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/peter-toth/spark SPARK-25062

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22603.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22603
    
----
commit 5735dba9a43c336843ccc531831105d6c23b4586
Author: Peter Toth <pe...@...>
Date:   2018-09-30T13:01:36Z

    SPARK-25062: clean up BlockLocations in InMemoryFileIndex

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22603: [SPARK-25062][SQL] Clean up BlockLocations in InMemoryFi...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22603
  
    Could you review this, @cloud-fan , @gatorsmile , @HyukjinKwon ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22603: [SPARK-25062][SQL] Clean up BlockLocations in InMemoryFi...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22603
  
    @peter-toth . What is your Apache JIRA user id? I need to assign you to the resolved SPARK-25062, but I cannot find your id and user name `Peter Toth`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22603: [SPARK-25062][SQL] Clean up BlockLocations in InMemoryFi...

Posted by peter-toth <gi...@git.apache.org>.

Github user peter-toth commented on the issue:

    https://github.com/apache/spark/pull/22603
  
    Thanks @cloud-fan for the review. I've fixed your findings.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22603: [SPARK-25062][SQL] Clean up BlockLocations in InMemoryFi...

Posted by peter-toth <gi...@git.apache.org>.

Github user peter-toth commented on the issue:

    https://github.com/apache/spark/pull/22603
  
    Thanks @dongjoon-hyun for the review. I've fixed your findings.  


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22603: [SPARK-25062][SQL] Clean up BlockLocations in InMemoryFi...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22603
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96856/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22603: [SPARK-25062][SQL] Clean up BlockLocations in InMemoryFi...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22603
  
    **[Test build #96932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96932/testReport)** for PR 22603 at commit [`7b0bc56`](https://github.com/apache/spark/commit/7b0bc568baa69e74fc98ac34f13b6c54b3d4f7a7).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #22603: [SPARK-25062][SQL] Clean up BlockLocations in InM...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22603#discussion_r222410955
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala ---
    @@ -248,6 +248,25 @@ class FileIndexSuite extends SharedSQLContext {
           assert(spark.read.parquet(path.getAbsolutePath).schema.exists(_.name == colToUnescape))
         }
       }
    +
    +  test("SPARK-25062 - InMemoryCache stores only simple BlockLocations") {
    +    withSQLConf("fs.file.impl" -> classOf[SpecialBlockLocationFileSystem].getName) {
    +      withTempDir { dir =>
    +        val file = new File(dir, "text.txt")
    +        stringToFile(file, "text")
    +
    +        val inMemoryFileIndex = new InMemoryFileIndex(
    +          spark, Seq(new Path(file.getCanonicalPath)), Map.empty, None) {
    +          def leafFileStatuses = leafFiles.map(_._2)
    --- End diff --
    
    nit, `def leafFileStatuses = leafFiles.values`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org