You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/02 05:52:11 UTC

[GitHub] [spark] c21 opened a new pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

c21 opened a new pull request #29625:
URL: https://github.com/apache/spark/pull/29625


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   This PR is to support reading sorted bucket table efficiently in data source v1 read path. Previously in `FileSourceScanExec` -> `FileScanRDD`, we read input file sequentially one by one. For sorted bucket table which potentially can have multiple sorted files per bucket, the sort ordering for each bucket cannot be preserved as we read sorted files one by one. This PR is to add the support to read sorted files all together in a sort-merge way to preserve the ordering for each bucket.
   
   Specifically the code change is:
   * Add a parameter `ScanMode` (which can be either `RowMode` (read row by row - non-vectorization), `BatchMode` (read batch by batch - vectorization), or `SortedBucketMode` (read rows in sort-merge way - sorted bucketed file case)) for `FileScanRDD`. `FileScanRDD.compute()` decides which iterator to use based on this `ScanMode`.
   
   * Extract the existing iterator logic in `FileScanRDD` to `BaseFileScanIterator`, which holds common logic for file scan. `BaseFileScanIterator` has 3 subclasses where `FileRowScanIterator` reads row by row for each file (`RowMode`), and `FileBatchScanIterator` reads batch by batch for each file (`BatchMode`), and `FileSortedBucketScanIterator` reads row by row based on each row ordering across multiple files (`SortedBucketMode`).
   
   * `FileSortedBucketScanIterator`: a priority queue is used to output rows from multiple sorted files based on table sorted columns.
   
   The PR also separates the logic for existing row scan and batch scan, so we don't need to do `if (nextElement.isInstanceOf[ColumnarBatch]) {` at line 100 of `FileScanRDD` per row, which was a long-standing TODO.
   
   The whole feature is controlled by a new config `spark.sql.sources.bucketing.sortedScan.enabled` which is disabled by default, as there can be a risk to read more data in task and cause OOM (especially with vectorization, we need to keep each batch of each file in memory at any time). In addition, we need to set `InputFileBlockHolder` for each row (as any row from any file can be outputted), so it can be costly.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   Right now for sorted bucket table, even though each individual file is sorted, the current file scan approach does not preserve sort ordering across files in one bucket, so a sort still needs for a sort-merge join or sort aggregate. If the table is big, the external sort can happen and cause extra CPU and IO overhead. Introducing the code path here to read multiple sorted files in a sort-merge way to avoid later sort before join or aggregate, to save CPU and IO resource.
   
   At our internal fork, we added the similar support (https://www.youtube.com/watch?v=brzInUisshY&feature=youtu.be&t=910), though the code path is different (we are reading/writing hive table).
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   Yes. A new user-facing config `spark.sql.sources.bucketing.sortedScan.enabled` is introduced to allow users to read sorted bucketed table efficiently (e.g. no sort for sort merge join / sort aggregate on bucketed columns). In addition, slightly change the query plan for `FileSourceScanExec` physical operator, instead of printing `Batched`, print out `ScanMode` now.
   
   Example query plan with `ScanMode` in `FileScan`, and no `Sort` before `SortMergeJoin`:
   
   ```
   SortMergeJoin [i#38, j#39], [i#44, j#45], Inner
   :- Filter (isnotnull(i#38) AND isnotnull(j#39))
   :  +- FileScan parquet default.bucketed_table1[i#38,j#39,k#40] DataFilters: [isnotnull(i#38), isnotnull(j#39)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/chengsu/spark/sql/core/spark-warehouse/org.apache.spark.sql.sources..., PartitionFilters: [], PushedFilters: [IsNotNull(i), IsNotNull(j)], ReadSchema: struct<i:int,j:int,k:string>, ScanMode: SortedBucketMode, SelectedBucketsCount: 8 out of 8
   +- Filter (isnotnull(i#44) AND isnotnull(j#45))
      +- FileScan parquet default.bucketed_table2[i#44,j#45,k#46] DataFilters: [isnotnull(i#44), isnotnull(j#45)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/chengsu/spark/sql/core/spark-warehouse/org.apache.spark.sql.sources..., PartitionFilters: [], PushedFilters: [IsNotNull(i), IsNotNull(j)], ReadSchema: struct<i:int,j:int,k:string>, ScanMode: SortedBucketMode, SelectedBucketsCount: 8 out of 8
   ```
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   
   Added unit tests in `BucketedReadSuite.scala` and `ExplainSuite.scala`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-685393166






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686268465


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128228/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-685319506






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on a change in pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29625:
URL: https://github.com/apache/spark/pull/29625#discussion_r481752972



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanIterators.scala
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.{FileNotFoundException, IOException}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.{Partition => RDDPartition, SparkUpgradeException, TaskContext}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.executor.InputMetrics
+import org.apache.spark.internal.Logging
+import org.apache.spark.rdd.InputFileBlockHolder
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.{QueryExecutionException, RowIterator}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.NextIterator
+
+/**
+ * Holds common logic for iterators to scan files
+ */
+abstract class BaseFileScanIterator(
+    protected val split: RDDPartition,
+    context: TaskContext,
+    ignoreCorruptFiles: Boolean,
+    ignoreMissingFiles: Boolean,
+    readFunction: PartitionedFile => Iterator[InternalRow])
+  extends Iterator[Object]
+  with AutoCloseable
+  with Logging {
+
+  protected val inputMetrics: InputMetrics = context.taskMetrics().inputMetrics
+  private val existingBytesRead = inputMetrics.bytesRead
+
+  // Find a function that will return the FileSystem bytes read by this thread. Do this before
+  // apply readFunction, because it might read some bytes.
+  private val getBytesReadCallback =
+    SparkHadoopUtil.get.getFSBytesReadOnThreadCallback()
+
+  // We get our input bytes from thread-local Hadoop FileSystem statistics.
+  // If we do a coalesce, however, we are likely to compute multiple partitions in the same
+  // task and in the same thread, in which case we need to avoid override values written by
+  // previous partitions (SPARK-13071).
+  protected def incTaskInputMetricsBytesRead(): Unit = {
+    inputMetrics.setBytesRead(existingBytesRead + getBytesReadCallback())
+  }
+
+  private[this] val files = split.asInstanceOf[FilePartition].files.toIterator
+  protected[this] var currentFile: PartitionedFile = null
+  protected[this] var currentIterator: Iterator[Object] = null
+
+  override def hasNext: Boolean = {
+    // Kill the task in case it has been marked as killed. This logic is from
+    // InterruptibleIterator, but we inline it here instead of wrapping the iterator in order
+    // to avoid performance overhead.
+    context.killTaskIfInterrupted()
+    (currentIterator != null && currentIterator.hasNext) || nextIterator()
+  }
+
+  override def next(): Object
+
+  private def readFile(file: PartitionedFile): Iterator[InternalRow] = {

Review comment:
       This is changed compared to `FileScanRDD.readCurrentFile`, which takes a parameter instead of relying on `currentFile`, because we need to use `nextIterator()` to call `readFile()` multiple times for different files in `initializeHeapWithFirstRows()`.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanIterators.scala
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.{FileNotFoundException, IOException}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.{Partition => RDDPartition, SparkUpgradeException, TaskContext}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.executor.InputMetrics
+import org.apache.spark.internal.Logging
+import org.apache.spark.rdd.InputFileBlockHolder
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.{QueryExecutionException, RowIterator}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.NextIterator
+
+/**
+ * Holds common logic for iterators to scan files
+ */
+abstract class BaseFileScanIterator(
+    protected val split: RDDPartition,
+    context: TaskContext,
+    ignoreCorruptFiles: Boolean,
+    ignoreMissingFiles: Boolean,
+    readFunction: PartitionedFile => Iterator[InternalRow])
+  extends Iterator[Object]
+  with AutoCloseable
+  with Logging {
+
+  protected val inputMetrics: InputMetrics = context.taskMetrics().inputMetrics
+  private val existingBytesRead = inputMetrics.bytesRead
+
+  // Find a function that will return the FileSystem bytes read by this thread. Do this before
+  // apply readFunction, because it might read some bytes.
+  private val getBytesReadCallback =
+    SparkHadoopUtil.get.getFSBytesReadOnThreadCallback()
+
+  // We get our input bytes from thread-local Hadoop FileSystem statistics.
+  // If we do a coalesce, however, we are likely to compute multiple partitions in the same
+  // task and in the same thread, in which case we need to avoid override values written by
+  // previous partitions (SPARK-13071).
+  protected def incTaskInputMetricsBytesRead(): Unit = {
+    inputMetrics.setBytesRead(existingBytesRead + getBytesReadCallback())
+  }
+
+  private[this] val files = split.asInstanceOf[FilePartition].files.toIterator
+  protected[this] var currentFile: PartitionedFile = null
+  protected[this] var currentIterator: Iterator[Object] = null
+
+  override def hasNext: Boolean = {
+    // Kill the task in case it has been marked as killed. This logic is from
+    // InterruptibleIterator, but we inline it here instead of wrapping the iterator in order
+    // to avoid performance overhead.
+    context.killTaskIfInterrupted()
+    (currentIterator != null && currentIterator.hasNext) || nextIterator()
+  }
+
+  override def next(): Object
+
+  private def readFile(file: PartitionedFile): Iterator[InternalRow] = {
+    try {
+      readFunction(file)
+    } catch {
+      case e: FileNotFoundException =>
+        throw new FileNotFoundException(
+          e.getMessage + "\n" +
+            "It is possible the underlying files have been updated. " +
+            "You can explicitly invalidate the cache in Spark by " +
+            "running 'REFRESH TABLE tableName' command in SQL or " +
+            "by recreating the Dataset/DataFrame involved.")
+    }
+  }
+
+  /** Advances to the next file. Returns true if a new non-empty iterator is available. */
+  protected def nextIterator(): Boolean = {
+    if (files.hasNext) {
+      currentFile = files.next()
+      logInfo(s"Reading File $currentFile")
+      // Sets InputFileBlockHolder for the file block's information
+      InputFileBlockHolder.set(currentFile.filePath, currentFile.start, currentFile.length)
+
+      if (ignoreMissingFiles || ignoreCorruptFiles) {
+        currentIterator = new NextIterator[Object] {
+          private val file = currentFile
+          // The readFunction may read some bytes before consuming the iterator, e.g.,
+          // vectorized Parquet reader. Here we use lazy val to delay the creation of
+          // iterator so that we will throw exception in `getNext`.
+          private lazy val internalIter = readFile(file)
+
+          override def getNext(): AnyRef = {
+            try {
+              if (internalIter.hasNext) {
+                internalIter.next()
+              } else {
+                finished = true
+                null
+              }
+            } catch {
+              case e: FileNotFoundException if ignoreMissingFiles =>
+                logWarning(s"Skipped missing file: $currentFile", e)
+                finished = true
+                null
+              // Throw FileNotFoundException even if `ignoreCorruptFiles` is true
+              case e: FileNotFoundException if !ignoreMissingFiles => throw e
+              case e @ (_: RuntimeException | _: IOException) if ignoreCorruptFiles =>
+                logWarning(
+                  s"Skipped the rest of the content in the corrupted file: $currentFile", e)
+                finished = true
+                null
+            }
+          }
+
+          override def close(): Unit = {}
+        }
+      } else {
+        currentIterator = readFile(currentFile)
+      }
+
+      try {
+        currentIterator.hasNext

Review comment:
       This is changed from `hasNext` to `currentIterator.hasNext` because `FileSortedBucketScanIterator` needs to override `hasNext`. Given `currentIterator` is not null here anyway, this change should be a no-op. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686595346


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-685319506






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686299569


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128229/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on a change in pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29625:
URL: https://github.com/apache/spark/pull/29625#discussion_r482708458



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -350,7 +367,8 @@ case class FileSourceScanExec(
         "PartitionFilters" -> seqToString(partitionFilters),
         "PushedFilters" -> seqToString(pushedDownFilters),
         "DataFilters" -> seqToString(dataFilters),
-        "Location" -> locationDesc)
+        "Location" -> locationDesc,
+        "ScanMode" -> scanMode.toString)

Review comment:
       Will replace `Batched` with `ScanMode`. Keep `Batched` here to make sure some existing unit tests to pass. If we are good with the overall approach of PR, I will make the change here and fix more unit tests.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanIterators.scala
##########
@@ -0,0 +1,317 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.{FileNotFoundException, IOException}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.{Partition => RDDPartition, SparkUpgradeException, TaskContext}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.executor.InputMetrics
+import org.apache.spark.internal.Logging
+import org.apache.spark.rdd.InputFileBlockHolder
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.{QueryExecutionException, RowIterator}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.NextIterator
+
+/**
+ * Holds common logic for iterators to scan files
+ */
+abstract class BaseFileScanIterator(
+    split: RDDPartition,
+    context: TaskContext,
+    ignoreCorruptFiles: Boolean,
+    ignoreMissingFiles: Boolean,
+    readFunction: PartitionedFile => Iterator[InternalRow])
+  extends Iterator[Object]
+  with AutoCloseable
+  with Logging {
+
+  protected val inputMetrics: InputMetrics = context.taskMetrics().inputMetrics
+  private val existingBytesRead = inputMetrics.bytesRead
+
+  // Find a function that will return the FileSystem bytes read by this thread. Do this before
+  // apply readFunction, because it might read some bytes.
+  private val getBytesReadCallback =
+    SparkHadoopUtil.get.getFSBytesReadOnThreadCallback()
+
+  // We get our input bytes from thread-local Hadoop FileSystem statistics.
+  // If we do a coalesce, however, we are likely to compute multiple partitions in the same
+  // task and in the same thread, in which case we need to avoid override values written by
+  // previous partitions (SPARK-13071).
+  protected def incTaskInputMetricsBytesRead(): Unit = {
+    inputMetrics.setBytesRead(existingBytesRead + getBytesReadCallback())
+  }
+
+  private[this] val files = split.asInstanceOf[FilePartition].files.toIterator
+  protected[this] var currentFile: PartitionedFile = null
+  protected[this] var currentIterator: Iterator[Object] = null
+
+  override def hasNext: Boolean = {
+    // Kill the task in case it has been marked as killed. This logic is from
+    // InterruptibleIterator, but we inline it here instead of wrapping the iterator in order
+    // to avoid performance overhead.
+    context.killTaskIfInterrupted()
+    (currentIterator != null && currentIterator.hasNext) || nextIterator()
+  }
+
+  override def next(): Object
+
+  private def readFile(file: PartitionedFile): Iterator[InternalRow] = {
+    try {
+      readFunction(file)
+    } catch {
+      case e: FileNotFoundException =>
+        throw new FileNotFoundException(
+          e.getMessage + "\n" +
+            "It is possible the underlying files have been updated. " +
+            "You can explicitly invalidate the cache in Spark by " +
+            "running 'REFRESH TABLE tableName' command in SQL or " +
+            "by recreating the Dataset/DataFrame involved.")
+    }
+  }
+
+  /** Advances to the next file. Returns true if a new non-empty iterator is available. */
+  protected def nextIterator(): Boolean = {
+    if (files.hasNext) {
+      currentFile = files.next()
+      logInfo(s"Reading File $currentFile")
+      // Sets InputFileBlockHolder for the file block's information
+      InputFileBlockHolder.set(currentFile.filePath, currentFile.start, currentFile.length)
+
+      if (ignoreMissingFiles || ignoreCorruptFiles) {
+        currentIterator = new NextIterator[Object] {
+          private val file = currentFile
+          // The readFunction may read some bytes before consuming the iterator, e.g.,
+          // vectorized Parquet reader. Here we use lazy val to delay the creation of
+          // iterator so that we will throw exception in `getNext`.
+          private lazy val internalIter = readFile(file)
+
+          override def getNext(): AnyRef = {
+            try {
+              if (internalIter.hasNext) {
+                internalIter.next()
+              } else {
+                finished = true
+                null
+              }
+            } catch {
+              case e: FileNotFoundException if ignoreMissingFiles =>
+                logWarning(s"Skipped missing file: $currentFile", e)
+                finished = true
+                null
+              // Throw FileNotFoundException even if `ignoreCorruptFiles` is true
+              case e: FileNotFoundException if !ignoreMissingFiles => throw e
+              case e @ (_: RuntimeException | _: IOException) if ignoreCorruptFiles =>
+                logWarning(
+                  s"Skipped the rest of the content in the corrupted file: $currentFile", e)
+                finished = true
+                null
+            }
+          }
+
+          override def close(): Unit = {}
+        }
+      } else {
+        currentIterator = readFile(currentFile)
+      }
+
+      try {
+        context.killTaskIfInterrupted()
+        (currentIterator != null && currentIterator.hasNext) || nextIterator()

Review comment:
       This is changed compared to `FileScanRDD.nextIterator`, not call `hasNext()` because `FileSortedBucketScanIterator` needs to override `hasNext()`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] github-actions[bot] closed pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #29625:
URL: https://github.com/apache/spark/pull/29625


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-773571433


   @rahij - yes I am. Will raise a PR soon, thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686268301


   **[Test build #128228 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128228/testReport)** for PR 29625 at commit [`96a682a`](https://github.com/apache/spark/commit/96a682ac3939e7437fa8a88fb2efb36fc14cf4eb).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686600435






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686600435






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-685318923


   **[Test build #128187 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128187/testReport)** for PR 29625 at commit [`3a49ab0`](https://github.com/apache/spark/commit/3a49ab00c8f1f0c4bf9d53fa1356039444bc91a3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rahij commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
rahij commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-793098792


   Thank you, that makes sense. You don't have to take a look at https://github.com/palantir/spark/pull/730 - the current master branch is not up to date with apache/spark right now (but hopefully should soon). So we can just cherry pick whatever you get merged upstream once our branch is up to date.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on a change in pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29625:
URL: https://github.com/apache/spark/pull/29625#discussion_r481754353



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanIterators.scala
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.{FileNotFoundException, IOException}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.parquet.io.ParquetDecodingException
+
+import org.apache.spark.{Partition => RDDPartition, SparkUpgradeException, TaskContext}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.executor.InputMetrics
+import org.apache.spark.internal.Logging
+import org.apache.spark.rdd.InputFileBlockHolder
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.{QueryExecutionException, RowIterator}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.NextIterator
+
+/**
+ * Holds common logic for iterators to scan files
+ */
+abstract class BaseFileScanIterator(
+    protected val split: RDDPartition,
+    context: TaskContext,
+    ignoreCorruptFiles: Boolean,
+    ignoreMissingFiles: Boolean,
+    readFunction: PartitionedFile => Iterator[InternalRow])
+  extends Iterator[Object]
+  with AutoCloseable
+  with Logging {
+
+  protected val inputMetrics: InputMetrics = context.taskMetrics().inputMetrics
+  private val existingBytesRead = inputMetrics.bytesRead
+
+  // Find a function that will return the FileSystem bytes read by this thread. Do this before
+  // apply readFunction, because it might read some bytes.
+  private val getBytesReadCallback =
+    SparkHadoopUtil.get.getFSBytesReadOnThreadCallback()
+
+  // We get our input bytes from thread-local Hadoop FileSystem statistics.
+  // If we do a coalesce, however, we are likely to compute multiple partitions in the same
+  // task and in the same thread, in which case we need to avoid override values written by
+  // previous partitions (SPARK-13071).
+  protected def incTaskInputMetricsBytesRead(): Unit = {
+    inputMetrics.setBytesRead(existingBytesRead + getBytesReadCallback())
+  }
+
+  private[this] val files = split.asInstanceOf[FilePartition].files.toIterator
+  protected[this] var currentFile: PartitionedFile = null
+  protected[this] var currentIterator: Iterator[Object] = null
+
+  override def hasNext: Boolean = {
+    // Kill the task in case it has been marked as killed. This logic is from
+    // InterruptibleIterator, but we inline it here instead of wrapping the iterator in order
+    // to avoid performance overhead.
+    context.killTaskIfInterrupted()
+    (currentIterator != null && currentIterator.hasNext) || nextIterator()
+  }
+
+  override def next(): Object
+
+  private def readFile(file: PartitionedFile): Iterator[InternalRow] = {
+    try {
+      readFunction(file)
+    } catch {
+      case e: FileNotFoundException =>
+        throw new FileNotFoundException(
+          e.getMessage + "\n" +
+            "It is possible the underlying files have been updated. " +
+            "You can explicitly invalidate the cache in Spark by " +
+            "running 'REFRESH TABLE tableName' command in SQL or " +
+            "by recreating the Dataset/DataFrame involved.")
+    }
+  }
+
+  /** Advances to the next file. Returns true if a new non-empty iterator is available. */
+  protected def nextIterator(): Boolean = {
+    if (files.hasNext) {
+      currentFile = files.next()
+      logInfo(s"Reading File $currentFile")
+      // Sets InputFileBlockHolder for the file block's information
+      InputFileBlockHolder.set(currentFile.filePath, currentFile.start, currentFile.length)
+
+      if (ignoreMissingFiles || ignoreCorruptFiles) {
+        currentIterator = new NextIterator[Object] {
+          private val file = currentFile
+          // The readFunction may read some bytes before consuming the iterator, e.g.,
+          // vectorized Parquet reader. Here we use lazy val to delay the creation of
+          // iterator so that we will throw exception in `getNext`.
+          private lazy val internalIter = readFile(file)
+
+          override def getNext(): AnyRef = {
+            try {
+              if (internalIter.hasNext) {
+                internalIter.next()
+              } else {
+                finished = true
+                null
+              }
+            } catch {
+              case e: FileNotFoundException if ignoreMissingFiles =>
+                logWarning(s"Skipped missing file: $currentFile", e)
+                finished = true
+                null
+              // Throw FileNotFoundException even if `ignoreCorruptFiles` is true
+              case e: FileNotFoundException if !ignoreMissingFiles => throw e
+              case e @ (_: RuntimeException | _: IOException) if ignoreCorruptFiles =>
+                logWarning(
+                  s"Skipped the rest of the content in the corrupted file: $currentFile", e)
+                finished = true
+                null
+            }
+          }
+
+          override def close(): Unit = {}
+        }
+      } else {
+        currentIterator = readFile(currentFile)
+      }
+
+      try {
+        currentIterator.hasNext

Review comment:
       This is changed from `hasNext` to `currentIterator.hasNext` because `FileSortedBucketScanIterator` needs to override `hasNext`. Given `currentIterator` is not null here anyway, this change should be a no-op. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686256849






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686271954


   **[Test build #128229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128229/testReport)** for PR 29625 at commit [`774bd5e`](https://github.com/apache/spark/commit/774bd5e0da8a58fb2cdd8a68976a226d36c0b696).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-685317979


   cc @cloud-fan , @viirya , @imback82 , and @sameeragarwal to take a look when you guys have time, thanks.
   More unit tests will be added if we agree on overall approach, thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686268460


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-691279601


   After discussing with @cloud-fan , it would be better to have a rule to automatically decide whether to do bucket sorted scan based on query shape, and same for bucket scan. So I will first do bucket scan in https://issues.apache.org/jira/browse/SPARK-24528, and redo this one after the PR for https://issues.apache.org/jira/browse/SPARK-24528 is merged.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-773571433


   @rahij - yes I am. Will raise a PR soon, thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686599570


   **[Test build #128264 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128264/testReport)** for PR 29625 at commit [`774bd5e`](https://github.com/apache/spark/commit/774bd5e0da8a58fb2cdd8a68976a226d36c0b696).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686256849






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686599570


   **[Test build #128264 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128264/testReport)** for PR 29625 at commit [`774bd5e`](https://github.com/apache/spark/commit/774bd5e0da8a58fb2cdd8a68976a226d36c0b696).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686272494






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rahij commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
rahij commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-773412026


   @c21 I wanted to ask if you were planning on continuing this PR now that https://github.com/apache/spark/pull/29804/files has been merged?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686299556


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686256453


   **[Test build #128228 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128228/testReport)** for PR 29625 at commit [`96a682a`](https://github.com/apache/spark/commit/96a682ac3939e7437fa8a88fb2efb36fc14cf4eb).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686766392






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-691279601


   After discussing with @cloud-fan , it would be better to have a rule to automatically decide whether to do bucket sorted scan based on query shape, and same for bucket scan. So I will first do bucket scan in https://issues.apache.org/jira/browse/SPARK-24528, and redo this one after the PR for https://issues.apache.org/jira/browse/SPARK-24528 is merged.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686765578


   **[Test build #128264 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128264/testReport)** for PR 29625 at commit [`774bd5e`](https://github.com/apache/spark/commit/774bd5e0da8a58fb2cdd8a68976a226d36c0b696).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-793100380


   @rahij - yes sure. sorry I was just reading as "review" instead of "revive" by mistake :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on a change in pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #29625:
URL: https://github.com/apache/spark/pull/29625#discussion_r482708458



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -350,7 +367,8 @@ case class FileSourceScanExec(
         "PartitionFilters" -> seqToString(partitionFilters),
         "PushedFilters" -> seqToString(pushedDownFilters),
         "DataFilters" -> seqToString(dataFilters),
-        "Location" -> locationDesc)
+        "Location" -> locationDesc,
+        "ScanMode" -> scanMode.toString)

Review comment:
       Will replace `Batched` with `ScanMode`. Keep `Batched` here to make sure some existing unit tests to pass. If we are good with the overall approach of PR, I will make the change here and fix more unit tests.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-685318923


   **[Test build #128187 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128187/testReport)** for PR 29625 at commit [`3a49ab0`](https://github.com/apache/spark/commit/3a49ab00c8f1f0c4bf9d53fa1356039444bc91a3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rahij commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
rahij commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-793079201


   @c21 Just wanted to check if you had any time to revive this PR. I'm happy to create a new PR off this branch, but I figured you know this code much better than I do in case we need to change/fix something. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686271954


   **[Test build #128229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128229/testReport)** for PR 29625 at commit [`774bd5e`](https://github.com/apache/spark/commit/774bd5e0da8a58fb2cdd8a68976a226d36c0b696).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686299291


   **[Test build #128229 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128229/testReport)** for PR 29625 at commit [`774bd5e`](https://github.com/apache/spark/commit/774bd5e0da8a58fb2cdd8a68976a226d36c0b696).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686268460






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-685393166


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-685390635


   **[Test build #128187 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128187/testReport)** for PR 29625 at commit [`3a49ab0`](https://github.com/apache/spark/commit/3a49ab00c8f1f0c4bf9d53fa1356039444bc91a3).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686766392






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-691279601






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rahij commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
rahij commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-773412026


   @c21 I wanted to ask if you were planning on continuing this PR now that https://github.com/apache/spark/pull/29804/files has been merged?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-685393198


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128187/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686299556






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686256453


   **[Test build #128228 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128228/testReport)** for PR 29625 at commit [`96a682a`](https://github.com/apache/spark/commit/96a682ac3939e7437fa8a88fb2efb36fc14cf4eb).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-686272500






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-793097454


   @rahij - sure I will take a look of https://github.com/palantir/spark/pull/730. Sorry I was busy with something else, I will have a PR ready in this week.
   
   Btw, I got feedback earlier for this PR, that we want to enable/disable sorted bucketed scan automatically similar to https://github.com/apache/spark/pull/29804 . So I need more time to craft for that, thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] github-actions[bot] commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-748702666


   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] c21 commented on pull request #29625: [SPARK-24528][SQL] Add support to read multiple sorted bucket files for data source v1

Posted by GitBox <gi...@apache.org>.
c21 commented on pull request #29625:
URL: https://github.com/apache/spark/pull/29625#issuecomment-691279601


   After discussing with @cloud-fan , it would be better to have a rule to automatically decide whether to do bucket sorted scan based on query shape, and same for bucket scan. So I will first do bucket scan in https://issues.apache.org/jira/browse/SPARK-24528, and redo this one after the PR for https://issues.apache.org/jira/browse/SPARK-24528 is merged.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org