You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "jinchengchenghh (via GitHub)" <gi...@apache.org> on 2024/04/18 06:42:27 UTC
[PR] [VL] Support read CSV [incubator-gluten]
jinchengchenghh opened a new pull request, #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2100340273
Can you help merge it? Thanks! @zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2099626710
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2066196244
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2071875910
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1570334204
##########
gluten-data/src/main/scala/org/apache/gluten/datasource/ArrowFileFormat.scala:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.datasource
+
+import org.apache.gluten.utils.{ArrowUtil, Iterators}
+import org.apache.gluten.vectorized.ArrowWritableColumnVector
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.memory.SparkMemoryUtil.UnsafeItr
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources.{FileFormat, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.sources.{DataSourceRegister, Filter}
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.arrow.dataset.scanner.ScanOptions
+import org.apache.arrow.vector.VectorUnloader
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import java.net.URLDecoder
+import java.util.Optional
+
+import scala.collection.JavaConverters.asScalaBufferConverter
+
+class ArrowFileFormat(fileFormat: org.apache.arrow.dataset.file.FileFormat)
+ extends FileFormat
+ with DataSourceRegister
+ with Logging
+ with Serializable {
+
+ override def isSplitable(
+ sparkSession: SparkSession,
+ options: Map[String, String],
+ path: Path): Boolean = {
+ true
Review Comment:
Perhaps using `super.isSplitable(sparkSession, options, path)` would be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081750526
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1571603391
##########
backends-velox/src/test/scala/org/apache/gluten/execution/TestOperator.scala:
##########
@@ -458,6 +458,17 @@ class TestOperator extends VeloxWholeStageTransformerSuite {
}
}
+ test("csv scan") {
Review Comment:
yes, this PR supports compression codec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2078772780
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578836085
##########
shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala:
##########
@@ -1460,6 +1462,13 @@ object GlutenConfig {
.booleanConf
.createOptional
+ val NATIVE_ARROW_READER_ENABLED =
+ buildConf("spark.gluten.sql.native.arrow.reader.enabled")
+ .internal()
+ .doc("This is config to specify whether to enable the native columnar csv reader")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
Should we make it `false` util we reached some milestone, e.g., when we fixed `GlutenCSVSuite`?
##########
shims/common/src/main/scala/org/apache/gluten/sql/shims/SparkShims.scala:
##########
@@ -229,4 +230,6 @@ trait SparkShims {
def withAnsiEvalMode(expr: Expression): Boolean = false
+ def dateTimestampFormatInReadDefault(csvOptions: CSVOptions, timeZone: String): Boolean
Review Comment:
Don't quite understand what the method name means. If no self-explaining name can be used, could you add some comments for this method?
##########
gluten-data/src/main/scala/org/apache/gluten/utils/ArrowUtil.scala:
##########
@@ -86,4 +101,223 @@ object ArrowUtil extends Logging {
}
new Schema(fields)
}
+
+ def getFormat(format: String): FileFormat = {
+ format match {
+ case "parquet" => FileFormat.PARQUET
+ case "orc" => FileFormat.ORC
+ case "csv" => FileFormat.CSV
+ case _ => throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ def getFormat(format: org.apache.spark.sql.execution.datasources.FileFormat): FileFormat = {
+ format match {
+ case _: ParquetFileFormat =>
+ FileFormat.PARQUET
+ case _: CSVFileFormat =>
+ FileFormat.CSV
+ case _ =>
+ throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ private def rewriteUri(encodeUri: String): String = {
+ val decodedUri = encodeUri
+ val uri = URI.create(decodedUri)
+ if (uri.getScheme == "s3" || uri.getScheme == "s3a") {
+ val s3Rewritten =
+ new URI("s3", uri.getAuthority, uri.getPath, uri.getQuery, uri.getFragment).toString
+ return s3Rewritten
+ }
+ val sch = uri.getScheme match {
+ case "hdfs" => "hdfs"
+ case "file" => "file"
+ }
+ val ssp = uri.getScheme match {
+ case "hdfs" => uri.getSchemeSpecificPart
+ case "file" => "//" + uri.getSchemeSpecificPart
+ }
+ val rewritten = new URI(sch, ssp, uri.getFragment)
+ rewritten.toString
+ }
+
+ def makeArrowDiscovery(encodedUri: String, format: FileFormat): FileSystemDatasetFactory = {
+ val allocator = ArrowBufferAllocators.contextInstance()
+ val factory = new FileSystemDatasetFactory(
+ allocator,
+ NativeMemoryPool.getDefault, // TODO: wait to change
+ format,
+ rewriteUri(encodedUri))
+ factory
+ }
+
+ def readSchema(file: FileStatus, format: FileFormat): Option[StructType] = {
+ val factory: FileSystemDatasetFactory =
+ makeArrowDiscovery(file.getPath.toString, format)
+ val schema = factory.inspect()
+ try {
+ Option(SparkSchemaUtil.fromArrowSchema(schema))
+ } finally {
+ factory.close()
+ }
+ }
+
+ def readSchema(files: Seq[FileStatus], format: FileFormat): Option[StructType] = {
+ if (files.isEmpty) {
+ throw new IllegalArgumentException("No input file specified")
+ }
+
+ readSchema(files.head, format)
+ }
+
+ def compareStringFunc(caseSensitive: Boolean): (String, String) => Boolean = {
+ if (caseSensitive) { (str1: String, str2: String) => str1.equals(str2) }
+ else { (str1: String, str2: String) => str1.equalsIgnoreCase(str2) }
+ }
+
+ // If user specify schema by .schema(newSchemaDifferentWithFile)
+ def checkSchema(
+ requiredField: DataType,
+ parquetFileFieldType: ArrowType,
+ parquetFileFields: mutable.Buffer[Field]): Unit = {
+ val requiredFieldType =
+ SparkArrowUtil.toArrowType(requiredField, SparkSchemaUtil.getLocalTimezoneID)
+ if (!requiredFieldType.equals(parquetFileFieldType)) {
+ val arrowFileSchema = parquetFileFields
+ .map(f => f.toString)
+ .reduceLeft((f1, f2) => f1 + "\n" + f2)
+ throw new SchemaMismatchException(
+ s"Not support specified schema is different with file schema\n$arrowFileSchema")
+ }
+ }
+
+ def getRequestedField(
+ requiredSchema: StructType,
+ parquetFileFields: mutable.Buffer[Field],
+ caseSensitive: Boolean): Schema = {
+ val compareFunc = compareStringFunc(caseSensitive)
+ requiredSchema.foreach {
+ readField =>
+ // TODO: check schema inside of complex type
+ val matchedFields =
+ parquetFileFields.filter(field => compareFunc(field.getName, readField.name))
+ if (!caseSensitive && matchedFields.size > 1) {
+ // Need to fail if there is ambiguity, i.e. more than one field is matched
+ val fieldsString = matchedFields.map(_.getName).mkString("[", ", ", "]")
+ throw new RuntimeException(
+ s"""
+ |Found duplicate field(s) "${readField.name}": $fieldsString
+
+ |in case-insensitive mode""".stripMargin.replaceAll("\n", " "))
+ }
+ if (matchedFields.nonEmpty) {
+ checkSchema(
+ readField.dataType,
+ matchedFields.head.getFieldType.getType,
+ parquetFileFields)
+ }
+ }
+
+ val requestColNames = requiredSchema.map(_.name)
+ new Schema(parquetFileFields.filter {
+ field => requestColNames.exists(col => compareFunc(col, field.getName))
+ }.asJava)
+ }
+
+ def loadMissingColumns(
+ rowCount: Int,
+ missingSchema: StructType): Array[ArrowWritableColumnVector] = {
+
+ val vectors =
+ ArrowWritableColumnVector.allocateColumns(rowCount, missingSchema)
+ vectors.foreach {
+ vector =>
+ vector.putNulls(0, rowCount)
+ vector.setValueCount(rowCount)
+ }
+
+ SparkMemoryUtil.addLeakSafeTaskCompletionListener(
+ (_: TaskContext) => {
+ vectors.foreach(_.close())
+ })
+
+ vectors
+ }
+
+ def loadPartitionColumns(
+ rowCount: Int,
+ partitionSchema: StructType,
+ partitionValues: InternalRow): Array[ArrowWritableColumnVector] = {
+ val partitionColumns = ArrowWritableColumnVector.allocateColumns(rowCount, partitionSchema)
+ (0 until partitionColumns.length).foreach(
+ i => {
+ ArrowColumnVectorUtils.populate(partitionColumns(i), partitionValues, i)
+ partitionColumns(i).setValueCount(rowCount)
+ partitionColumns(i).setIsConstant()
+ })
+
+ SparkMemoryUtil.addLeakSafeTaskCompletionListener(
+ (_: TaskContext) => {
+ partitionColumns.foreach(_.close())
+ })
Review Comment:
ditto
##########
gluten-data/src/main/scala/org/apache/gluten/utils/ArrowUtil.scala:
##########
@@ -86,4 +101,223 @@ object ArrowUtil extends Logging {
}
new Schema(fields)
}
+
+ def getFormat(format: String): FileFormat = {
+ format match {
+ case "parquet" => FileFormat.PARQUET
+ case "orc" => FileFormat.ORC
+ case "csv" => FileFormat.CSV
+ case _ => throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ def getFormat(format: org.apache.spark.sql.execution.datasources.FileFormat): FileFormat = {
+ format match {
+ case _: ParquetFileFormat =>
+ FileFormat.PARQUET
+ case _: CSVFileFormat =>
+ FileFormat.CSV
+ case _ =>
+ throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ private def rewriteUri(encodeUri: String): String = {
+ val decodedUri = encodeUri
+ val uri = URI.create(decodedUri)
+ if (uri.getScheme == "s3" || uri.getScheme == "s3a") {
+ val s3Rewritten =
+ new URI("s3", uri.getAuthority, uri.getPath, uri.getQuery, uri.getFragment).toString
+ return s3Rewritten
+ }
+ val sch = uri.getScheme match {
+ case "hdfs" => "hdfs"
+ case "file" => "file"
+ }
+ val ssp = uri.getScheme match {
+ case "hdfs" => uri.getSchemeSpecificPart
+ case "file" => "//" + uri.getSchemeSpecificPart
+ }
+ val rewritten = new URI(sch, ssp, uri.getFragment)
+ rewritten.toString
+ }
+
+ def makeArrowDiscovery(encodedUri: String, format: FileFormat): FileSystemDatasetFactory = {
+ val allocator = ArrowBufferAllocators.contextInstance()
+ val factory = new FileSystemDatasetFactory(
+ allocator,
+ NativeMemoryPool.getDefault, // TODO: wait to change
+ format,
+ rewriteUri(encodedUri))
+ factory
+ }
+
+ def readSchema(file: FileStatus, format: FileFormat): Option[StructType] = {
+ val factory: FileSystemDatasetFactory =
+ makeArrowDiscovery(file.getPath.toString, format)
+ val schema = factory.inspect()
+ try {
+ Option(SparkSchemaUtil.fromArrowSchema(schema))
+ } finally {
+ factory.close()
+ }
+ }
+
+ def readSchema(files: Seq[FileStatus], format: FileFormat): Option[StructType] = {
+ if (files.isEmpty) {
+ throw new IllegalArgumentException("No input file specified")
+ }
+
+ readSchema(files.head, format)
+ }
+
+ def compareStringFunc(caseSensitive: Boolean): (String, String) => Boolean = {
+ if (caseSensitive) { (str1: String, str2: String) => str1.equals(str2) }
+ else { (str1: String, str2: String) => str1.equalsIgnoreCase(str2) }
+ }
+
+ // If user specify schema by .schema(newSchemaDifferentWithFile)
+ def checkSchema(
+ requiredField: DataType,
+ parquetFileFieldType: ArrowType,
+ parquetFileFields: mutable.Buffer[Field]): Unit = {
+ val requiredFieldType =
+ SparkArrowUtil.toArrowType(requiredField, SparkSchemaUtil.getLocalTimezoneID)
+ if (!requiredFieldType.equals(parquetFileFieldType)) {
+ val arrowFileSchema = parquetFileFields
+ .map(f => f.toString)
+ .reduceLeft((f1, f2) => f1 + "\n" + f2)
+ throw new SchemaMismatchException(
+ s"Not support specified schema is different with file schema\n$arrowFileSchema")
+ }
+ }
+
+ def getRequestedField(
+ requiredSchema: StructType,
+ parquetFileFields: mutable.Buffer[Field],
+ caseSensitive: Boolean): Schema = {
+ val compareFunc = compareStringFunc(caseSensitive)
+ requiredSchema.foreach {
+ readField =>
+ // TODO: check schema inside of complex type
+ val matchedFields =
+ parquetFileFields.filter(field => compareFunc(field.getName, readField.name))
+ if (!caseSensitive && matchedFields.size > 1) {
+ // Need to fail if there is ambiguity, i.e. more than one field is matched
+ val fieldsString = matchedFields.map(_.getName).mkString("[", ", ", "]")
+ throw new RuntimeException(
+ s"""
+ |Found duplicate field(s) "${readField.name}": $fieldsString
+
+ |in case-insensitive mode""".stripMargin.replaceAll("\n", " "))
+ }
+ if (matchedFields.nonEmpty) {
+ checkSchema(
+ readField.dataType,
+ matchedFields.head.getFieldType.getType,
+ parquetFileFields)
+ }
+ }
+
+ val requestColNames = requiredSchema.map(_.name)
+ new Schema(parquetFileFields.filter {
+ field => requestColNames.exists(col => compareFunc(col, field.getName))
+ }.asJava)
+ }
+
+ def loadMissingColumns(
+ rowCount: Int,
+ missingSchema: StructType): Array[ArrowWritableColumnVector] = {
+
+ val vectors =
+ ArrowWritableColumnVector.allocateColumns(rowCount, missingSchema)
+ vectors.foreach {
+ vector =>
+ vector.putNulls(0, rowCount)
+ vector.setValueCount(rowCount)
+ }
+
+ SparkMemoryUtil.addLeakSafeTaskCompletionListener(
+ (_: TaskContext) => {
+ vectors.foreach(_.close())
+ })
Review Comment:
Is there a way to avoid closing these vectors through task completion listener?
##########
gluten-core/src/main/scala/org/apache/gluten/utils/PlanUtil.scala:
##########
@@ -70,6 +70,9 @@ object PlanUtil {
case _: InputAdapter => false
case _: WholeStageCodegenExec => false
case r: ReusedExchangeExec => isVanillaColumnarOp(r.child)
+ case f: FileSourceScanExec =>
+ !f.relation.fileFormat.getClass.getSimpleName.equals("ArrowCSVFileFormat") &&
+ f.supportsColumnar
Review Comment:
Would you like to explain this change? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1580752830
##########
gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformSingleNode.scala:
##########
@@ -303,7 +303,15 @@ object TransformOthers {
class ReplaceSingleNode() extends LogLevelUtil with Logging {
def doReplace(p: SparkPlan): SparkPlan = {
- val plan = p
+ val plan = p match {
+ case plan: FileSourceScanExec
+ if plan.relation.fileFormat.getClass.getSimpleName == "ArrowCSVFileFormat" =>
+ val arrowScan = ArrowFileSourceScanExec(plan)
+ TransformHints.tagNotTransformable(arrowScan, "Arrow scan cannot transform")
+ return arrowScan
+ case p => p
+ }
+
if (TransformHints.isNotTransformable(plan)) {
Review Comment:
shall we handle `ArrowFileSourceScanExec` inside this branch ?
##########
gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformSingleNode.scala:
##########
@@ -322,6 +330,8 @@ object TransformOthers {
applyScanTransformer(plan)
case plan: FileSourceScanExec =>
applyScanTransformer(plan)
+ case plan: ArrowFileSourceScanExec =>
+ applyScanTransformer(plan)
Review Comment:
it seems we can never reach here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2090088047
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2100285538
@jinchengchenghh
I have successfully read CSV files in HDFS. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1570100475
##########
gluten-core/src/main/scala/org/apache/spark/memory/SparkMemoryUtil.scala:
##########
@@ -209,9 +209,50 @@ object SparkMemoryUtil {
sb.toString()
}
+ def addLeakSafeTaskCompletionListener[U](f: TaskContext => U): TaskContext = {
+ TaskContext.get().addTaskCompletionListener(f)
+ }
+
private case class PrintableMemoryUsageStats(
name: String,
used: Option[Long],
peak: Option[Long],
children: Iterable[PrintableMemoryUsageStats])
+
+ class GenericRetainer[T <: AutoCloseable] {
+ private var retained: Option[T] = None
+
+ def retain(batch: T): Unit = {
+ if (retained.isDefined) {
+ throw new IllegalStateException
+ }
+ retained = Some(batch)
+ }
+
+ def release(): Unit = {
+ retained.foreach(b => b.close())
+ retained = None
+ }
+ }
+
+ class UnsafeItr[T <: AutoCloseable](delegate: Iterator[T]) extends Iterator[T] {
+ val holder = new GenericRetainer[T]()
+
+ addLeakSafeTaskCompletionListener[Unit](
+ (_: TaskContext) => {
+ holder.release()
+ })
+
+ override def hasNext: Boolean = {
+ holder.release()
+ val hasNext = delegate.hasNext
+ hasNext
+ }
+
+ override def next(): T = {
+ val b = delegate.next()
+ holder.retain(b)
+ b
+ }
+ }
Review Comment:
Had some new APIs to replace this sort of wrappers
https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/utils/Iterators.scala
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2099972500
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2097422138
> the higher protobuf version will cause UnsatisfiedLinkError: /tmp/jnilib-4372912739792055919.tmp: /tmp/jnilib-4372912739792055919.tmp: undefined symbol: _ZTIN6google8protobuf7MessageE
Do we have a workaround for this issue? I also encountered it during testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2071508700
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1579033915
##########
gluten-core/src/main/scala/org/apache/gluten/utils/PlanUtil.scala:
##########
@@ -70,6 +70,9 @@ object PlanUtil {
case _: InputAdapter => false
case _: WholeStageCodegenExec => false
case r: ReusedExchangeExec => isVanillaColumnarOp(r.child)
+ case f: FileSourceScanExec =>
+ !f.relation.fileFormat.getClass.getSimpleName.equals("ArrowCSVFileFormat") &&
+ f.supportsColumnar
Review Comment:
We can simply add a validation line to https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala
And add some transformation code to https://github.com/apache/incubator-gluten/blob/c4b4c72732f2d662c6c7b0b41c46d47f77b281d0/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformSingleNode.scala#L289
Could just follow the normal way we transform file scan to file scan transformer.
It's awkward that we still need to take time to find a way to correctly inject some rules after such long time of development of Gluten. We need some complete refactors against the base arch eventually...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1579033915
##########
gluten-core/src/main/scala/org/apache/gluten/utils/PlanUtil.scala:
##########
@@ -70,6 +70,9 @@ object PlanUtil {
case _: InputAdapter => false
case _: WholeStageCodegenExec => false
case r: ReusedExchangeExec => isVanillaColumnarOp(r.child)
+ case f: FileSourceScanExec =>
+ !f.relation.fileFormat.getClass.getSimpleName.equals("ArrowCSVFileFormat") &&
+ f.supportsColumnar
Review Comment:
We can simply add a validation line to https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala
And add some transformation code to https://github.com/apache/incubator-gluten/blob/c4b4c72732f2d662c6c7b0b41c46d47f77b281d0/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformSingleNode.scala#L289
Could just follow the normal way we convert file scan to file scan transformer.
It's awkward that we still need to take time to find a way to correctly inject some rules after such long time of development of Gluten. We need some complete refactors against the base arch eventually...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081921086
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2090335218
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2105615316
TPCH SF2000 Q6 performance, query:
`select sum(l_extendedprice * l_discount) as revenue from lineitem where l_shipdate >= '1994-01-01' and l_shipdate < '1995-01-01' and l_discount between .06 - 0.01 and .06 + 0.01 and l_quantity < 24`
lineitem data: 622G
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:///C:/Users/cjin/AppData/Local/Temp/msohtmlclip1/01/clip.htm">
<link rel=File-List
href="file:///C:/Users/cjin/AppData/Local/Temp/msohtmlclip1/01/clip_filelist.xml">
<style>
<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:.75in .7in .75in .7in;
mso-header-margin:.3in;
mso-footer-margin:.3in;}
tr
{mso-height-source:auto;}
col
{mso-width-source:auto;}
br
{mso-data-placement:same-cell;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:11.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:Calibri, sans-serif;
mso-font-charset:0;
mso-number-format:General;
text-align:general;
vertical-align:bottom;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
-->
</style>
</head>
<body link="#0563C1" vlink="#954F72">
csv gluten without native reader | csv gluten native csv reader
-- | --
8333.039907 | 2456
</body>
</html>
Test script:
```
val schema = new StructType().add("l_orderkey", LongType).add("l_partkey", LongType).add("l_suppkey", LongType).add("l_linenumber", LongType).add("l_quantity", DoubleType).add("l_extendedprice", DoubleType).add("l_discount", DoubleType).add("l_tax", DoubleType).add("l_returnflag", StringType).add("l_linestatus", StringType).add("l_shipdate", DateType).add("l_commitdate", DateType).add("l_receiptdate", DateType).add("l_shipinstruct", StringType).add("l_shipmode", StringType).add("l_comment", StringType)
val lineitem = spark.read.format("csv").option("header","true").schema(schema).load("file:///mnt/DP_disk2/tpch/csvdata/")
spark.sql(q6)
```
Note: because the file schema should match Arrow schema, so we should specify the schema by `.schema(arrow_matched_schema)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2097465296
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1591796615
##########
backends-velox/src/main/scala/org/apache/gluten/datasource/ArrowConvertorRule.scala:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.datasource
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.sql.shims.SparkShimLoader
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.csv.CSVOptions
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.util.PermissiveMode
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.execution.datasources.csv.CSVFileFormat
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.utils.SparkSchemaUtil
+
+import java.nio.charset.StandardCharsets
+
+@Experimental
+case class ArrowConvertorRule(session: SparkSession) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (!BackendsApiManager.getSettings.enableNativeArrowReadFiles()) {
+ return plan
+ }
+ plan.resolveOperators {
+ // Read path
+ case l @ LogicalRelation(
+ r @ HadoopFsRelation(_, _, dataSchema, _, _: CSVFileFormat, options),
+ _,
+ _,
+ _) =>
+ val csvOptions = new CSVOptions(
+ options,
+ columnPruning = session.sessionState.conf.csvColumnPruning,
+ session.sessionState.conf.sessionLocalTimeZone)
+ if (
+ checkSchema(dataSchema) &&
+ checkCsvOptions(csvOptions, session.sessionState.conf.sessionLocalTimeZone)
+ ) {
+ l.copy(relation = r.copy(fileFormat = new ArrowCSVFileFormat())(session))
+ } else l
+ case r =>
+ r
+ }
+ }
+
+ private def checkCsvOptions(csvOptions: CSVOptions, timeZone: String): Boolean = {
+ csvOptions.headerFlag && !csvOptions.multiLine && csvOptions.delimiter == "," &&
Review Comment:
Yes, this is the default delimiter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2063332740
https://github.com/apache/incubator-gluten/issues/5414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2071266203
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2074195306
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1571515128
##########
gluten-data/src/main/scala/org/apache/gluten/datasource/ArrowFileFormat.scala:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.datasource
+
+import org.apache.gluten.utils.{ArrowUtil, Iterators}
+import org.apache.gluten.vectorized.ArrowWritableColumnVector
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.memory.SparkMemoryUtil.UnsafeItr
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources.{FileFormat, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.sources.{DataSourceRegister, Filter}
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.arrow.dataset.scanner.ScanOptions
+import org.apache.arrow.vector.VectorUnloader
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import java.net.URLDecoder
+import java.util.Optional
+
+import scala.collection.JavaConverters.asScalaBufferConverter
+
+class ArrowFileFormat(fileFormat: org.apache.arrow.dataset.file.FileFormat)
+ extends FileFormat
+ with DataSourceRegister
+ with Logging
+ with Serializable {
+
+ override def isSplitable(
+ sparkSession: SparkSession,
+ options: Map[String, String],
+ path: Path): Boolean = {
+ true
Review Comment:
super is false, but here is true. when codec is not empty, it cannot split, I will refactor here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1570665866
##########
backends-velox/src/test/scala/org/apache/gluten/execution/TestOperator.scala:
##########
@@ -458,6 +458,17 @@ class TestOperator extends VeloxWholeStageTransformerSuite {
}
}
+ test("csv scan") {
Review Comment:
does it support csv with compression like snappy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081300700
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2063121870
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICEGJ-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title*** in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2099774721
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2064982504
Can you paste a UI diagram in the first comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2073623397
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578928045
##########
gluten-core/src/main/scala/org/apache/gluten/utils/PlanUtil.scala:
##########
@@ -70,6 +70,9 @@ object PlanUtil {
case _: InputAdapter => false
case _: WholeStageCodegenExec => false
case r: ReusedExchangeExec => isVanillaColumnarOp(r.child)
+ case f: FileSourceScanExec =>
+ !f.relation.fileFormat.getClass.getSimpleName.equals("ArrowCSVFileFormat") &&
+ f.supportsColumnar
Review Comment:
So what if we add a rule to convert `FileSourceScanExec` to `GlutenFileSourceScanExec` (or something like that) that outputs Arrow columnar data by default? Which is reasonable since Gluten is guaranteed compatible with Arrow at arch level. The code can be placed in `gluten-data` and `cpp/core`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1581973308
##########
gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformSingleNode.scala:
##########
@@ -303,7 +303,15 @@ object TransformOthers {
class ReplaceSingleNode() extends LogLevelUtil with Logging {
def doReplace(p: SparkPlan): SparkPlan = {
- val plan = p
+ val plan = p match {
+ case plan: FileSourceScanExec
+ if plan.relation.fileFormat.getClass.getSimpleName == "ArrowCSVFileFormat" =>
+ val arrowScan = ArrowFileSourceScanExec(plan)
+ TransformHints.tagNotTransformable(arrowScan, "Arrow scan cannot transform")
+ return arrowScan
+ case p => p
+ }
+
if (TransformHints.isNotTransformable(plan)) {
Review Comment:
No, we the partitionFilters is null, it is taged not transformable, https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformHintRule.scala#L343, otherwise, the plan has no tag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1581993254
##########
gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformSingleNode.scala:
##########
@@ -303,7 +303,15 @@ object TransformOthers {
class ReplaceSingleNode() extends LogLevelUtil with Logging {
def doReplace(p: SparkPlan): SparkPlan = {
- val plan = p
+ val plan = p match {
+ case plan: FileSourceScanExec
+ if plan.relation.fileFormat.getClass.getSimpleName == "ArrowCSVFileFormat" =>
+ val arrowScan = ArrowFileSourceScanExec(plan)
+ TransformHints.tagNotTransformable(arrowScan, "Arrow scan cannot transform")
+ return arrowScan
+ case p => p
+ }
+
if (TransformHints.isNotTransformable(plan)) {
Review Comment:
The `ArrowFileSourceScanExec` is always tagged as fallback. So if we do not handle `ArrowFileSourceScanExec` here, what happens if `ArrowFileSourceScanExec` has a dynamic partition filter ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081748719
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578831193
##########
gluten-data/src/main/scala/org/apache/gluten/utils/ArrowUtil.scala:
##########
@@ -86,4 +101,223 @@ object ArrowUtil extends Logging {
}
new Schema(fields)
}
+
+ def getFormat(format: String): FileFormat = {
+ format match {
+ case "parquet" => FileFormat.PARQUET
+ case "orc" => FileFormat.ORC
+ case "csv" => FileFormat.CSV
+ case _ => throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ def getFormat(format: org.apache.spark.sql.execution.datasources.FileFormat): FileFormat = {
+ format match {
+ case _: ParquetFileFormat =>
+ FileFormat.PARQUET
+ case _: CSVFileFormat =>
+ FileFormat.CSV
+ case _ =>
+ throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ private def rewriteUri(encodeUri: String): String = {
+ val decodedUri = encodeUri
+ val uri = URI.create(decodedUri)
+ if (uri.getScheme == "s3" || uri.getScheme == "s3a") {
+ val s3Rewritten =
+ new URI("s3", uri.getAuthority, uri.getPath, uri.getQuery, uri.getFragment).toString
+ return s3Rewritten
+ }
+ val sch = uri.getScheme match {
+ case "hdfs" => "hdfs"
+ case "file" => "file"
+ }
+ val ssp = uri.getScheme match {
+ case "hdfs" => uri.getSchemeSpecificPart
+ case "file" => "//" + uri.getSchemeSpecificPart
+ }
+ val rewritten = new URI(sch, ssp, uri.getFragment)
+ rewritten.toString
+ }
+
+ def makeArrowDiscovery(encodedUri: String, format: FileFormat): FileSystemDatasetFactory = {
+ val allocator = ArrowBufferAllocators.contextInstance()
+ val factory = new FileSystemDatasetFactory(
+ allocator,
+ NativeMemoryPool.getDefault, // TODO: wait to change
Review Comment:
We can try `NativeMemoryPool#createListenable`. Though In Gluten we already had a listener named `ReservationListener` so we should do some tricks on naming.
##########
gluten-data/src/main/scala/org/apache/gluten/utils/ArrowUtil.scala:
##########
@@ -86,4 +101,223 @@ object ArrowUtil extends Logging {
}
new Schema(fields)
}
+
+ def getFormat(format: String): FileFormat = {
+ format match {
+ case "parquet" => FileFormat.PARQUET
+ case "orc" => FileFormat.ORC
+ case "csv" => FileFormat.CSV
+ case _ => throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ def getFormat(format: org.apache.spark.sql.execution.datasources.FileFormat): FileFormat = {
+ format match {
+ case _: ParquetFileFormat =>
+ FileFormat.PARQUET
+ case _: CSVFileFormat =>
+ FileFormat.CSV
+ case _ =>
+ throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ private def rewriteUri(encodeUri: String): String = {
+ val decodedUri = encodeUri
+ val uri = URI.create(decodedUri)
+ if (uri.getScheme == "s3" || uri.getScheme == "s3a") {
+ val s3Rewritten =
+ new URI("s3", uri.getAuthority, uri.getPath, uri.getQuery, uri.getFragment).toString
+ return s3Rewritten
+ }
+ val sch = uri.getScheme match {
+ case "hdfs" => "hdfs"
+ case "file" => "file"
+ }
+ val ssp = uri.getScheme match {
+ case "hdfs" => uri.getSchemeSpecificPart
+ case "file" => "//" + uri.getSchemeSpecificPart
+ }
+ val rewritten = new URI(sch, ssp, uri.getFragment)
+ rewritten.toString
+ }
+
+ def makeArrowDiscovery(encodedUri: String, format: FileFormat): FileSystemDatasetFactory = {
+ val allocator = ArrowBufferAllocators.contextInstance()
+ val factory = new FileSystemDatasetFactory(
+ allocator,
+ NativeMemoryPool.getDefault, // TODO: wait to change
Review Comment:
We can try `NativeMemoryPool#createListenable`. Though in Gluten we already had a listener named `ReservationListener` so we should do some tricks on naming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1590502221
##########
gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformSingleNode.scala:
##########
@@ -303,7 +303,15 @@ object TransformOthers {
class ReplaceSingleNode() extends LogLevelUtil with Logging {
def doReplace(p: SparkPlan): SparkPlan = {
- val plan = p
+ val plan = p match {
+ case plan: FileSourceScanExec
+ if plan.relation.fileFormat.getClass.getSimpleName == "ArrowCSVFileFormat" =>
+ val arrowScan = ArrowFileSourceScanExec(plan)
+ TransformHints.tagNotTransformable(arrowScan, "Arrow scan cannot transform")
+ return arrowScan
+ case p => p
+ }
Review Comment:
The code is placed in `gluten-core` but relies on code in `backend-velox`. Would you like to check whether we can add a rule via backend API?
Probably through `BackendsApiManager.getSparkPlanExecApiInstance.genExtendedColumnarValidationRules` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081350890
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081941062
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1582001603
##########
gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformSingleNode.scala:
##########
@@ -303,7 +303,15 @@ object TransformOthers {
class ReplaceSingleNode() extends LogLevelUtil with Logging {
def doReplace(p: SparkPlan): SparkPlan = {
- val plan = p
+ val plan = p match {
+ case plan: FileSourceScanExec
+ if plan.relation.fileFormat.getClass.getSimpleName == "ArrowCSVFileFormat" =>
+ val arrowScan = ArrowFileSourceScanExec(plan)
+ TransformHints.tagNotTransformable(arrowScan, "Arrow scan cannot transform")
+ return arrowScan
+ case p => p
+ }
+
if (TransformHints.isNotTransformable(plan)) {
Review Comment:
The csv arrow reader does not support filter pushdown now, so all the filter will left to FilterExec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2098074967
We need to add shaded exclude in `package/pom.xml`.
```xml
<exclude>org.apache.arrow.dataset.*</exclude>
```
Otherwise, JNI cannot find the correct dataset java method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1591761227
##########
backends-velox/src/main/scala/org/apache/gluten/datasource/ArrowConvertorRule.scala:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.datasource
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.sql.shims.SparkShimLoader
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.csv.CSVOptions
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.util.PermissiveMode
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.execution.datasources.csv.CSVFileFormat
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.utils.SparkSchemaUtil
+
+import java.nio.charset.StandardCharsets
+
+@Experimental
+case class ArrowConvertorRule(session: SparkSession) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (!BackendsApiManager.getSettings.enableNativeArrowReadFiles()) {
+ return plan
+ }
+ plan.resolveOperators {
+ // Read path
+ case l @ LogicalRelation(
+ r @ HadoopFsRelation(_, _, dataSchema, _, _: CSVFileFormat, options),
+ _,
+ _,
+ _) =>
+ val csvOptions = new CSVOptions(
+ options,
+ columnPruning = session.sessionState.conf.csvColumnPruning,
+ session.sessionState.conf.sessionLocalTimeZone)
+ if (
+ checkSchema(dataSchema) &&
+ checkCsvOptions(csvOptions, session.sessionState.conf.sessionLocalTimeZone)
+ ) {
+ l.copy(relation = r.copy(fileFormat = new ArrowCSVFileFormat())(session))
+ } else l
+ case r =>
+ r
+ }
+ }
+
+ private def checkCsvOptions(csvOptions: CSVOptions, timeZone: String): Boolean = {
+ csvOptions.headerFlag && !csvOptions.multiLine && csvOptions.delimiter == "," &&
Review Comment:
Does Arrow only support "," delimiter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2088256038
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2092662746
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2065602962
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2071502559
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081477348
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2079160777
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2099459389
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2099651706
After I enable HDFS in ARROW, I can successfully read csv file in HDFS.
```
scala> val filePath = "/input/student.csv"
filePath: String = /input/student.csv
scala> val df = spark.read.format("csv").option("header", "true").load(filePath)
E0508 10:25:42.841267 3005137 Exceptions.h:69] Line: /mnt/DP_disk1/code/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1850, Function:terminate, Expression: Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
df: org.apache.spark.sql.DataFrame = [Name: string, Language: string]
scala>
| df.show()
+-----+--------+
| Name|Language|
+-----+--------+
| Juno| Java|
|Peter| Python|
|Celin| C++|
+-----+--------+
scala> print(df.queryExecution.executedPlan)
*(1) ColumnarToRow
+- ArrowFileScan arrowcsv [Name#17,Language#18] Batched: true, DataFilters: [], Format: org.apache.gluten.datasource.ArrowCSVFileFormat@485f3327, Location: InMemoryFileIndex(1 paths)[hdfs://0.0.0.0:9000/input/student.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<Name:string,Language:string>
```
My local protobuf version, I'm not sure if it is related to protobuf version @liujiayi771
```
root@sr249:/mnt/DP_disk2/tpcds/scripts# protoc --version
libprotoc 3.21.4
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan merged PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2099960638
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2106536617
Th
> TPCH SF2000 Q6 performance, query: `select sum(l_extendedprice * l_discount) as revenue from lineitem where l_shipdate >= '1994-01-01' and l_shipdate < '1995-01-01' and l_discount between .06 - 0.01 and .06 + 0.01 and l_quantity < 24`
>
> lineitem data: 622G
>
> csv gluten without native reader csv gluten native csv reader
> 8333.039907 2456
> Test script:
>
> ```
> val schema = new StructType().add("l_orderkey", LongType).add("l_partkey", LongType).add("l_suppkey", LongType).add("l_linenumber", LongType).add("l_quantity", DoubleType).add("l_extendedprice", DoubleType).add("l_discount", DoubleType).add("l_tax", DoubleType).add("l_returnflag", StringType).add("l_linestatus", StringType).add("l_shipdate", DateType).add("l_commitdate", DateType).add("l_receiptdate", DateType).add("l_shipinstruct", StringType).add("l_shipmode", StringType).add("l_comment", StringType)
>
> val lineitem = spark.read.format("csv").option("header","true").schema(schema).load("file:///mnt/DP_disk2/tpch/csvdata/")
> spark.sql(q6)
> ```
>
> Note: because the file schema should match Arrow schema, so we should specify the schema by `.schema(arrow_matched_schema)`
Thank you, Chengcheng. What's the vanilla spark performance in this case? And how many task threads did you use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2097922641
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2097350683
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1591868977
##########
gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformSingleNode.scala:
##########
@@ -303,7 +303,15 @@ object TransformOthers {
class ReplaceSingleNode() extends LogLevelUtil with Logging {
def doReplace(p: SparkPlan): SparkPlan = {
- val plan = p
+ val plan = p match {
+ case plan: FileSourceScanExec
+ if plan.relation.fileFormat.getClass.getSimpleName == "ArrowCSVFileFormat" =>
+ val arrowScan = ArrowFileSourceScanExec(plan)
+ TransformHints.tagNotTransformable(arrowScan, "Arrow scan cannot transform")
+ return arrowScan
+ case p => p
+ }
Review Comment:
Can I change it in a following PR? This PR is too big
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578769775
##########
gluten-data/src/main/scala/org/apache/gluten/utils/ArrowUtil.scala:
##########
@@ -86,4 +101,223 @@ object ArrowUtil extends Logging {
}
new Schema(fields)
}
+
+ def getFormat(format: String): FileFormat = {
+ format match {
+ case "parquet" => FileFormat.PARQUET
+ case "orc" => FileFormat.ORC
+ case "csv" => FileFormat.CSV
+ case _ => throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ def getFormat(format: org.apache.spark.sql.execution.datasources.FileFormat): FileFormat = {
+ format match {
+ case _: ParquetFileFormat =>
+ FileFormat.PARQUET
+ case _: CSVFileFormat =>
+ FileFormat.CSV
+ case _ =>
+ throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ private def rewriteUri(encodeUri: String): String = {
+ val decodedUri = encodeUri
+ val uri = URI.create(decodedUri)
+ if (uri.getScheme == "s3" || uri.getScheme == "s3a") {
+ val s3Rewritten =
+ new URI("s3", uri.getAuthority, uri.getPath, uri.getQuery, uri.getFragment).toString
+ return s3Rewritten
+ }
+ val sch = uri.getScheme match {
+ case "hdfs" => "hdfs"
+ case "file" => "file"
+ }
+ val ssp = uri.getScheme match {
+ case "hdfs" => uri.getSchemeSpecificPart
+ case "file" => "//" + uri.getSchemeSpecificPart
+ }
+ val rewritten = new URI(sch, ssp, uri.getFragment)
+ rewritten.toString
+ }
+
+ def makeArrowDiscovery(encodedUri: String, format: FileFormat): FileSystemDatasetFactory = {
+ val allocator = ArrowBufferAllocators.contextInstance()
+ val factory = new FileSystemDatasetFactory(
+ allocator,
+ NativeMemoryPool.getDefault, // TODO: wait to change
Review Comment:
This memory pool usage is not tracked. Looks like we does not have a wrapper in Gluten. @zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2099604964
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2068405186
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2063188676
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2069127351
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2063405653
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2071499770
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081700557
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081837625
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2074056245
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578902257
##########
gluten-core/src/main/scala/org/apache/gluten/utils/PlanUtil.scala:
##########
@@ -70,6 +70,9 @@ object PlanUtil {
case _: InputAdapter => false
case _: WholeStageCodegenExec => false
case r: ReusedExchangeExec => isVanillaColumnarOp(r.child)
+ case f: FileSourceScanExec =>
+ !f.relation.fileFormat.getClass.getSimpleName.equals("ArrowCSVFileFormat") &&
+ f.supportsColumnar
Review Comment:
This will cause add extra c2r and r2c, the function is invoked in fallback to add c2r operator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081370967
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081333610
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578769775
##########
gluten-data/src/main/scala/org/apache/gluten/utils/ArrowUtil.scala:
##########
@@ -86,4 +101,223 @@ object ArrowUtil extends Logging {
}
new Schema(fields)
}
+
+ def getFormat(format: String): FileFormat = {
+ format match {
+ case "parquet" => FileFormat.PARQUET
+ case "orc" => FileFormat.ORC
+ case "csv" => FileFormat.CSV
+ case _ => throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ def getFormat(format: org.apache.spark.sql.execution.datasources.FileFormat): FileFormat = {
+ format match {
+ case _: ParquetFileFormat =>
+ FileFormat.PARQUET
+ case _: CSVFileFormat =>
+ FileFormat.CSV
+ case _ =>
+ throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ private def rewriteUri(encodeUri: String): String = {
+ val decodedUri = encodeUri
+ val uri = URI.create(decodedUri)
+ if (uri.getScheme == "s3" || uri.getScheme == "s3a") {
+ val s3Rewritten =
+ new URI("s3", uri.getAuthority, uri.getPath, uri.getQuery, uri.getFragment).toString
+ return s3Rewritten
+ }
+ val sch = uri.getScheme match {
+ case "hdfs" => "hdfs"
+ case "file" => "file"
+ }
+ val ssp = uri.getScheme match {
+ case "hdfs" => uri.getSchemeSpecificPart
+ case "file" => "//" + uri.getSchemeSpecificPart
+ }
+ val rewritten = new URI(sch, ssp, uri.getFragment)
+ rewritten.toString
+ }
+
+ def makeArrowDiscovery(encodedUri: String, format: FileFormat): FileSystemDatasetFactory = {
+ val allocator = ArrowBufferAllocators.contextInstance()
+ val factory = new FileSystemDatasetFactory(
+ allocator,
+ NativeMemoryPool.getDefault, // TODO: wait to change
Review Comment:
This memory pool usage is not tracked. Looks like we do not have a wrapper in Gluten. @zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2078756595
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578928045
##########
gluten-core/src/main/scala/org/apache/gluten/utils/PlanUtil.scala:
##########
@@ -70,6 +70,9 @@ object PlanUtil {
case _: InputAdapter => false
case _: WholeStageCodegenExec => false
case r: ReusedExchangeExec => isVanillaColumnarOp(r.child)
+ case f: FileSourceScanExec =>
+ !f.relation.fileFormat.getClass.getSimpleName.equals("ArrowCSVFileFormat") &&
+ f.supportsColumnar
Review Comment:
So what if we add a rule to convert `FileSourceScanExec` to `ArrowFileSourceScanExec` that outputs Arrow columnar data by default? Which is reasonable since Gluten is guaranteed compatible with Arrow at arch level. The code can be placed in `gluten-data` and `cpp/core`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578930561
##########
gluten-core/src/main/scala/org/apache/gluten/utils/PlanUtil.scala:
##########
@@ -70,6 +70,9 @@ object PlanUtil {
case _: InputAdapter => false
case _: WholeStageCodegenExec => false
case r: ReusedExchangeExec => isVanillaColumnarOp(r.child)
+ case f: FileSourceScanExec =>
+ !f.relation.fileFormat.getClass.getSimpleName.equals("ArrowCSVFileFormat") &&
+ f.supportsColumnar
Review Comment:
If we don't want to copy&paste code from `FileSourceScanExec`, we can adopt decorator/proxy design pattern to wrap a `FileSourceScanExec` instance inside `ArrowFileSourceScanExec`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578992672
##########
gluten-core/src/main/scala/org/apache/gluten/utils/PlanUtil.scala:
##########
@@ -70,6 +70,9 @@ object PlanUtil {
case _: InputAdapter => false
case _: WholeStageCodegenExec => false
case r: ReusedExchangeExec => isVanillaColumnarOp(r.child)
+ case f: FileSourceScanExec =>
+ !f.relation.fileFormat.getClass.getSimpleName.equals("ArrowCSVFileFormat") &&
+ f.supportsColumnar
Review Comment:
Then we should change the inject rule logic, replace FileSourceScanExec to other class. Like https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/spark/sql/execution/datasources/GlutenWriterColumnarRules.scala, https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/spark/sql/execution/datasources/GlutenWriterColumnarRules.scalathis implement looks complex, do we have easier way to inject the rule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578903451
##########
shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala:
##########
@@ -1460,6 +1462,13 @@ object GlutenConfig {
.booleanConf
.createOptional
+ val NATIVE_ARROW_READER_ENABLED =
+ buildConf("spark.gluten.sql.native.arrow.reader.enabled")
+ .internal()
+ .doc("This is config to specify whether to enable the native columnar csv reader")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
ok , default true is used to pass CI, I will only set it to true in `GlutenCSVSuite` and other relevant tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2068426943
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081301603
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2074233842
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1591868977
##########
gluten-core/src/main/scala/org/apache/gluten/extension/columnar/TransformSingleNode.scala:
##########
@@ -303,7 +303,15 @@ object TransformOthers {
class ReplaceSingleNode() extends LogLevelUtil with Logging {
def doReplace(p: SparkPlan): SparkPlan = {
- val plan = p
+ val plan = p match {
+ case plan: FileSourceScanExec
+ if plan.relation.fileFormat.getClass.getSimpleName == "ArrowCSVFileFormat" =>
+ val arrowScan = ArrowFileSourceScanExec(plan)
+ TransformHints.tagNotTransformable(arrowScan, "Arrow scan cannot transform")
+ return arrowScan
+ case p => p
+ }
Review Comment:
Can I change it in a following PR? This PR is too big @zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2095007542
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2092769051
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1591799839
##########
backends-velox/src/main/scala/org/apache/gluten/datasource/ArrowConvertorRule.scala:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.datasource
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.sql.shims.SparkShimLoader
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.csv.CSVOptions
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.util.PermissiveMode
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.execution.datasources.csv.CSVFileFormat
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.utils.SparkSchemaUtil
+
+import java.nio.charset.StandardCharsets
+
+@Experimental
+case class ArrowConvertorRule(session: SparkSession) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (!BackendsApiManager.getSettings.enableNativeArrowReadFiles()) {
+ return plan
+ }
+ plan.resolveOperators {
+ // Read path
+ case l @ LogicalRelation(
+ r @ HadoopFsRelation(_, _, dataSchema, _, _: CSVFileFormat, options),
+ _,
+ _,
+ _) =>
+ val csvOptions = new CSVOptions(
+ options,
+ columnPruning = session.sessionState.conf.csvColumnPruning,
+ session.sessionState.conf.sessionLocalTimeZone)
+ if (
+ checkSchema(dataSchema) &&
+ checkCsvOptions(csvOptions, session.sessionState.conf.sessionLocalTimeZone)
+ ) {
+ l.copy(relation = r.copy(fileFormat = new ArrowCSVFileFormat())(session))
+ } else l
+ case r =>
+ r
+ }
+ }
+
+ private def checkCsvOptions(csvOptions: CSVOptions, timeZone: String): Boolean = {
+ csvOptions.headerFlag && !csvOptions.multiLine && csvOptions.delimiter == "," &&
Review Comment:
And I will discuss with Arrow community to support more options, https://github.com/apache/arrow/issues/28866#issuecomment-2095354665
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2097427364
> > the higher protobuf version will cause UnsatisfiedLinkError: /tmp/jnilib-4372912739792055919.tmp: /tmp/jnilib-4372912739792055919.tmp: undefined symbol: _ZTIN6google8protobuf7MessageE
>
> Do we have a workaround for this issue? I also encountered it during testing.
Yes, I have fixed it by compiling Arrow https://github.com/apache/incubator-gluten/pull/5447/files#diff-e2b6a0ac0f3384d96902fbc545a697d0aa305c7ed711493df5e1bddf6c6d401bR295, and if your platform is centos, you may get the openssl link zlib issue, you could compile static openssl and set `export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig` to find correct `libcrypto.pc` `https://github.com/apache/incubator-gluten/pull/5447/files#diff-38eda6989d285e6a3d79a3a66cc656d0b6d483b3d5c77bd0969d878642fd0f0eR824
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2098336309
We also need to add HDFS support for arrow dataset.
```
caused by: java.lang.RuntimeException: Got HDFS URI but Arrow compiled without HDFS support
at org.apache.arrow.dataset.file.JniWrapper.makeFileSystemDatasetFactory(Native Method)
at org.apache.arrow.dataset.file.FileSystemDatasetFactory.createNative(FileSystemDatasetFactory.java:40)
at org.apache.arrow.dataset.file.FileSystemDatasetFactory.<init>(FileSystemDatasetFactory.java:31)
at org.apache.gluten.utils.ArrowUtil$.makeArrowDiscovery(ArrowUtil.scala:149)
at org.apache.gluten.datasource.ArrowCSVFileFormat.$anonfun$buildReaderWithPartitionValues$3(ArrowCSVFileFormat.scala:281)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:231)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:125)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2099996143
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2100958860
@jinchengchenghh can you do a test of performance improvement? csv parser only and csv + Gluten. You may create a large table table and run TPCH Q6.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2099529883
> We also need to add HDFS support for arrow dataset.
>
> ```
> caused by: java.lang.RuntimeException: Got HDFS URI but Arrow compiled without HDFS support
> at org.apache.arrow.dataset.file.JniWrapper.makeFileSystemDatasetFactory(Native Method)
> at org.apache.arrow.dataset.file.FileSystemDatasetFactory.createNative(FileSystemDatasetFactory.java:40)
> at org.apache.arrow.dataset.file.FileSystemDatasetFactory.<init>(FileSystemDatasetFactory.java:31)
> at org.apache.gluten.utils.ArrowUtil$.makeArrowDiscovery(ArrowUtil.scala:149)
> at org.apache.gluten.datasource.ArrowCSVFileFormat.$anonfun$buildReaderWithPartitionValues$3(ArrowCSVFileFormat.scala:281)
> at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:231)
> at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
> at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:125)
> ```
>
> But even after enabling ARROW_HDFS, there are still issues when accessing csv in HDFS.
>
> ```
> [libprotobuf ERROR google/protobuf/descriptor_database.cc:642] File already exists in database: Security.proto
> [libprotobuf FATAL google/protobuf/descriptor.cc:1986] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
> terminate called after throwing an instance of 'google::protobuf::FatalException'
> what(): CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
> ```
This may be in case of protobuf version, I will try to test CSV in HDFS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081334569
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081433304
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2081355647
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2063501509
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1571603391
##########
backends-velox/src/test/scala/org/apache/gluten/execution/TestOperator.scala:
##########
@@ -458,6 +458,17 @@ class TestOperator extends VeloxWholeStageTransformerSuite {
}
}
+ test("csv scan") {
Review Comment:
yes, this PR supports compression codec https://github.com/apache/arrow/pull/9685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2071326533
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2068604830
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2063122937
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2076282826
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578851690
##########
gluten-data/src/main/scala/org/apache/gluten/utils/ArrowUtil.scala:
##########
@@ -86,4 +101,223 @@ object ArrowUtil extends Logging {
}
new Schema(fields)
}
+
+ def getFormat(format: String): FileFormat = {
+ format match {
+ case "parquet" => FileFormat.PARQUET
+ case "orc" => FileFormat.ORC
+ case "csv" => FileFormat.CSV
+ case _ => throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ def getFormat(format: org.apache.spark.sql.execution.datasources.FileFormat): FileFormat = {
+ format match {
+ case _: ParquetFileFormat =>
+ FileFormat.PARQUET
+ case _: CSVFileFormat =>
+ FileFormat.CSV
+ case _ =>
+ throw new IllegalArgumentException("Unrecognizable format")
+ }
+ }
+
+ private def rewriteUri(encodeUri: String): String = {
+ val decodedUri = encodeUri
+ val uri = URI.create(decodedUri)
+ if (uri.getScheme == "s3" || uri.getScheme == "s3a") {
+ val s3Rewritten =
+ new URI("s3", uri.getAuthority, uri.getPath, uri.getQuery, uri.getFragment).toString
+ return s3Rewritten
+ }
+ val sch = uri.getScheme match {
+ case "hdfs" => "hdfs"
+ case "file" => "file"
+ }
+ val ssp = uri.getScheme match {
+ case "hdfs" => uri.getSchemeSpecificPart
+ case "file" => "//" + uri.getSchemeSpecificPart
+ }
+ val rewritten = new URI(sch, ssp, uri.getFragment)
+ rewritten.toString
+ }
+
+ def makeArrowDiscovery(encodedUri: String, format: FileFormat): FileSystemDatasetFactory = {
+ val allocator = ArrowBufferAllocators.contextInstance()
+ val factory = new FileSystemDatasetFactory(
+ allocator,
+ NativeMemoryPool.getDefault, // TODO: wait to change
Review Comment:
> We can try `NativeMemoryPool#createListenable`. Though in Gluten we already had a listener named `ReservationListener` so we should do some tricks on naming.
This can be done in separate PR if we can leave some TODO notes in code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578930561
##########
gluten-core/src/main/scala/org/apache/gluten/utils/PlanUtil.scala:
##########
@@ -70,6 +70,9 @@ object PlanUtil {
case _: InputAdapter => false
case _: WholeStageCodegenExec => false
case r: ReusedExchangeExec => isVanillaColumnarOp(r.child)
+ case f: FileSourceScanExec =>
+ !f.relation.fileFormat.getClass.getSimpleName.equals("ArrowCSVFileFormat") &&
+ f.supportsColumnar
Review Comment:
If we don't want to copy&paste code from `FileSourceScanExec`, we can adopt decorator design pattern to wrap a `FileSourceScanExec` instance inside `ArrowFileSourceScanExec`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1578936542
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala:
##########
@@ -748,6 +749,14 @@ class VeloxSparkPlanExecApi extends SparkPlanExecApi {
SparkShimLoader.getSparkShims.getExtendedColumnarPostRules() ::: List()
}
+ override def genInjectPostHocResolutionRules(): List[SparkSession => Rule[LogicalPlan]] = {
+ val buf: ListBuffer[SparkSession => Rule[LogicalPlan]] = ListBuffer()
+ if (BackendsApiManager.getSettings.enableNativeArrowReadFiles()) {
+ buf += ArrowConvertorRule
Review Comment:
Would you like to explain a bit about the reason to inject the rule as PostHocResolutionRule? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#discussion_r1579051612
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala:
##########
@@ -748,6 +749,14 @@ class VeloxSparkPlanExecApi extends SparkPlanExecApi {
SparkShimLoader.getSparkShims.getExtendedColumnarPostRules() ::: List()
}
+ override def genInjectPostHocResolutionRules(): List[SparkSession => Rule[LogicalPlan]] = {
+ val buf: ListBuffer[SparkSession => Rule[LogicalPlan]] = ListBuffer()
+ if (BackendsApiManager.getSettings.enableNativeArrowReadFiles()) {
+ buf += ArrowConvertorRule
Review Comment:
Because it replace the LogicalRelation( HadoopFsRelation(_, CSVFileFormat))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2074245858
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5414][VL] FEAT: Support read CSV [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5447:
URL: https://github.com/apache/incubator-gluten/pull/5447#issuecomment-2076163845
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org