You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "guixiaowen (via GitHub)" <gi...@apache.org> on 2026/03/10 08:00:39 UTC
[PR] [AURON #2080]Support Hive Parquet table to native [auron]
guixiaowen opened a new pull request, #2081:
URL: https://github.com/apache/auron/pull/2081
<!--
- Start the PR title with the related issue ID, e.g. '[AURON #XXXX] Short summary...'.
-->
# Which issue does this PR close?
Closes #2080
# Rationale for this change
Switch the Spark execution plan to HiveTableScanExec, and when the table storage type is Parquet, the native execution plan NativeParquetHiveTableScanExec executes.
# What changes are included in this PR?
# Are there any user-facing changes?
NO.
# How was this patch tested?
UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2080] Support Hive Parquet table to NativeParquetHiveTableScanExec [auron]
Posted by "guixiaowen (via GitHub)" <gi...@apache.org>.
guixiaowen commented on code in PR #2081:
URL: https://github.com/apache/auron/pull/2081#discussion_r3465416603
##########
dev/mvn-build-helper/assembly/pom.xml:
##########
@@ -120,10 +120,6 @@
<exclude>org.apache.arrow.c.**</exclude>
</excludes>
</relocation>
- <relocation>
- <pattern>io.netty</pattern>
Review Comment:
> This `io.netty` relocation was added in #1597 to fix `NoClassDefFoundError: io/netty/buffer/Unpooled` under shaded-spark — that change bundled `netty-buffer`/`netty-common` into the assembly _and_ relocated them so they wouldn't clash. This PR removes the relocation but keeps those bundled deps (`pom.xml:48-55`), so netty ships at its original `io.netty.*` coordinates again, which is the state #1597 fixed away. Does dropping it reopen that shaded-spark conflict? Since the relocation only affects the shaded assembly — not the unit tests or the Hive Parquet path here — it'd help to know what in this PR needs it gone, along with the `arrow-memory-netty` exclusions added to the shims `pom.xml`.
@weiqingy wei Thank you very much for the code review. I've been a bit busy these past couple of days, but I will complete the revisions as soon as possible. Thank you again for reviewing the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2080] Support Hive Parquet table to NativeParquetHiveTableScanExec [auron]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2081:
URL: https://github.com/apache/auron/pull/2081#issuecomment-5964219783
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2080] Support Hive Parquet table to NativeParquetHiveTableScanExec [auron]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #2081:
URL: https://github.com/apache/auron/pull/2081#discussion_r3450237808
##########
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/hive/execution/auron/plan/HiveConvertProvider.scala:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.hive.execution.auron.plan
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.auron.{AuronConverters, AuronConvertProvider}
+import org.apache.spark.sql.auron.AuronConverters.getBooleanConf
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.hive.client.HiveClientImpl
+import org.apache.spark.sql.hive.execution.HiveTableScanExec
+
+class HiveConvertProvider extends AuronConvertProvider with Logging {
+ override def isEnabled: Boolean =
Review Comment:
`AuronConvertProvider` declares `isEnabled(exec: SparkPlan): Boolean`, and the dispatcher calls it with the plan (`AuronConverters.scala:216` and `:307`). This override drops the parameter, so it doesn't implement the trait method — the abstract `isEnabled(SparkPlan)` stays unimplemented and `override` matches nothing, which the compiler rejects. That looks like the CI break flagged above. `PaimonConvertProvider` keeps the parameter and branches inside — would matching that signature work here?
```scala
override def isEnabled(exec: SparkPlan): Boolean =
getBooleanConf("spark.auron.enable.hiveTable", defaultValue = true)
```
##########
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/hive/execution/auron/plan/HiveConvertProvider.scala:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.hive.execution.auron.plan
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.auron.{AuronConverters, AuronConvertProvider}
+import org.apache.spark.sql.auron.AuronConverters.getBooleanConf
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.hive.client.HiveClientImpl
+import org.apache.spark.sql.hive.execution.HiveTableScanExec
+
+class HiveConvertProvider extends AuronConvertProvider with Logging {
+ override def isEnabled: Boolean =
+ getBooleanConf("spark.auron.enable.hiveTable", defaultValue = true)
+
+ private def enableHiveTableScanExec: Boolean =
+ getBooleanConf("spark.auron.enable.parquetHiveTableScanExec", defaultValue = false)
+
+ override def isSupported(exec: SparkPlan): Boolean =
Review Comment:
`isSupported` returns true for any table whose provider is `hive`, but `convert` only builds a native plan when `isParquetTable` holds and otherwise returns `exec` unchanged (line 49). So a non-Parquet Hive table (ORC, text) with both flags on would be reported as supported and then handed back un-converted, instead of falling through to the normal conversion path. Would gating `isSupported` on `isParquetTable` too keep the two in lockstep?
##########
dev/mvn-build-helper/assembly/pom.xml:
##########
@@ -120,10 +120,6 @@
<exclude>org.apache.arrow.c.**</exclude>
</excludes>
</relocation>
- <relocation>
- <pattern>io.netty</pattern>
Review Comment:
This `io.netty` relocation was added in #1597 to fix `NoClassDefFoundError: io/netty/buffer/Unpooled` under shaded-spark — that change bundled `netty-buffer`/`netty-common` into the assembly *and* relocated them so they wouldn't clash. This PR removes the relocation but keeps those bundled deps (`pom.xml:48-55`), so netty ships at its original `io.netty.*` coordinates again, which is the state #1597 fixed away. Does dropping it reopen that shaded-spark conflict? Since the relocation only affects the shaded assembly — not the unit tests or the Hive Parquet path here — it'd help to know what in this PR needs it gone, along with the `arrow-memory-netty` exclusions added to the shims `pom.xml`.
##########
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/hive/execution/auron/plan/NativeParquetHiveTableScanExec.scala:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.hive.execution.auron.plan
+
+import java.util.UUID
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.conf.Configurable
+import org.apache.hadoop.hive.ql.exec.Utilities
+import org.apache.hadoop.hive.ql.metadata.{Table => HiveTable}
+import org.apache.hadoop.hive.ql.plan.TableDesc
+import org.apache.hadoop.hive.serde.serdeConstants
+import org.apache.hadoop.hive.serde2.objectinspector.{ObjectInspectorUtils, StructObjectInspector}
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.ObjectInspectorCopyOption
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils
+import org.apache.hadoop.io.Writable
+import org.apache.hadoop.mapred.{FileSplit, InputFormat, JobConf}
+import org.apache.hadoop.mapreduce.{InputFormat => newInputClass}
+import org.apache.hadoop.util.ReflectionUtils
+import org.apache.spark.{Partition, TaskContext}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.HADOOP_RDD_IGNORE_EMPTY_SPLITS
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.auron.{NativeRDD, Shims}
+import org.apache.spark.sql.catalyst.expressions.{AttributeMap, GenericInternalRow}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.hive.{HadoopTableReader, HiveShim}
+import org.apache.spark.sql.hive.client.HiveClientImpl
+import org.apache.spark.sql.hive.execution.HiveTableScanExec
+
+import org.apache.auron.{protobuf => pb, sparkver}
+import org.apache.auron.metric.SparkMetricNode
+
+case class NativeParquetHiveTableScanExec(basedHiveScan: HiveTableScanExec)
+ extends NativeHiveTableScanBase(basedHiveScan)
+ with Logging {
+
+ @transient private lazy val nativeTable: HiveTable =
+ HiveClientImpl.toHiveTable(relation.tableMeta)
+ @transient private lazy val fileFormat =
+ ParquetHiveTableUtil.getFileFormat(nativeTable.getInputFormatClass)
+ @transient private lazy val nativeTableDesc = new TableDesc(
+ nativeTable.getInputFormatClass,
+ nativeTable.getOutputFormatClass,
+ nativeTable.getMetadata)
+
+ @transient private lazy val nativeHadoopConf = {
+ val hiveConf = SparkSession.getActiveSession.get.sessionState.newHadoopConf()
+ // append columns ids and names before broadcast
+ val columnOrdinals = AttributeMap(relation.dataCols.zipWithIndex)
+ val neededColumnIDs = output.flatMap(columnOrdinals.get).map(o => o: Integer)
+ val neededColumnNames = output.filter(columnOrdinals.contains).map(_.name)
+
+ HiveShim.appendReadColumns(hiveConf, neededColumnIDs, neededColumnNames)
+
+ val deserializer = nativeTableDesc.getDeserializerClass.getConstructor().newInstance()
+ deserializer.initialize(hiveConf, nativeTableDesc.getProperties)
+
+ // Specifies types and object inspectors of columns to be scanned.
+ val structOI = ObjectInspectorUtils
+ .getStandardObjectInspector(deserializer.getObjectInspector, ObjectInspectorCopyOption.JAVA)
+ .asInstanceOf[StructObjectInspector]
+
+ val columnTypeNames = structOI.getAllStructFieldRefs.asScala
+ .map(_.getFieldObjectInspector)
+ .map(TypeInfoUtils.getTypeInfoFromObjectInspector(_).getTypeName)
+ .mkString(",")
+
+ hiveConf.set(serdeConstants.LIST_COLUMN_TYPES, columnTypeNames)
+ hiveConf.set(serdeConstants.LIST_COLUMNS, relation.dataCols.map(_.name).mkString(","))
+ hiveConf
+ }
+
+ private val minPartitions = if (SparkSession.getActiveSession.get.sparkContext.isLocal) {
+ 0 // will splitted based on block by default.
+ } else {
+ math.max(
+ nativeHadoopConf.getInt("mapreduce.job.maps", 1),
+ SparkSession.getActiveSession.get.sparkContext.defaultMinPartitions)
+ }
+
+ private val ignoreEmptySplits =
+ SparkSession.getActiveSession.get.sparkContext.conf.get(HADOOP_RDD_IGNORE_EMPTY_SPLITS)
+
+ override val nodeName: String =
+ s"NativeHiveTableScan $tableName"
+
+ override def doExecuteNative(): NativeRDD = {
+ val nativeMetrics = SparkMetricNode(
+ metrics,
+ Nil,
+ Some({
+ case ("bytes_scanned", v) =>
+ val inputMetric = TaskContext.get.taskMetrics().inputMetrics
+ inputMetric.incBytesRead(v)
+ case ("output_rows", v) =>
+ val inputMetric = TaskContext.get.taskMetrics().inputMetrics
+ inputMetric.incRecordsRead(v)
+ case _ =>
+ }))
+ val nativeFileSchema = this.nativeFileSchema
+ val nativeFileGroups = this.nativeFileGroups
+ val nativePartitionSchema = this.nativePartitionSchema
+
+ val projection = schema.map(field => relation.schema.fieldIndex(field.name))
+ val broadcastedHadoopConf = this.broadcastedHadoopConf
+ val numPartitions = partitions.length
+
+ new NativeRDD(
+ sparkContext,
+ nativeMetrics,
+ partitions.asInstanceOf[Array[Partition]],
+ None,
+ Nil,
+ rddShuffleReadFull = true,
+ (partition, _) => {
+ val resourceId = s"NativeParquetHiveTableScan:${UUID.randomUUID().toString}"
+ putJniBridgeResource(resourceId, broadcastedHadoopConf)
+
+ val nativeFileGroup = nativeFileGroups(partition.asInstanceOf[FilePartition])
+ val nativeFileScanConf = pb.FileScanExecConf
+ .newBuilder()
+ .setNumPartitions(numPartitions)
+ .setPartitionIndex(partition.index)
+ .setStatistics(pb.Statistics.getDefaultInstance)
+ .setSchema(nativeFileSchema)
+ .setFileGroup(nativeFileGroup)
+ .addAllProjection(projection.map(Integer.valueOf).asJava)
+ .setPartitionSchema(nativePartitionSchema)
+ .build()
+ fileFormat match {
+ case "parquet" =>
+ val nativeParquetScanExecBuilder = pb.ParquetScanExecNode
+ .newBuilder()
+ .setBaseConf(nativeFileScanConf)
+ .setFsResourceId(resourceId)
+ .addAllPruningPredicates(new java.util.ArrayList()) // not support this filter
+
+ pb.PhysicalPlanNode
+ .newBuilder()
+ .setParquetScan(nativeParquetScanExecBuilder.build())
+ .build()
+ case "other" =>
+ throw new Exception("HiveTableExec only support parquet")
+ }
+ },
+ friendlyName = "NativeRDD.ParquetHiveTableScan")
+ }
+
+ @sparkver("3.1 / 3.2 / 3.3 / 3.4 / 3.5 / 4.0 / 4.1")
+ private def getPrunedPartitions(): Seq[org.apache.hadoop.hive.ql.metadata.Partition] = {
+ basedHiveScan.prunedPartitions
+ }
+
+ @sparkver("3.0")
+ private def getPrunedPartitions(): Seq[org.apache.hadoop.hive.ql.metadata.Partition] = {
+ basedHiveScan.rawPartitions
+ }
+
+ override def getFilePartitions(): Array[FilePartition] = {
+ val newJobConf = new JobConf(nativeHadoopConf)
+ val arrayFilePartition = ArrayBuffer[FilePartition]()
+ val partitionedFiles = if (relation.isPartitioned) {
+ val partitions = getPrunedPartitions
+ val arrayPartitionedFile = ArrayBuffer[PartitionedFile]()
+ partitions.foreach { partition =>
+ val partDesc = Utilities.getPartitionDescFromTableDesc(nativeTableDesc, partition, true)
+ val partPath = partition.getDataLocation
+ HadoopTableReader.initializeLocalJobConfFunc(partPath.toString, nativeTableDesc)(
+ newJobConf)
+ val partitionValues = partition.getTPartition.getValues
+
+ val partitionInternalRow = new GenericInternalRow(partitionValues.size())
+ for (partitionIndex <- 0 until partitionValues.size) {
+ partitionInternalRow.update(partitionIndex, partitionValues.get(partitionIndex))
+ }
+
+ val inputFormatClass = partDesc.getInputFileFormatClass
+ .asInstanceOf[Class[newInputClass[Writable, Writable]]]
+ newJobConf.set("mapred.input.dir", partPath.toString)
+ arrayPartitionedFile ++= getArrayPartitionedFile(
+ newJobConf,
+ inputFormatClass,
+ partitionInternalRow)
+ }
+ arrayPartitionedFile
+ .sortBy(_.length)(implicitly[Ordering[Long]].reverse)
+ .toArray
+ } else {
+ newJobConf.set("mapred.input.dir", nativeTableDesc.getProperties().getProperty("location"))
+ val inputFormatClass =
+ nativeTable.getInputFormatClass.asInstanceOf[Class[newInputClass[Writable, Writable]]]
+ getArrayPartitionedFile(newJobConf, inputFormatClass, new GenericInternalRow(0))
+ .sortBy(_.length)(implicitly[Ordering[Long]].reverse)
+ .toArray
+ }
+ arrayFilePartition ++= FilePartition.getFilePartitions(
+ SparkSession.getActiveSession.get,
+ partitionedFiles,
+ getMaxSplitBytes(SparkSession.getActiveSession.get))
+
+ arrayFilePartition.toArray
+ }
+
+ private def getMaxSplitBytes(sparkSession: SparkSession): Long = {
+ val defaultMaxSplitBytes = sparkSession.sessionState.conf.filesMaxPartitionBytes
+ val openCostInBytes = sparkSession.sessionState.conf.filesOpenCostInBytes
+ Math.min(defaultMaxSplitBytes, openCostInBytes)
Review Comment:
`filesMaxPartitionBytes` defaults to 128MB and `filesOpenCostInBytes` to 4MB, so `Math.min` here always returns the open-cost (4MB) and every split is capped at it — which would fan a large table out into many tiny partitions. Spark computes the cap as `min(maxPartitionBytes, max(openCostInBytes, bytesPerCore))` rather than a straight min of the two. Was `min` intended here? `FilePartition.maxSplitBytes` already implements that formula if it helps, though note it's keyed on `Seq[PartitionDirectory]` rather than the `Seq[PartitionedFile]` you have here, so it's not a literal drop-in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2080] Support Hive Parquet table to NativeParquetHiveTableScanExec [auron]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2081: [AURON #2080] Support Hive Parquet table to NativeParquetHiveTableScanExec
URL: https://github.com/apache/auron/pull/2081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org