You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2022/11/28 08:07:47 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #3803] Improve the data interchange performance via Arrow serialization/deserialization
This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new a97b84330 [KYUUBI #3803] Improve the data interchange performance via Arrow serialization/deserialization
a97b84330 is described below
commit a97b8433035b45745d43bb4cf536f802db8693cd
Author: Fu Chen <cf...@gmail.com>
AuthorDate: Mon Nov 28 16:07:37 2022 +0800
[KYUUBI #3803] Improve the data interchange performance via Arrow serialization/deserialization
### _Why are the changes needed?_
to close #3794
This PR introduces the Apache Arrow library to improve the performance of data interchange.
data flow diagram with saprk engine
```
1. Spark Executor -> 2. Spark Driver -> 3. Kyuubi Server -> 4. Beeline/KyuubiHiveDriver
```
1. for simple types(Numeric/String/Binary/Boolean/Datetime/Interval), we serialize the columns on the Spark executor side and deserialize them on the client side.
2. for complex types(Array/Map/Struct), we first convert these columns to hive string and serialize the string columns on the executor side, and finally deserialize them as string type on the client side.
## Local Test
### Input dataset
```scala
spark.sql(
"""
|select * from tpcds.sf10.catalog_sales limit 2000000
|""".stripMargin)
.write
.mode("overwrite")
.save("/tmp/parquet/tpcds/sf10/catalog_sales_2000000")
```
### Query
cat /tmp/b.sql
```sql
select * from parquet.`/tmp/parquet/tpcds/sf10/catalog_sales_2000000`;
```
### Beeline command
```
bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/;' --hiveconf kyuubi.session.result.codec=simple/arrow --hiveconf kyuubi.operation.incremental.collect=false/true -f /tmp/b.sql > /dev/null
```
### Spark
```
/Users/fchen/Software/spark-3.3.0-bin-hadoop3/bin/spark-submit \
--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
--driver-memory 6g \
--master local[*] \
......
```
### Result
| kyuubi.session.result.codec | kyuubi.operation.incremental.collect | time spent|
| --- | --- | --- |
| arrow | false | 272.178 seconds |
| arrow | true | 146.76 seconds |
| simple | true | 607.523 seconds |
| simple | false | 429.374 seconds |
Closes #3803 from cfmcgrady/arrow.
Closes #3803
05b2b6f5 [Fu Chen] address comment
37e54e93 [Fu Chen] rename `kyuubi.session.result.codec` to `kyuubi.operation.result.codec`
fa44b102 [Fu Chen] address comment
e66b3bbd [Fu Chen] rebase master
08e2a97b [Fu Chen] fix shade
ab796bc6 [Fu Chen] address comment
1c979318 [Fu Chen] only consider spark engine major/minor version
dd61c015 [Fu Chen] add missing relocate dep `netty`
02f1d2f7 [Fu Chen] relocate arrow/jackson deps
4a2f7f09 [Fu Chen] minor refactor SparkDataTypeTests with SemanticVersion
627cc0a2 [Fu Chen] update kyuubi-jdbc-high-shaded NOTICE
815c68c2 [Fu Chen] reduce kyuubi-hive-jdbc/pom.xml
a7e2fcd9 [Fu Chen] fix style
825fda67 [Fu Chen] skip some tests when spark version < 3.3
039dbba1 [Fu Chen] minor refactor
95ad84c4 [Fu Chen] address comment
e1e81f01 [Fu Chen] revert unnecessary changes
36bce1d9 [Fu Chen] clean up
c789327c [Fu Chen] make sure SparkArrowbasedOperationSuite run with arrow enabled
d60c3120 [Fu Chen] clean up array-related implementations
8ca1fcd6 [Fu Chen] clean up unused arrow array implementations
8f96fdf3 [Fu Chen] fix
cdb83cf0 [Fu Chen] rename conf "kyuubi.operation.result.codec" to "kyuubi.session.result.codec"
f30c2c9d [Fu Chen] rename conf "kyuubi.beeline.arrow.enabled" to "kyuubi.operation.result.codec"
109a4f32 [Fu Chen] Fork Apache Spark-3.3.1 org.apache.spark.sql.catalyst.util.quoteIfNeeded to adapt to Spark-3.1.x
9ec23de3 [Fu Chen] revert kyuubi-server/src/main/scala/org/apache/kyuubi/server/BackendServiceMetric.scala
f2b7f2c3 [Fu Chen] revert kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiCallableStatement.java
33e820c7 [Fu Chen] revert temporary `pom.xml` changes
daa259e5 [Fu Chen] add SparkArrowbasedOperationSuite
eb5ed344 [Fu Chen] make kyuubi.beeline.arrow.enabled default to false and enabled via url support
5c3a48c7 [Fu Chen] fix ut SparkOperationSuite: execute statement - select column name with dots
0dfd442b [Fu Chen] complex type(array/map/array) support
86b0a485 [Fu Chen] INTERVAL_YEAR_MONTH_TYPE type support
10da4fe9 [Fu Chen] INTERVAL_DAY_TIME_TYPE type support
3a8b0da3 [Fu Chen] implement KyuubiArrowBasedResultSet.getDate
a9fd48c1 [Fu Chen] implement KyuubiArrowBasedResultSet.getInt
a37b5564 [Fu Chen] only ExecuteStatement support arrow
fe47fdc4 [Fu Chen] add java version ArrowUtils
d162a594 [Fu Chen] lisence
5414ccc7 [Fu Chen] remove debug println
335b4e8f [Fu Chen] drop scala dependency for module kyuubi-hive-jdbc
f826e262 [Fu Chen] fix ut
bcf9744d [Fu Chen] license
3ebca27c [Fu Chen] fix style
540423d5 [Fu Chen] timestamp support
0965b8f4 [Fu Chen] revert extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala
ad1a7092 [Fu Chen] revert
5b4ee403 [Fu Chen] BackendServiceMetric
21b76d29 [Fu Chen] add more types support
ace05a24 [Fu Chen] arrow-based collect
Authored-by: Fu Chen <cf...@gmail.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
---
.../engine/spark/operation/ExecuteStatement.scala | 40 +-
.../engine/spark/operation/SparkOperation.scala | 33 +-
.../apache/kyuubi/engine/spark/schema/RowSet.scala | 28 ++
.../kyuubi/engine/spark/schema/SchemaHelper.scala | 16 +-
.../spark/sql/kyuubi/SparkDatasetHelper.scala | 75 ++++
.../operation/SparkArrowbasedOperationSuite.scala | 43 ++
.../spark/operation/SparkOperationSuite.scala | 6 +-
.../engine/spark/schema/SchemaHelperSuite.scala | 22 +-
.../org/apache/kyuubi/engine/SemanticVersion.scala | 8 +
.../kyuubi/operation/SparkDataTypeTests.scala | 329 +++++++++++++++
.../apache/kyuubi/operation/SparkQueryTests.scala | 306 +-------------
kyuubi-hive-jdbc-shaded/pom.xml | 27 +-
.../src/main/resources/META-INF/NOTICE | 23 +
kyuubi-hive-jdbc/pom.xml | 32 +-
.../kyuubi/jdbc/hive/JdbcColumnAttributes.java | 9 +-
.../jdbc/hive/KyuubiArrowBasedResultSet.java | 452 ++++++++++++++++++++
...sultSet.java => KyuubiArrowQueryResultSet.java} | 93 ++++-
.../apache/kyuubi/jdbc/hive/KyuubiConnection.java | 18 +
.../kyuubi/jdbc/hive/KyuubiQueryResultSet.java | 2 -
.../apache/kyuubi/jdbc/hive/KyuubiStatement.java | 68 ++-
.../kyuubi/jdbc/hive/arrow/ArrowColumnVector.java | 464 +++++++++++++++++++++
.../kyuubi/jdbc/hive/arrow/ArrowColumnarBatch.java | 116 ++++++
.../jdbc/hive/arrow/ArrowColumnarBatchRow.java | 150 +++++++
.../apache/kyuubi/jdbc/hive/arrow/ArrowUtils.java | 101 +++++
.../apache/kyuubi/jdbc/hive/common/DateUtils.java | 19 +
.../kyuubi/server/BackendServiceMetric.scala | 1 +
pom.xml | 11 +
27 files changed, 2123 insertions(+), 369 deletions(-)
diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala
index 0b0e438f3..97d657459 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala
@@ -23,7 +23,8 @@ import scala.collection.JavaConverters._
import org.apache.hive.service.rpc.thrift.TProgressUpdateResp
import org.apache.spark.kyuubi.{SparkProgressMonitor, SQLOperationListener}
-import org.apache.spark.sql.Row
+import org.apache.spark.sql.{DataFrame, Row}
+import org.apache.spark.sql.kyuubi.SparkDatasetHelper
import org.apache.spark.sql.types._
import org.apache.kyuubi.{KyuubiSQLException, Logging}
@@ -92,19 +93,39 @@ class ExecuteStatement(
Thread.currentThread().setContextClassLoader(spark.sharedState.jarClassLoader)
operationListener.foreach(spark.sparkContext.addSparkListener(_))
result = spark.sql(statement)
+
iter =
if (incrementalCollect) {
info("Execute in incremental collect mode")
- new IterableFetchIterator[Row](result.toLocalIterator().asScala.toIterable)
+ if (arrowEnabled) {
+ new IterableFetchIterator[Array[Byte]](
+ SparkDatasetHelper.toArrowBatchRdd(
+ convertComplexType(result)).toLocalIterator.toIterable)
+ } else {
+ new IterableFetchIterator[Row](result.toLocalIterator().asScala.toIterable)
+ }
} else {
val resultMaxRows = spark.conf.getOption(OPERATION_RESULT_MAX_ROWS.key).map(_.toInt)
.getOrElse(session.sessionManager.getConf.get(OPERATION_RESULT_MAX_ROWS))
if (resultMaxRows <= 0) {
info("Execute in full collect mode")
- new ArrayFetchIterator(result.collect())
+ if (arrowEnabled) {
+ new ArrayFetchIterator(
+ SparkDatasetHelper.toArrowBatchRdd(
+ convertComplexType(result)).collect())
+ } else {
+ new ArrayFetchIterator(result.collect())
+ }
} else {
info(s"Execute with max result rows[$resultMaxRows]")
- new ArrayFetchIterator(result.take(resultMaxRows))
+ if (arrowEnabled) {
+ // this will introduce shuffle and hurt performance
+ new ArrayFetchIterator(
+ SparkDatasetHelper.toArrowBatchRdd(
+ convertComplexType(result.limit(resultMaxRows))).collect())
+ } else {
+ new ArrayFetchIterator(result.take(resultMaxRows))
+ }
}
}
setCompiledStateIfNeeded()
@@ -190,4 +211,15 @@ class ExecuteStatement(
SparkOperationEvent(this, operationListener.flatMap(_.getExecutionId)))
}
}
+
+ // TODO:(fchen) make this configurable
+ val kyuubiBeelineConvertToString = true
+
+ def convertComplexType(df: DataFrame): DataFrame = {
+ if (kyuubiBeelineConvertToString) {
+ SparkDatasetHelper.convertTopLevelComplexTypeToHiveString(df)
+ } else {
+ df
+ }
+ }
}
diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkOperation.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkOperation.scala
index 1d08f540d..8bb512384 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkOperation.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkOperation.scala
@@ -36,7 +36,7 @@ import org.apache.kyuubi.operation.{AbstractOperation, FetchIterator, OperationS
import org.apache.kyuubi.operation.FetchOrientation._
import org.apache.kyuubi.operation.OperationState.OperationState
import org.apache.kyuubi.operation.log.OperationLog
-import org.apache.kyuubi.session.Session
+import org.apache.kyuubi.session.{AbstractSession, Session}
abstract class SparkOperation(session: Session)
extends AbstractOperation(session) {
@@ -49,7 +49,7 @@ abstract class SparkOperation(session: Session)
}.getOrElse(ZoneId.systemDefault())
}
- protected var iter: FetchIterator[Row] = _
+ protected var iter: FetchIterator[_] = _
protected var result: DataFrame = _
@@ -146,7 +146,8 @@ abstract class SparkOperation(session: Session)
}
}
- override def getResultSetSchema: TTableSchema = SchemaHelper.toTTableSchema(resultSchema)
+ override def getResultSetSchema: TTableSchema =
+ SchemaHelper.toTTableSchema(resultSchema, timeZone.toString)
override def getNextRowSet(order: FetchOrientation, rowSetSize: Int): TRowSet =
withLocalProperties {
@@ -160,9 +161,22 @@ abstract class SparkOperation(session: Session)
case FETCH_PRIOR => iter.fetchPrior(rowSetSize);
case FETCH_FIRST => iter.fetchAbsolute(0);
}
- val taken = iter.take(rowSetSize)
resultRowSet =
- RowSet.toTRowSet(taken.toList, resultSchema, getProtocolVersion, timeZone)
+ if (arrowEnabled) {
+ if (iter.hasNext) {
+ val taken = iter.next().asInstanceOf[Array[Byte]]
+ RowSet.toTRowSet(taken, getProtocolVersion)
+ } else {
+ RowSet.emptyTRowSet()
+ }
+ } else {
+ val taken = iter.take(rowSetSize)
+ RowSet.toTRowSet(
+ taken.toList.asInstanceOf[List[Row]],
+ resultSchema,
+ getProtocolVersion,
+ timeZone)
+ }
resultRowSet.setStartRowOffset(iter.getPosition)
} catch onError(cancel = true)
@@ -170,6 +184,15 @@ abstract class SparkOperation(session: Session)
}
override def shouldRunAsync: Boolean = false
+
+ protected def arrowEnabled(): Boolean = {
+ // normalized config is required, to pass unit test
+ session.asInstanceOf[AbstractSession].normalizedConf
+ .getOrElse("kyuubi.operation.result.codec", "simple")
+ .equalsIgnoreCase("arrow") &&
+ // TODO: (fchen) make all operation support arrow
+ getClass.getCanonicalName == classOf[ExecuteStatement].getCanonicalName
+ }
}
object SparkOperation {
diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/RowSet.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/RowSet.scala
index 6e4257f7a..e240ed8e1 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/RowSet.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/RowSet.scala
@@ -34,6 +34,27 @@ import org.apache.kyuubi.util.RowSetUtils._
object RowSet {
+ def toTRowSet(
+ bytes: Array[Byte],
+ protocolVersion: TProtocolVersion): TRowSet = {
+ if (protocolVersion.getValue < TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V6.getValue) {
+ throw new UnsupportedOperationException
+ } else {
+ toColumnBasedSet(bytes)
+ }
+ }
+
+ def emptyTRowSet(): TRowSet = {
+ new TRowSet(0, new java.util.ArrayList[TRow](0))
+ }
+
+ def toColumnBasedSet(data: Array[Byte]): TRowSet = {
+ val tRowSet = new TRowSet(0, new java.util.ArrayList[TRow](1))
+ val tColumn = toTColumn(data)
+ tRowSet.addToColumns(tColumn)
+ tRowSet
+ }
+
def toTRowSet(
rows: Seq[Row],
schema: StructType,
@@ -279,4 +300,11 @@ object RowSet {
other.toString
}
}
+
+ private def toTColumn(data: Array[Byte]): TColumn = {
+ val values = new java.util.ArrayList[ByteBuffer](1)
+ values.add(ByteBuffer.wrap(data))
+ val nulls = new java.util.BitSet()
+ TColumn.binaryVal(new TBinaryColumn(values, nulls))
+ }
}
diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelper.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelper.scala
index 29d720c53..8db46e2b7 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelper.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelper.scala
@@ -69,40 +69,42 @@ object SchemaHelper {
throw new IllegalArgumentException(s"Unrecognized type name: ${other.catalogString}")
}
- def toTTypeQualifiers(typ: DataType): TTypeQualifiers = {
+ def toTTypeQualifiers(typ: DataType, timeZone: String): TTypeQualifiers = {
val ret = new TTypeQualifiers()
val qualifiers = typ match {
case d: DecimalType =>
Map(
TCLIServiceConstants.PRECISION -> TTypeQualifierValue.i32Value(d.precision),
TCLIServiceConstants.SCALE -> TTypeQualifierValue.i32Value(d.scale)).asJava
+ case _: TimestampType =>
+ Map("session.timeZone" -> TTypeQualifierValue.stringValue(timeZone)).asJava
case _ => Collections.emptyMap[String, TTypeQualifierValue]()
}
ret.setQualifiers(qualifiers)
ret
}
- def toTTypeDesc(typ: DataType): TTypeDesc = {
+ def toTTypeDesc(typ: DataType, timeZone: String): TTypeDesc = {
val typeEntry = new TPrimitiveTypeEntry(toTTypeId(typ))
- typeEntry.setTypeQualifiers(toTTypeQualifiers(typ))
+ typeEntry.setTypeQualifiers(toTTypeQualifiers(typ, timeZone))
val tTypeDesc = new TTypeDesc()
tTypeDesc.addToTypes(TTypeEntry.primitiveEntry(typeEntry))
tTypeDesc
}
- def toTColumnDesc(field: StructField, pos: Int): TColumnDesc = {
+ def toTColumnDesc(field: StructField, pos: Int, timeZone: String): TColumnDesc = {
val tColumnDesc = new TColumnDesc()
tColumnDesc.setColumnName(field.name)
- tColumnDesc.setTypeDesc(toTTypeDesc(field.dataType))
+ tColumnDesc.setTypeDesc(toTTypeDesc(field.dataType, timeZone))
tColumnDesc.setComment(field.getComment().getOrElse(""))
tColumnDesc.setPosition(pos)
tColumnDesc
}
- def toTTableSchema(schema: StructType): TTableSchema = {
+ def toTTableSchema(schema: StructType, timeZone: String): TTableSchema = {
val tTableSchema = new TTableSchema()
schema.zipWithIndex.foreach { case (f, i) =>
- tTableSchema.addToColumns(toTColumnDesc(f, i))
+ tTableSchema.addToColumns(toTColumnDesc(f, i, timeZone))
}
tTableSchema
}
diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala
new file mode 100644
index 000000000..23f7df213
--- /dev/null
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.kyuubi
+
+import java.time.ZoneId
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructField, StructType}
+
+import org.apache.kyuubi.engine.spark.schema.RowSet
+
+object SparkDatasetHelper {
+ def toArrowBatchRdd[T](ds: Dataset[T]): RDD[Array[Byte]] = {
+ ds.toArrowBatchRdd
+ }
+
+ def convertTopLevelComplexTypeToHiveString(df: DataFrame): DataFrame = {
+ val timeZone = ZoneId.of(df.sparkSession.sessionState.conf.sessionLocalTimeZone)
+
+ val quotedCol = (name: String) => col(quoteIfNeeded(name))
+
+ // an udf to call `RowSet.toHiveString` on complex types(struct/array/map).
+ val toHiveStringUDF = udf[String, Row, String]((row, schemaDDL) => {
+ val dt = DataType.fromDDL(schemaDDL)
+ dt match {
+ case StructType(Array(StructField(_, st: StructType, _, _))) =>
+ RowSet.toHiveString((row, st), timeZone)
+ case StructType(Array(StructField(_, at: ArrayType, _, _))) =>
+ RowSet.toHiveString((row.toSeq.head, at), timeZone)
+ case StructType(Array(StructField(_, mt: MapType, _, _))) =>
+ RowSet.toHiveString((row.toSeq.head, mt), timeZone)
+ case _ =>
+ throw new UnsupportedOperationException
+ }
+ })
+
+ val cols = df.schema.map {
+ case sf @ StructField(name, _: StructType, _, _) =>
+ toHiveStringUDF(quotedCol(name), lit(sf.toDDL)).as(name)
+ case sf @ StructField(name, (_: MapType | _: ArrayType), _, _) =>
+ toHiveStringUDF(struct(quotedCol(name)), lit(sf.toDDL)).as(name)
+ case StructField(name, _, _, _) => quotedCol(name)
+ }
+ df.select(cols: _*)
+ }
+
+ /**
+ * Fork from Apache Spark-3.3.1 org.apache.spark.sql.catalyst.util.quoteIfNeeded to adapt to
+ * Spark-3.1.x
+ */
+ def quoteIfNeeded(part: String): String = {
+ if (part.matches("[a-zA-Z0-9_]+") && !part.matches("\\d+")) {
+ part
+ } else {
+ s"`${part.replace("`", "``")}`"
+ }
+ }
+}
diff --git a/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/operation/SparkArrowbasedOperationSuite.scala b/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/operation/SparkArrowbasedOperationSuite.scala
new file mode 100644
index 000000000..7145add65
--- /dev/null
+++ b/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/operation/SparkArrowbasedOperationSuite.scala
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.engine.spark.operation
+
+import org.apache.kyuubi.engine.spark.WithSparkSQLEngine
+import org.apache.kyuubi.operation.SparkDataTypeTests
+
+class SparkArrowbasedOperationSuite extends WithSparkSQLEngine with SparkDataTypeTests {
+
+ override protected def jdbcUrl: String = getJdbcUrl
+
+ override def withKyuubiConf: Map[String, String] = Map.empty
+
+ override def jdbcVars: Map[String, String] = {
+ Map("kyuubi.operation.result.codec" -> resultCodec)
+ }
+
+ override def resultCodec: String = "arrow"
+
+ test("make sure kyuubi.operation.result.codec=arrow") {
+ withJdbcStatement() { statement =>
+ val resultSet =
+ statement.executeQuery("SELECT '${hivevar:kyuubi.operation.result.codec}' AS col")
+ assert(resultSet.next())
+ assert(resultSet.getString("col") === "arrow")
+ }
+ }
+}
diff --git a/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/operation/SparkOperationSuite.scala b/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/operation/SparkOperationSuite.scala
index 8d3e1d7ac..9508ad9bb 100644
--- a/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/operation/SparkOperationSuite.scala
+++ b/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/operation/SparkOperationSuite.scala
@@ -92,14 +92,12 @@ class SparkOperationSuite extends WithSparkSQLEngine with HiveMetadataTests with
.add("c17", "struct<X: string>", nullable = true, "17")
// since spark3.3.0
- if (SPARK_ENGINE_MAJOR_MINOR_VERSION._1 > 3 ||
- (SPARK_ENGINE_MAJOR_MINOR_VERSION._1 == 3 && SPARK_ENGINE_MAJOR_MINOR_VERSION._2 >= 3)) {
+ if (SPARK_ENGINE_VERSION >= "3.3") {
schema = schema.add("c18", "interval day", nullable = true, "18")
.add("c19", "interval year", nullable = true, "19")
}
// since spark3.4.0
- if (SPARK_ENGINE_MAJOR_MINOR_VERSION._1 > 3 ||
- (SPARK_ENGINE_MAJOR_MINOR_VERSION._1 == 3 && SPARK_ENGINE_MAJOR_MINOR_VERSION._2 >= 4)) {
+ if (SPARK_ENGINE_VERSION >= "3.4") {
schema = schema.add("c20", "timestamp_ntz", nullable = true, "20")
}
diff --git a/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelperSuite.scala b/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelperSuite.scala
index 3e0f61207..6bd0364f7 100644
--- a/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelperSuite.scala
+++ b/externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelperSuite.scala
@@ -17,6 +17,8 @@
package org.apache.kyuubi.engine.spark.schema
+import java.time.ZoneId
+
import scala.collection.JavaConverters._
import org.apache.hive.service.rpc.thrift.{TCLIServiceConstants, TTypeId}
@@ -75,20 +77,30 @@ class SchemaHelperSuite extends KyuubiFunSuite {
}
test("toTTypeQualifiers") {
- val qualifiers = toTTypeQualifiers(outerSchema(9).dataType)
+
+ // decimal
+ val qualifiers = toTTypeQualifiers(outerSchema(9).dataType, "")
val q = qualifiers.getQualifiers
assert(q.size() === 2)
assert(q.get(TCLIServiceConstants.PRECISION).getI32Value === 10)
assert(q.get(TCLIServiceConstants.SCALE).getI32Value === 8)
+ // timestamp
+ val timeZone = ZoneId.systemDefault().toString
+ val qualifiers2 = toTTypeQualifiers(outerSchema(11).dataType, timeZone)
+ val q2 = qualifiers2.getQualifiers
+ assert(q2.size() === 1)
+ assert(q2.get("session.timeZone").getStringValue === timeZone)
+
outerSchema.foreach {
- case f if f.dataType == DecimalType(10, 8) =>
- case f => assert(toTTypeQualifiers(f.dataType).getQualifiers.isEmpty)
+ case f if f.dataType == DecimalType(10, 8) || f.dataType == TimestampType =>
+ case f => assert(toTTypeQualifiers(f.dataType, "").getQualifiers.isEmpty)
}
}
test("toTTableSchema") {
- val tTableSchema = toTTableSchema(outerSchema)
+ val timeZone = ZoneId.systemDefault().toString
+ val tTableSchema = toTTableSchema(outerSchema, timeZone)
assert(tTableSchema.getColumnsSize === outerSchema.size)
val iter = tTableSchema.getColumns
@@ -102,6 +114,8 @@ class SchemaHelperSuite extends KyuubiFunSuite {
if (pos == 9) {
assert(qualifiers.get(TCLIServiceConstants.PRECISION).getI32Value === 10)
assert(qualifiers.get(TCLIServiceConstants.SCALE).getI32Value === 8)
+ } else if (pos == 11) {
+ assert(qualifiers.get("session.timeZone").getStringValue === timeZone)
} else {
assert(qualifiers.isEmpty)
}
diff --git a/kyuubi-common/src/main/scala/org/apache/kyuubi/engine/SemanticVersion.scala b/kyuubi-common/src/main/scala/org/apache/kyuubi/engine/SemanticVersion.scala
index 7eedf0bb8..a36896ac9 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/engine/SemanticVersion.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/engine/SemanticVersion.scala
@@ -23,6 +23,14 @@ package org.apache.kyuubi.engine
*/
case class SemanticVersion(majorVersion: Int, minorVersion: Int) {
+ def <=(targetVersionString: String): Boolean = isVersionAtMost(targetVersionString)
+
+ def >=(targetVersionString: String): Boolean = isVersionAtLeast(targetVersionString)
+
+ def >(targetVersionString: String): Boolean = !isVersionAtMost(targetVersionString)
+
+ def <(targetVersionString: String): Boolean = !isVersionAtLeast(targetVersionString)
+
def isVersionAtMost(targetVersionString: String): Boolean = {
this.compareVersion(
targetVersionString,
diff --git a/kyuubi-common/src/test/scala/org/apache/kyuubi/operation/SparkDataTypeTests.scala b/kyuubi-common/src/test/scala/org/apache/kyuubi/operation/SparkDataTypeTests.scala
new file mode 100644
index 000000000..2ffe13063
--- /dev/null
+++ b/kyuubi-common/src/test/scala/org/apache/kyuubi/operation/SparkDataTypeTests.scala
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.kyuubi.engine.SemanticVersion
+
+trait SparkDataTypeTests extends HiveJDBCTestHelper {
+ protected lazy val SPARK_ENGINE_VERSION = sparkEngineMajorMinorVersion
+
+ def resultCodec: String = "simple"
+
+ test("execute statement - select null") {
+ assume(resultCodec == "simple" || (resultCodec == "arrow" && SPARK_ENGINE_VERSION >= "3.2"))
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT NULL AS col")
+ assert(resultSet.next())
+ assert(resultSet.getString("col") === null)
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.NULL)
+ assert(metaData.getPrecision(1) === 0)
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+
+ test("execute statement - select boolean") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT false AS col")
+ assert(resultSet.next())
+ assert(!resultSet.getBoolean("col"))
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.BOOLEAN)
+ assert(metaData.getPrecision(1) === 1)
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+
+ test("execute statement - select tinyint") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT 1Y AS col")
+ assert(resultSet.next())
+ assert(resultSet.getByte("col") === 1.toByte)
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.TINYINT)
+ assert(metaData.getPrecision(1) === 3)
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+
+ test("execute statement - select smallint") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT 1S AS col")
+ assert(resultSet.next())
+ assert(resultSet.getShort("col") === 1.toShort)
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.SMALLINT)
+ assert(metaData.getPrecision(1) === 5)
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+
+ test("execute statement - select int") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT 4 AS col")
+ assert(resultSet.next())
+ assert(resultSet.getInt("col") === 4)
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.INTEGER)
+ assert(metaData.getPrecision(1) === 10)
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+
+ test("execute statement - select long") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT 4L AS col")
+ assert(resultSet.next())
+ assert(resultSet.getLong("col") === 4L)
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.BIGINT)
+ assert(metaData.getPrecision(1) === 19)
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+
+ test("execute statement - select float") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT cast(1.2 as float) AS col")
+ assert(resultSet.next())
+ assert(resultSet.getFloat("col") === 1.2f)
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.FLOAT)
+ assert(metaData.getPrecision(1) === 7)
+ assert(metaData.getScale(1) === 7)
+ }
+ }
+
+ test("execute statement - select double") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT 4.2D AS col")
+ assert(resultSet.next())
+ assert(resultSet.getDouble("col") === 4.2d)
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.DOUBLE)
+ assert(metaData.getPrecision(1) === 15)
+ assert(metaData.getScale(1) === 15)
+ }
+ }
+
+ test("execute statement - select string") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT 'kentyao' AS col")
+ assert(resultSet.next())
+ assert(resultSet.getString("col") === "kentyao")
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.VARCHAR)
+ assert(metaData.getPrecision(1) === Int.MaxValue)
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+
+ test("execute statement - select binary") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT cast('kyuubi' as binary) AS col")
+ assert(resultSet.next())
+ assert(resultSet.getObject("col") === "kyuubi".getBytes)
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.BINARY)
+ assert(metaData.getPrecision(1) === Int.MaxValue)
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+
+ test("execute statement - select date") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT DATE '2018-11-17' AS col")
+ assert(resultSet.next())
+ assert(resultSet.getDate("col") === Date.valueOf("2018-11-17"))
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.DATE)
+ assert(metaData.getPrecision(1) === 10)
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+
+ test("execute statement - select timestamp") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SELECT TIMESTAMP '2018-11-17 13:33:33' AS col")
+ assert(resultSet.next())
+ assert(resultSet.getTimestamp("col") === Timestamp.valueOf("2018-11-17 13:33:33"))
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.TIMESTAMP)
+ assert(metaData.getPrecision(1) === 29)
+ assert(metaData.getScale(1) === 9)
+ }
+ }
+
+ test("execute statement - select timestamp_ntz") {
+ assume(SPARK_ENGINE_VERSION >= "3.4")
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery(
+ "SELECT make_timestamp_ntz(2022, 03, 24, 18, 08, 31.800) AS col")
+ assert(resultSet.next())
+ assert(resultSet.getTimestamp("col") === Timestamp.valueOf("2022-03-24 18:08:31.800"))
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.TIMESTAMP)
+ assert(metaData.getPrecision(1) === 29)
+ assert(metaData.getScale(1) === 9)
+ }
+ }
+
+ test("execute statement - select daytime interval") {
+ assume(resultCodec == "simple" || (resultCodec == "arrow" && SPARK_ENGINE_VERSION >= "3.3"))
+ withJdbcStatement() { statement =>
+ Map(
+ "interval 1 day 1 hour -60 minutes 30 seconds" ->
+ Tuple2("1 00:00:30.000000000", "1 days 30 seconds"),
+ "interval 30 seconds 12345 milliseconds" ->
+ Tuple2("0 00:00:42.345000000", "42.345 seconds"),
+ "interval 1 hour 59 minutes 30 seconds 12345 milliseconds" ->
+ Tuple2("0 01:59:42.345000000", "1 hours 59 minutes 42.345 seconds"),
+ "-interval 2 day" -> Tuple2("-2 00:00:00.000000000", "-2 days"),
+ "interval 59 minutes 30 seconds 12345 milliseconds" ->
+ Tuple2("0 00:59:42.345000000", "59 minutes 42.345 seconds"),
+ "interval 25 hour" -> Tuple2("1 01:00:00.000000000", "25 hours"),
+ "interval 1 hour 62 minutes" -> Tuple2("0 02:02:00.000000000", "2 hours 2 minutes"),
+ "interval 1 day 1 hour 59 minutes 30 seconds 12345 milliseconds" ->
+ Tuple2("1 01:59:42.345000000", "1 days 1 hours 59 minutes 42.345 seconds"),
+ "interval 1 day 1 hour -60 minutes" -> Tuple2("1 00:00:00.000000000", "1 days"),
+ "INTERVAL 30 SECONDS" -> Tuple2("0 00:00:30.000000000", "30 seconds"),
+ "interval -60 minutes 30 seconds" ->
+ Tuple2("-0 00:59:30.000000000", "-59 minutes -30 seconds"),
+ "-interval 200 day" -> Tuple2("-200 00:00:00.000000000", "-200 days"),
+ "interval 1 hour -60 minutes 30 seconds" -> Tuple2("0 00:00:30.000000000", "30 seconds"),
+ "interval 62 minutes" -> Tuple2("0 01:02:00.000000000", "1 hours 2 minutes"),
+ "interval 1 day 1 hour" -> Tuple2("1 01:00:00.000000000", "1 days 1 hours")).foreach {
+ kv => // value -> result pair
+ val resultSet = statement.executeQuery(s"SELECT ${kv._1} AS col")
+ assert(resultSet.next())
+ val result = resultSet.getString("col")
+ val metaData = resultSet.getMetaData
+ if (SPARK_ENGINE_VERSION < "3.2") {
+ // for spark 3.1 and backwards
+ assert(result === kv._2._2)
+ assert(metaData.getPrecision(1) === Int.MaxValue)
+ assert(resultSet.getMetaData.getColumnType(1) === java.sql.Types.VARCHAR)
+ } else {
+ assert(result === kv._2._1)
+ assert(metaData.getPrecision(1) === 29)
+ assert(resultSet.getMetaData.getColumnType(1) === java.sql.Types.OTHER)
+ }
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+ }
+
+ test("execute statement - select year/month interval") {
+ assume(resultCodec == "simple" || (resultCodec == "arrow" && SPARK_ENGINE_VERSION >= "3.3"))
+ withJdbcStatement() { statement =>
+ Map(
+ "INTERVAL 2022 YEAR" -> Tuple2("2022-0", "2022 years"),
+ "INTERVAL '2021-07' YEAR TO MONTH" -> Tuple2("2021-7", "2021 years 7 months"),
+ "INTERVAL 3 MONTH" -> Tuple2("0-3", "3 months"),
+ "INTERVAL 241 MONTH" -> Tuple2("20-1", "20 years 1 months"),
+ "INTERVAL -1 year -25 MONTH" -> Tuple2("-3-1", "-3 years -1 months"),
+ "INTERVAL 3 year -25 MONTH" -> Tuple2("0-11", "11 months")).foreach { kv =>
+ val resultSet = statement.executeQuery(s"SELECT ${kv._1} AS col")
+ assert(resultSet.next())
+ val result = resultSet.getString("col")
+ val metaData = resultSet.getMetaData
+ if (SPARK_ENGINE_VERSION < "3.2") {
+ // for spark 3.1 and backwards
+ assert(result === kv._2._2)
+ assert(metaData.getPrecision(1) === Int.MaxValue)
+ assert(resultSet.getMetaData.getColumnType(1) === java.sql.Types.VARCHAR)
+ } else {
+ assert(result === kv._2._1)
+ assert(metaData.getPrecision(1) === 11)
+ assert(resultSet.getMetaData.getColumnType(1) === java.sql.Types.OTHER)
+ }
+ assert(metaData.getScale(1) === 0)
+ }
+ }
+ }
+
+ test("execute statement - select array") {
+ assume(resultCodec == "simple" || (resultCodec == "arrow" && SPARK_ENGINE_VERSION >= "3.2"))
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery(
+ "SELECT array() AS col1, array(1) AS col2, array(null) AS col3")
+ assert(resultSet.next())
+ assert(resultSet.getObject("col1") === "[]")
+ assert(resultSet.getObject("col2") === "[1]")
+ assert(resultSet.getObject("col3") === "[null]")
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.ARRAY)
+ assert(metaData.getPrecision(1) === Int.MaxValue)
+ assert(metaData.getPrecision(2) == Int.MaxValue)
+ assert(metaData.getScale(1) == 0)
+ assert(metaData.getScale(2) == 0)
+ }
+ }
+
+ test("execute statement - select map") {
+ assume(resultCodec == "simple" || (resultCodec == "arrow" && SPARK_ENGINE_VERSION >= "3.2"))
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery(
+ "SELECT map() AS col1, map(1, 2, 3, 4) AS col2, map(1, null) AS col3")
+ assert(resultSet.next())
+ assert(resultSet.getObject("col1") === "{}")
+ assert(resultSet.getObject("col2") === "{1:2,3:4}")
+ assert(resultSet.getObject("col3") === "{1:null}")
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.JAVA_OBJECT)
+ assert(metaData.getPrecision(1) === Int.MaxValue)
+ assert(metaData.getPrecision(2) == Int.MaxValue)
+ assert(metaData.getScale(1) == 0)
+ assert(metaData.getScale(2) == 0)
+ }
+ }
+
+ test("execute statement - select struct") {
+ assume(resultCodec == "simple" || (resultCodec == "arrow" && SPARK_ENGINE_VERSION >= "3.2"))
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery(
+ "SELECT struct('1', '2') AS col1," +
+ " named_struct('a', 2, 'b', 4) AS col2," +
+ " named_struct('a', null, 'b', null) AS col3")
+ assert(resultSet.next())
+ assert(resultSet.getObject("col1") === """{"col1":"1","col2":"2"}""")
+ assert(resultSet.getObject("col2") === """{"a":2,"b":4}""")
+ assert(resultSet.getObject("col3") === """{"a":null,"b":null}""")
+
+ val metaData = resultSet.getMetaData
+ assert(metaData.getColumnType(1) === java.sql.Types.STRUCT)
+ assert(metaData.getPrecision(1) === Int.MaxValue)
+ assert(metaData.getPrecision(2) == Int.MaxValue)
+ assert(metaData.getScale(1) == 0)
+ assert(metaData.getScale(2) == 0)
+ }
+ }
+
+ def sparkEngineMajorMinorVersion: SemanticVersion = {
+ var sparkRuntimeVer = ""
+ withJdbcStatement() { stmt =>
+ val result = stmt.executeQuery("SELECT version()")
+ assert(result.next())
+ sparkRuntimeVer = result.getString(1)
+ assert(!result.next())
+ }
+ SemanticVersion(sparkRuntimeVer)
+ }
+}
diff --git a/kyuubi-common/src/test/scala/org/apache/kyuubi/operation/SparkQueryTests.scala b/kyuubi-common/src/test/scala/org/apache/kyuubi/operation/SparkQueryTests.scala
index d227f4002..8b01252f1 100644
--- a/kyuubi-common/src/test/scala/org/apache/kyuubi/operation/SparkQueryTests.scala
+++ b/kyuubi-common/src/test/scala/org/apache/kyuubi/operation/SparkQueryTests.scala
@@ -17,7 +17,7 @@
package org.apache.kyuubi.operation
-import java.sql.{Date, SQLException, SQLTimeoutException, Timestamp}
+import java.sql.{SQLException, SQLTimeoutException}
import java.util.UUID
import scala.collection.JavaConverters._
@@ -28,302 +28,12 @@ import org.apache.hive.service.rpc.thrift.{TExecuteStatementReq, TFetchResultsRe
import org.apache.kyuubi.{KYUUBI_VERSION, Utils}
import org.apache.kyuubi.config.KyuubiConf
-import org.apache.kyuubi.engine.SemanticVersion
import org.apache.kyuubi.util.SparkVersionUtil.isSparkVersionAtLeast
-trait SparkQueryTests extends HiveJDBCTestHelper {
-
- protected lazy val SPARK_ENGINE_MAJOR_MINOR_VERSION: (Int, Int) = sparkEngineMajorMinorVersion
+trait SparkQueryTests extends SparkDataTypeTests with HiveJDBCTestHelper {
protected lazy val httpMode = false
- test("execute statement - select null") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT NULL AS col")
- assert(resultSet.next())
- assert(resultSet.getString("col") === null)
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.NULL)
- assert(metaData.getPrecision(1) === 0)
- assert(metaData.getScale(1) === 0)
- }
- }
-
- test("execute statement - select boolean") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT false AS col")
- assert(resultSet.next())
- assert(!resultSet.getBoolean("col"))
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.BOOLEAN)
- assert(metaData.getPrecision(1) === 1)
- assert(metaData.getScale(1) === 0)
- }
- }
-
- test("execute statement - select tinyint") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT 1Y AS col")
- assert(resultSet.next())
- assert(resultSet.getByte("col") === 1.toByte)
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.TINYINT)
- assert(metaData.getPrecision(1) === 3)
- assert(metaData.getScale(1) === 0)
- }
- }
-
- test("execute statement - select smallint") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT 1S AS col")
- assert(resultSet.next())
- assert(resultSet.getShort("col") === 1.toShort)
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.SMALLINT)
- assert(metaData.getPrecision(1) === 5)
- assert(metaData.getScale(1) === 0)
- }
- }
-
- test("execute statement - select int") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT 4 AS col")
- assert(resultSet.next())
- assert(resultSet.getInt("col") === 4)
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.INTEGER)
- assert(metaData.getPrecision(1) === 10)
- assert(metaData.getScale(1) === 0)
- }
- }
-
- test("execute statement - select long") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT 4L AS col")
- assert(resultSet.next())
- assert(resultSet.getLong("col") === 4L)
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.BIGINT)
- assert(metaData.getPrecision(1) === 19)
- assert(metaData.getScale(1) === 0)
- }
- }
-
- test("execute statement - select float") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT cast(1.2 as float) AS col")
- assert(resultSet.next())
- assert(resultSet.getFloat("col") === 1.2f)
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.FLOAT)
- assert(metaData.getPrecision(1) === 7)
- assert(metaData.getScale(1) === 7)
- }
- }
-
- test("execute statement - select double") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT 4.2D AS col")
- assert(resultSet.next())
- assert(resultSet.getDouble("col") === 4.2d)
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.DOUBLE)
- assert(metaData.getPrecision(1) === 15)
- assert(metaData.getScale(1) === 15)
- }
- }
-
- test("execute statement - select string") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT 'kentyao' AS col")
- assert(resultSet.next())
- assert(resultSet.getString("col") === "kentyao")
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.VARCHAR)
- assert(metaData.getPrecision(1) === Int.MaxValue)
- assert(metaData.getScale(1) === 0)
- }
- }
-
- test("execute statement - select binary") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT cast('kyuubi' as binary) AS col")
- assert(resultSet.next())
- assert(resultSet.getObject("col") === "kyuubi".getBytes)
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.BINARY)
- assert(metaData.getPrecision(1) === Int.MaxValue)
- assert(metaData.getScale(1) === 0)
- }
- }
-
- test("execute statement - select date") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT DATE '2018-11-17' AS col")
- assert(resultSet.next())
- assert(resultSet.getDate("col") === Date.valueOf("2018-11-17"))
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.DATE)
- assert(metaData.getPrecision(1) === 10)
- assert(metaData.getScale(1) === 0)
- }
- }
-
- test("execute statement - select timestamp") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT TIMESTAMP '2018-11-17 13:33:33' AS col")
- assert(resultSet.next())
- assert(resultSet.getTimestamp("col") === Timestamp.valueOf("2018-11-17 13:33:33"))
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.TIMESTAMP)
- assert(metaData.getPrecision(1) === 29)
- assert(metaData.getScale(1) === 9)
- }
- }
-
- test("execute statement - select timestamp_ntz") {
- assume(SPARK_ENGINE_MAJOR_MINOR_VERSION._1 > 3 ||
- (SPARK_ENGINE_MAJOR_MINOR_VERSION._1 == 3 && SPARK_ENGINE_MAJOR_MINOR_VERSION._2 >= 4))
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery(
- "SELECT make_timestamp_ntz(2022, 03, 24, 18, 08, 31.800) AS col")
- assert(resultSet.next())
- assert(resultSet.getTimestamp("col") === Timestamp.valueOf("2022-03-24 18:08:31.800"))
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.TIMESTAMP)
- assert(metaData.getPrecision(1) === 29)
- assert(metaData.getScale(1) === 9)
- }
- }
-
- test("execute statement - select daytime interval") {
- withJdbcStatement() { statement =>
- Map(
- "interval 1 day 1 hour -60 minutes 30 seconds" ->
- Tuple2("1 00:00:30.000000000", "1 days 30 seconds"),
- "interval 30 seconds 12345 milliseconds" ->
- Tuple2("0 00:00:42.345000000", "42.345 seconds"),
- "interval 1 hour 59 minutes 30 seconds 12345 milliseconds" ->
- Tuple2("0 01:59:42.345000000", "1 hours 59 minutes 42.345 seconds"),
- "-interval 2 day" -> Tuple2("-2 00:00:00.000000000", "-2 days"),
- "interval 59 minutes 30 seconds 12345 milliseconds" ->
- Tuple2("0 00:59:42.345000000", "59 minutes 42.345 seconds"),
- "interval 25 hour" -> Tuple2("1 01:00:00.000000000", "25 hours"),
- "interval 1 hour 62 minutes" -> Tuple2("0 02:02:00.000000000", "2 hours 2 minutes"),
- "interval 1 day 1 hour 59 minutes 30 seconds 12345 milliseconds" ->
- Tuple2("1 01:59:42.345000000", "1 days 1 hours 59 minutes 42.345 seconds"),
- "interval 1 day 1 hour -60 minutes" -> Tuple2("1 00:00:00.000000000", "1 days"),
- "INTERVAL 30 SECONDS" -> Tuple2("0 00:00:30.000000000", "30 seconds"),
- "interval -60 minutes 30 seconds" ->
- Tuple2("-0 00:59:30.000000000", "-59 minutes -30 seconds"),
- "-interval 200 day" -> Tuple2("-200 00:00:00.000000000", "-200 days"),
- "interval 1 hour -60 minutes 30 seconds" -> Tuple2("0 00:00:30.000000000", "30 seconds"),
- "interval 62 minutes" -> Tuple2("0 01:02:00.000000000", "1 hours 2 minutes"),
- "interval 1 day 1 hour" -> Tuple2("1 01:00:00.000000000", "1 days 1 hours")).foreach {
- kv => // value -> result pair
- val resultSet = statement.executeQuery(s"SELECT ${kv._1} AS col")
- assert(resultSet.next())
- val result = resultSet.getString("col")
- val metaData = resultSet.getMetaData
- if (SPARK_ENGINE_MAJOR_MINOR_VERSION._1 == 3
- && SPARK_ENGINE_MAJOR_MINOR_VERSION._2 < 2) {
- // for spark 3.1 and backwards
- assert(result === kv._2._2)
- assert(metaData.getPrecision(1) === Int.MaxValue)
- assert(resultSet.getMetaData.getColumnType(1) === java.sql.Types.VARCHAR)
- } else {
- assert(result === kv._2._1)
- assert(metaData.getPrecision(1) === 29)
- assert(resultSet.getMetaData.getColumnType(1) === java.sql.Types.OTHER)
- }
- assert(metaData.getScale(1) === 0)
- }
- }
- }
-
- test("execute statement - select year/month interval") {
- withJdbcStatement() { statement =>
- Map(
- "INTERVAL 2022 YEAR" -> Tuple2("2022-0", "2022 years"),
- "INTERVAL '2021-07' YEAR TO MONTH" -> Tuple2("2021-7", "2021 years 7 months"),
- "INTERVAL 3 MONTH" -> Tuple2("0-3", "3 months"),
- "INTERVAL 241 MONTH" -> Tuple2("20-1", "20 years 1 months"),
- "INTERVAL -1 year -25 MONTH" -> Tuple2("-3-1", "-3 years -1 months"),
- "INTERVAL 3 year -25 MONTH" -> Tuple2("0-11", "11 months")).foreach { kv =>
- val resultSet = statement.executeQuery(s"SELECT ${kv._1} AS col")
- assert(resultSet.next())
- val result = resultSet.getString("col")
- val metaData = resultSet.getMetaData
- if (SPARK_ENGINE_MAJOR_MINOR_VERSION._1 == 3
- && SPARK_ENGINE_MAJOR_MINOR_VERSION._2 < 2) {
- // for spark 3.1 and backwards
- assert(result === kv._2._2)
- assert(metaData.getPrecision(1) === Int.MaxValue)
- assert(resultSet.getMetaData.getColumnType(1) === java.sql.Types.VARCHAR)
- } else {
- assert(result === kv._2._1)
- assert(metaData.getPrecision(1) === 11)
- assert(resultSet.getMetaData.getColumnType(1) === java.sql.Types.OTHER)
- }
- assert(metaData.getScale(1) === 0)
- }
- }
- }
-
- test("execute statement - select array") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery(
- "SELECT array() AS col1, array(1) AS col2, array(null) AS col3")
- assert(resultSet.next())
- assert(resultSet.getObject("col1") === "[]")
- assert(resultSet.getObject("col2") === "[1]")
- assert(resultSet.getObject("col3") === "[null]")
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.ARRAY)
- assert(metaData.getPrecision(1) === Int.MaxValue)
- assert(metaData.getPrecision(2) == Int.MaxValue)
- assert(metaData.getScale(1) == 0)
- assert(metaData.getScale(2) == 0)
- }
- }
-
- test("execute statement - select map") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery(
- "SELECT map() AS col1, map(1, 2, 3, 4) AS col2, map(1, null) AS col3")
- assert(resultSet.next())
- assert(resultSet.getObject("col1") === "{}")
- assert(resultSet.getObject("col2") === "{1:2,3:4}")
- assert(resultSet.getObject("col3") === "{1:null}")
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.JAVA_OBJECT)
- assert(metaData.getPrecision(1) === Int.MaxValue)
- assert(metaData.getPrecision(2) == Int.MaxValue)
- assert(metaData.getScale(1) == 0)
- assert(metaData.getScale(2) == 0)
- }
- }
-
- test("execute statement - select struct") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery(
- "SELECT struct('1', '2') AS col1," +
- " named_struct('a', 2, 'b', 4) AS col2," +
- " named_struct('a', null, 'b', null) AS col3")
- assert(resultSet.next())
- assert(resultSet.getObject("col1") === """{"col1":"1","col2":"2"}""")
- assert(resultSet.getObject("col2") === """{"a":2,"b":4}""")
- assert(resultSet.getObject("col3") === """{"a":null,"b":null}""")
-
- val metaData = resultSet.getMetaData
- assert(metaData.getColumnType(1) === java.sql.Types.STRUCT)
- assert(metaData.getPrecision(1) === Int.MaxValue)
- assert(metaData.getPrecision(2) == Int.MaxValue)
- assert(metaData.getScale(1) == 0)
- assert(metaData.getScale(2) == 0)
- }
- }
-
test("execute statement - analysis exception") {
val sql = "select date_sub(date'2011-11-11', '1.2')"
val errors = Set(
@@ -699,16 +409,4 @@ trait SparkQueryTests extends HiveJDBCTestHelper {
}
}
}
-
- def sparkEngineMajorMinorVersion: (Int, Int) = {
- var sparkRuntimeVer = ""
- withJdbcStatement() { stmt =>
- val result = stmt.executeQuery("SELECT version()")
- assert(result.next())
- sparkRuntimeVer = result.getString(1)
- assert(!result.next())
- }
- val ver = SemanticVersion(sparkRuntimeVer)
- (ver.majorVersion, ver.minorVersion)
- }
}
diff --git a/kyuubi-hive-jdbc-shaded/pom.xml b/kyuubi-hive-jdbc-shaded/pom.xml
index 55c3cac06..0bfe88922 100644
--- a/kyuubi-hive-jdbc-shaded/pom.xml
+++ b/kyuubi-hive-jdbc-shaded/pom.xml
@@ -71,13 +71,19 @@
<exclude>META-INF/LICENSE.txt</exclude>
<exclude>META-INF/NOTICE.txt</exclude>
<exclude>META-INF/maven/**</exclude>
+ <exclude>META-INF/native-image/**</exclude>
<exclude>LICENSE.txt</exclude>
<exclude>NOTICE.txt</exclude>
<exclude>mozilla/**</exclude>
+ <exclude>arrow-git.properties</exclude>
</excludes>
</filter>
</filters>
<relocations>
+ <relocation>
+ <pattern>codegen</pattern>
+ <shadedPattern>${kyuubi.shade.packageName}.codegen</shadedPattern>
+ </relocation>
<relocation>
<pattern>com.facebook</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.facebook</shadedPattern>
@@ -87,12 +93,20 @@
<shadedPattern>${kyuubi.shade.packageName}.com.google</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.commons.codec</pattern>
- <shadedPattern>${kyuubi.shade.packageName}.org.apache.commons.codec</shadedPattern>
+ <pattern>com.fasterxml.jackson</pattern>
+ <shadedPattern>${kyuubi.shade.packageName}.com.fasterxml.jackson</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>io.netty</pattern>
+ <shadedPattern>${kyuubi.shade.packageName}.io.netty</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.commons.lang3</pattern>
- <shadedPattern>${kyuubi.shade.packageName}.org.apache.commons.lang3</shadedPattern>
+ <pattern>org.apache.arrow</pattern>
+ <shadedPattern>${kyuubi.shade.packageName}.org.apache.arrow</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.commons</pattern>
+ <shadedPattern>${kyuubi.shade.packageName}.org.apache.commons</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.curator</pattern>
@@ -126,6 +140,11 @@
<goal>shade</goal>
</goals>
<phase>package</phase>
+ <configuration>
+ <transformers>
+ <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"></transformer>
+ </transformers>
+ </configuration>
</execution>
</executions>
</plugin>
diff --git a/kyuubi-hive-jdbc-shaded/src/main/resources/META-INF/NOTICE b/kyuubi-hive-jdbc-shaded/src/main/resources/META-INF/NOTICE
index 27411660a..dd90d9c6f 100644
--- a/kyuubi-hive-jdbc-shaded/src/main/resources/META-INF/NOTICE
+++ b/kyuubi-hive-jdbc-shaded/src/main/resources/META-INF/NOTICE
@@ -54,3 +54,26 @@ Copyright 2006-2010 The Apache Software Foundation.
Apache Zookeeper
Copyright 2009-2019 The Apache Software Foundation.
+
+Apache Arrow
+Copyright 2016-2019 The Apache Software Foundation
+
+--------------------------------------------------------------------------------
+
+This binary artifact contains Jackson JSON processor
+
+Jackson is a high-performance, Free/Open Source JSON processing library.
+It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+been in development since 2007.
+It is currently developed by a community of developers.
+
+## Licensing
+
+Jackson 2.x core and extension components are licensed under Apache License 2.0
+To find the details that apply to this artifact see the accompanying LICENSE file.
+
+## Credits
+
+A list of contributors may be found from CREDITS(-2.x) file, which is included
+in some artifacts (usually source distributions); but is always available
+from the source code management (SCM) system project uses.
diff --git a/kyuubi-hive-jdbc/pom.xml b/kyuubi-hive-jdbc/pom.xml
index 6689e4c8e..4d9648e75 100644
--- a/kyuubi-hive-jdbc/pom.xml
+++ b/kyuubi-hive-jdbc/pom.xml
@@ -35,6 +35,37 @@
</properties>
<dependencies>
+
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-vector</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-memory-netty</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>com.fasterxml.jackson.datatype</groupId>
+ <artifactId>jackson-datatype-jsr310</artifactId>
+ </dependency>
+
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libfb303</artifactId>
@@ -140,7 +171,6 @@
</dependencies>
<build>
-
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/JdbcColumnAttributes.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/JdbcColumnAttributes.java
index 928df51b7..06fb39899 100644
--- a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/JdbcColumnAttributes.java
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/JdbcColumnAttributes.java
@@ -17,9 +17,10 @@
package org.apache.kyuubi.jdbc.hive;
-class JdbcColumnAttributes {
+public class JdbcColumnAttributes {
public int precision = 0;
public int scale = 0;
+ public String timeZone = "";
public JdbcColumnAttributes() {}
@@ -28,8 +29,12 @@ class JdbcColumnAttributes {
this.scale = scale;
}
+ public JdbcColumnAttributes(String timeZone) {
+ this.timeZone = timeZone;
+ }
+
@Override
public String toString() {
- return "(" + precision + "," + scale + ")";
+ return "(" + precision + "," + scale + "," + timeZone + ")";
}
}
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiArrowBasedResultSet.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiArrowBasedResultSet.java
new file mode 100644
index 000000000..e45b65454
--- /dev/null
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiArrowBasedResultSet.java
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.jdbc.hive;
+
+import java.io.ByteArrayInputStream;
+import java.io.InputStream;
+import java.math.BigDecimal;
+import java.math.MathContext;
+import java.nio.charset.StandardCharsets;
+import java.sql.*;
+import java.util.List;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hive.service.rpc.thrift.TTableSchema;
+import org.apache.hive.service.rpc.thrift.TTypeId;
+import org.apache.kyuubi.jdbc.hive.adapter.SQLResultSet;
+import org.apache.kyuubi.jdbc.hive.arrow.ArrowColumnarBatchRow;
+import org.apache.kyuubi.jdbc.hive.arrow.ArrowUtils;
+
+/** Data independent base class which implements the common part of all Kyuubi result sets. */
+public abstract class KyuubiArrowBasedResultSet implements SQLResultSet {
+
+ protected Statement statement = null;
+ protected SQLWarning warningChain = null;
+ protected boolean wasNull = false;
+ protected List<String> columnNames;
+ protected List<String> normalizedColumnNames;
+ protected List<TTypeId> columnTypes;
+ protected List<JdbcColumnAttributes> columnAttributes;
+ private TTableSchema schema;
+
+ // arrow related
+ protected Schema arrowSchema;
+ protected VectorSchemaRoot root;
+ protected ArrowColumnarBatchRow row;
+
+ protected BufferAllocator allocator;
+
+ protected void initArrowSchemaAndAllocator() {
+ if (arrowSchema == null) {
+ throw new IllegalStateException("arrow schema is null");
+ }
+ allocator = ArrowUtils.rootAllocator.newChildAllocator("ArrowResultSet", 0, Long.MAX_VALUE);
+ root = VectorSchemaRoot.create(arrowSchema, allocator);
+ }
+
+ @Override
+ public int findColumn(String columnName) throws SQLException {
+ int columnIndex = 0;
+ boolean findColumn = false;
+ for (String normalizedColumnName : normalizedColumnNames) {
+ ++columnIndex;
+ String[] names = normalizedColumnName.split("\\.");
+ String name = names[names.length - 1];
+ if (name.equalsIgnoreCase(columnName) || normalizedColumnName.equalsIgnoreCase(columnName)) {
+ findColumn = true;
+ break;
+ }
+ }
+ if (!findColumn) {
+ throw new KyuubiSQLException("Could not find " + columnName + " in " + normalizedColumnNames);
+ } else {
+ return columnIndex;
+ }
+ }
+
+ @Override
+ public BigDecimal getBigDecimal(int columnIndex) throws SQLException {
+ Object val = getObject(columnIndex);
+
+ if (val == null || val instanceof BigDecimal) {
+ return (BigDecimal) val;
+ }
+
+ throw new KyuubiSQLException("Illegal conversion");
+ }
+
+ @Override
+ public BigDecimal getBigDecimal(String columnName) throws SQLException {
+ return getBigDecimal(findColumn(columnName));
+ }
+
+ @Override
+ public BigDecimal getBigDecimal(int columnIndex, int scale) throws SQLException {
+ MathContext mc = new MathContext(scale);
+ return getBigDecimal(columnIndex).round(mc);
+ }
+
+ @Override
+ public BigDecimal getBigDecimal(String columnName, int scale) throws SQLException {
+ return getBigDecimal(findColumn(columnName), scale);
+ }
+
+ @Override
+ public InputStream getBinaryStream(int columnIndex) throws SQLException {
+ Object obj = getObject(columnIndex);
+ if (obj == null) {
+ return null;
+ } else if (obj instanceof InputStream) {
+ return (InputStream) obj;
+ } else if (obj instanceof byte[]) {
+ byte[] byteArray = (byte[]) obj;
+ return new ByteArrayInputStream(byteArray);
+ } else if (obj instanceof String) {
+ String str = (String) obj;
+ return new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8));
+ }
+ throw new KyuubiSQLException("Illegal conversion to binary stream from column " + columnIndex);
+ }
+
+ @Override
+ public InputStream getBinaryStream(String columnName) throws SQLException {
+ return getBinaryStream(findColumn(columnName));
+ }
+
+ @Override
+ public boolean getBoolean(int columnIndex) throws SQLException {
+ Object obj = getObject(columnIndex);
+ if (obj instanceof Boolean) {
+ return (Boolean) obj;
+ } else if (obj == null) {
+ return false;
+ } else if (obj instanceof Number) {
+ return ((Number) obj).intValue() != 0;
+ } else if (obj instanceof String) {
+ return !obj.equals("0");
+ }
+ throw new KyuubiSQLException("Cannot convert column " + columnIndex + " to boolean");
+ }
+
+ @Override
+ public boolean getBoolean(String columnName) throws SQLException {
+ return getBoolean(findColumn(columnName));
+ }
+
+ @Override
+ public byte getByte(int columnIndex) throws SQLException {
+ Object obj = getObject(columnIndex);
+ if (obj instanceof Number) {
+ return ((Number) obj).byteValue();
+ } else if (obj == null) {
+ return 0;
+ }
+ throw new KyuubiSQLException("Cannot convert column " + columnIndex + " to byte");
+ }
+
+ @Override
+ public byte getByte(String columnName) throws SQLException {
+ return getByte(findColumn(columnName));
+ }
+
+ @Override
+ public int getConcurrency() throws SQLException {
+ return ResultSet.CONCUR_READ_ONLY;
+ }
+
+ @Override
+ public Date getDate(int columnIndex) throws SQLException {
+ Object obj = getObject(columnIndex);
+ if (obj == null) {
+ return null;
+ }
+ if (obj instanceof Date) {
+ return (Date) obj;
+ }
+ try {
+ if (obj instanceof String) {
+ return Date.valueOf((String) obj);
+ }
+ } catch (Exception e) {
+ throw new KyuubiSQLException("Cannot convert column " + columnIndex + " to date: " + e, e);
+ }
+ // If we fell through to here this is not a valid type conversion
+ throw new KyuubiSQLException(
+ "Cannot convert column " + columnIndex + " to date: Illegal conversion");
+ }
+
+ @Override
+ public Date getDate(String columnName) throws SQLException {
+ return getDate(findColumn(columnName));
+ }
+
+ @Override
+ public double getDouble(int columnIndex) throws SQLException {
+ try {
+ Object obj = getObject(columnIndex);
+ if (obj instanceof Number) {
+ return ((Number) obj).doubleValue();
+ } else if (obj == null) {
+ return 0;
+ } else if (obj instanceof String) {
+ return Double.parseDouble((String) obj);
+ }
+ throw new Exception("Illegal conversion");
+ } catch (Exception e) {
+ throw new KyuubiSQLException("Cannot convert column " + columnIndex + " to double: " + e, e);
+ }
+ }
+
+ @Override
+ public double getDouble(String columnName) throws SQLException {
+ return getDouble(findColumn(columnName));
+ }
+
+ @Override
+ public int getFetchDirection() throws SQLException {
+ return ResultSet.FETCH_FORWARD;
+ }
+
+ @Override
+ public float getFloat(int columnIndex) throws SQLException {
+ try {
+ Object obj = getObject(columnIndex);
+ if (obj instanceof Number) {
+ return ((Number) obj).floatValue();
+ } else if (obj == null) {
+ return 0;
+ } else if (obj instanceof String) {
+ return Float.parseFloat((String) obj);
+ }
+ throw new Exception("Illegal conversion");
+ } catch (Exception e) {
+ throw new KyuubiSQLException("Cannot convert column " + columnIndex + " to float: " + e, e);
+ }
+ }
+
+ @Override
+ public float getFloat(String columnName) throws SQLException {
+ return getFloat(findColumn(columnName));
+ }
+
+ @Override
+ public int getInt(int columnIndex) throws SQLException {
+ try {
+ Object obj = getObject(columnIndex);
+ if (obj instanceof Number) {
+ return ((Number) obj).intValue();
+ } else if (obj == null) {
+ return 0;
+ } else if (obj instanceof String) {
+ return Integer.parseInt((String) obj);
+ }
+ throw new Exception("Illegal conversion");
+ } catch (Exception e) {
+ throw new KyuubiSQLException("Cannot convert column " + columnIndex + " to integer" + e, e);
+ }
+ }
+
+ @Override
+ public int getInt(String columnName) throws SQLException {
+ return getInt(findColumn(columnName));
+ }
+
+ @Override
+ public long getLong(int columnIndex) throws SQLException {
+ try {
+ Object obj = getObject(columnIndex);
+ if (obj instanceof Number) {
+ return ((Number) obj).longValue();
+ } else if (obj == null) {
+ return 0;
+ } else if (obj instanceof String) {
+ return Long.parseLong((String) obj);
+ }
+ throw new Exception("Illegal conversion");
+ } catch (Exception e) {
+ throw new KyuubiSQLException("Cannot convert column " + columnIndex + " to long: " + e, e);
+ }
+ }
+
+ @Override
+ public long getLong(String columnName) throws SQLException {
+ return getLong(findColumn(columnName));
+ }
+
+ @Override
+ public ResultSetMetaData getMetaData() throws SQLException {
+ return new KyuubiResultSetMetaData(columnNames, columnTypes, columnAttributes);
+ }
+
+ private Object getColumnValue(int columnIndex) throws SQLException {
+ if (row == null) {
+ throw new KyuubiSQLException("No row found.");
+ }
+ if (row.numFields() == 0) {
+ throw new KyuubiSQLException("RowSet does not contain any columns!");
+ }
+ if (columnIndex > columnNames.size()) {
+ throw new KyuubiSQLException("Invalid columnIndex: " + columnIndex);
+ }
+ TTypeId columnType = columnTypes.get(columnIndex - 1);
+
+ try {
+ wasNull = row.isNullAt(columnIndex - 1);
+ if (wasNull) {
+ return null;
+ } else {
+ return row.get(columnIndex - 1, columnType);
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ throw new KyuubiSQLException("Unrecognized column type:", e);
+ }
+ }
+
+ @Override
+ public Object getObject(int columnIndex) throws SQLException {
+ return getColumnValue(columnIndex);
+ }
+
+ @Override
+ public Object getObject(String columnName) throws SQLException {
+ return getObject(findColumn(columnName));
+ }
+
+ @Override
+ public short getShort(int columnIndex) throws SQLException {
+ try {
+ Object obj = getObject(columnIndex);
+ if (obj instanceof Number) {
+ return ((Number) obj).shortValue();
+ } else if (obj == null) {
+ return 0;
+ } else if (obj instanceof String) {
+ return Short.parseShort((String) obj);
+ }
+ throw new Exception("Illegal conversion");
+ } catch (Exception e) {
+ throw new KyuubiSQLException("Cannot convert column " + columnIndex + " to short: " + e, e);
+ }
+ }
+
+ @Override
+ public short getShort(String columnName) throws SQLException {
+ return getShort(findColumn(columnName));
+ }
+
+ @Override
+ public Statement getStatement() throws SQLException {
+ return this.statement;
+ }
+
+ /**
+ * @param columnIndex - the first column is 1, the second is 2, ...
+ * @see java.sql.ResultSet#getString(int)
+ */
+ @Override
+ public String getString(int columnIndex) throws SQLException {
+ Object value = getColumnValue(columnIndex);
+ if (wasNull) {
+ return null;
+ }
+ if (value instanceof byte[]) {
+ return new String((byte[]) value);
+ }
+ return value.toString();
+ }
+
+ @Override
+ public String getString(String columnName) throws SQLException {
+ return getString(findColumn(columnName));
+ }
+
+ @Override
+ public Timestamp getTimestamp(int columnIndex) throws SQLException {
+ Object obj = getObject(columnIndex);
+ if (obj == null) {
+ return null;
+ }
+ if (obj instanceof Timestamp) {
+ return (Timestamp) obj;
+ }
+ if (obj instanceof String) {
+ return Timestamp.valueOf((String) obj);
+ }
+ throw new KyuubiSQLException("Illegal conversion");
+ }
+
+ @Override
+ public Timestamp getTimestamp(String columnName) throws SQLException {
+ return getTimestamp(findColumn(columnName));
+ }
+
+ @Override
+ public int getType() throws SQLException {
+ return ResultSet.TYPE_FORWARD_ONLY;
+ }
+
+ @Override
+ public boolean rowDeleted() throws SQLException {
+ return false;
+ }
+
+ @Override
+ public boolean rowInserted() throws SQLException {
+ return false;
+ }
+
+ @Override
+ public boolean rowUpdated() throws SQLException {
+ return false;
+ }
+
+ @Override
+ public SQLWarning getWarnings() throws SQLException {
+ return warningChain;
+ }
+
+ @Override
+ public void clearWarnings() throws SQLException {
+ warningChain = null;
+ }
+
+ @Override
+ public boolean wasNull() throws SQLException {
+ return wasNull;
+ }
+
+ protected void setSchema(TTableSchema schema) {
+ this.schema = schema;
+ }
+
+ protected TTableSchema getSchema() {
+ return schema;
+ }
+
+ @Override
+ public void close() throws SQLException {
+ if (root != null) {
+ root.close();
+ }
+ if (allocator != null) {
+ allocator.close();
+ }
+ }
+}
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiQueryResultSet.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiArrowQueryResultSet.java
similarity index 80%
copy from kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiQueryResultSet.java
copy to kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiArrowQueryResultSet.java
index 13ba6395d..8bdff0f5d 100644
--- a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiQueryResultSet.java
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiArrowQueryResultSet.java
@@ -17,37 +17,50 @@
package org.apache.kyuubi.jdbc.hive;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.channels.Channels;
import java.sql.*;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.VectorLoader;
+import org.apache.arrow.vector.ipc.ReadChannel;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.ipc.message.MessageSerializer;
import org.apache.hive.service.rpc.thrift.*;
-import org.apache.kyuubi.jdbc.hive.cli.RowSet;
-import org.apache.kyuubi.jdbc.hive.cli.RowSetFactory;
+import org.apache.kyuubi.jdbc.hive.arrow.ArrowColumnVector;
+import org.apache.kyuubi.jdbc.hive.arrow.ArrowColumnarBatch;
+import org.apache.kyuubi.jdbc.hive.arrow.ArrowColumnarBatchRow;
+import org.apache.kyuubi.jdbc.hive.arrow.ArrowUtils;
import org.apache.kyuubi.jdbc.hive.common.HiveDecimal;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-/** KyuubiQueryResultSet. */
-public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
+public class KyuubiArrowQueryResultSet extends KyuubiArrowBasedResultSet {
- public static final Logger LOG = LoggerFactory.getLogger(KyuubiQueryResultSet.class);
+ public static final Logger LOG = LoggerFactory.getLogger(KyuubiArrowQueryResultSet.class);
private TCLIService.Iface client;
private TOperationHandle stmtHandle;
private TSessionHandle sessHandle;
+
private int maxRows;
private int fetchSize;
private int rowsFetched = 0;
- private RowSet fetchedRows;
- private Iterator<Object[]> fetchedRowsItr;
+ private Iterator<ArrowColumnarBatchRow> fetchedRowsItr;
private boolean isClosed = false;
private boolean emptyResultSet = false;
private boolean isScrollable = false;
private boolean fetchFirst = false;
+ // TODO:(fchen) make this configurable
+ protected boolean convertComplexTypeToString = true;
+
private final TProtocolVersion protocol;
public static class Builder {
@@ -145,8 +158,8 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
return this;
}
- public KyuubiQueryResultSet build() throws SQLException {
- return new KyuubiQueryResultSet(this);
+ public KyuubiArrowQueryResultSet build() throws SQLException {
+ return new KyuubiArrowQueryResultSet(this);
}
public TProtocolVersion getProtocolVersion() throws SQLException {
@@ -154,7 +167,7 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
}
}
- protected KyuubiQueryResultSet(Builder builder) throws SQLException {
+ protected KyuubiArrowQueryResultSet(Builder builder) throws SQLException {
this.statement = builder.statement;
this.client = builder.client;
this.stmtHandle = builder.stmtHandle;
@@ -177,6 +190,9 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
}
this.isScrollable = builder.isScrollable;
this.protocol = builder.getProtocolVersion();
+ if (allocator == null) {
+ initArrowSchemaAndAllocator();
+ }
}
/**
@@ -207,6 +223,10 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
prec == null ? HiveDecimal.USER_DEFAULT_PRECISION : prec.getI32Value(),
scale == null ? HiveDecimal.USER_DEFAULT_SCALE : scale.getI32Value());
break;
+ case TIMESTAMP_TYPE:
+ TTypeQualifierValue timeZone = tq.getQualifiers().get("session.timeZone");
+ ret = new JdbcColumnAttributes(timeZone == null ? "" : timeZone.getStringValue());
+ break;
default:
break;
}
@@ -247,6 +267,9 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
columnTypes.add(primitiveTypeEntry.getType());
columnAttributes.add(getColumnAttributes(primitiveTypeEntry));
}
+ arrowSchema =
+ ArrowUtils.toArrowSchema(
+ columnNames, convertComplexTypeToStringType(columnTypes), columnAttributes);
} catch (SQLException eS) {
throw eS; // rethrow the SQLException as is
} catch (Exception ex) {
@@ -269,6 +292,7 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
@Override
public void close() throws SQLException {
+ super.close();
if (this.statement != null && (this.statement instanceof KyuubiStatement)) {
KyuubiStatement s = (KyuubiStatement) this.statement;
s.closeClientOperation();
@@ -327,19 +351,34 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
if (fetchFirst) {
// If we are asked to start from begining, clear the current fetched resultset
orientation = TFetchOrientation.FETCH_FIRST;
- fetchedRows = null;
fetchedRowsItr = null;
fetchFirst = false;
}
- if (fetchedRows == null || !fetchedRowsItr.hasNext()) {
+
+ if (fetchedRowsItr == null || !fetchedRowsItr.hasNext()) {
+
TFetchResultsReq fetchReq = new TFetchResultsReq(stmtHandle, orientation, fetchSize);
TFetchResultsResp fetchResp;
fetchResp = client.FetchResults(fetchReq);
Utils.verifySuccessWithInfo(fetchResp.getStatus());
TRowSet results = fetchResp.getResults();
- fetchedRows = RowSetFactory.create(results, protocol);
- fetchedRowsItr = fetchedRows.iterator();
+ if (results == null || results.getColumnsSize() == 0) {
+ return false;
+ }
+ TColumn arrowColumn = results.getColumns().get(0);
+ byte[] batchBytes = arrowColumn.getBinaryVal().getValues().get(0).array();
+ ArrowRecordBatch recordBatch = loadArrowBatch(batchBytes, allocator);
+ VectorLoader vectorLoader = new VectorLoader(root);
+ vectorLoader.load(recordBatch);
+ recordBatch.close();
+ java.util.List<ArrowColumnVector> columns =
+ root.getFieldVectors().stream()
+ .map(vector -> new ArrowColumnVector(vector))
+ .collect(Collectors.toList());
+ ArrowColumnarBatch batch =
+ new ArrowColumnarBatch(columns.toArray(new ArrowColumnVector[0]), root.getRowCount());
+ fetchedRowsItr = batch.rowIterator();
}
if (fetchedRowsItr.hasNext()) {
@@ -359,6 +398,13 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
return true;
}
+ private ArrowRecordBatch loadArrowBatch(byte[] batchBytes, BufferAllocator allocator)
+ throws IOException {
+ ByteArrayInputStream in = new ByteArrayInputStream(batchBytes);
+ return MessageSerializer.deserializeRecordBatch(
+ new ReadChannel(Channels.newChannel(in)), allocator);
+ }
+
@Override
public ResultSetMetaData getMetaData() throws SQLException {
if (isClosed) {
@@ -430,4 +476,23 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
public boolean isClosed() {
return isClosed;
}
+
+ private List<TTypeId> convertComplexTypeToStringType(List<TTypeId> colTypes) {
+ if (convertComplexTypeToString) {
+ return colTypes.stream()
+ .map(
+ type -> {
+ if (type == TTypeId.ARRAY_TYPE
+ || type == TTypeId.MAP_TYPE
+ || type == TTypeId.STRUCT_TYPE) {
+ return TTypeId.STRING_TYPE;
+ } else {
+ return type;
+ }
+ })
+ .collect(Collectors.toList());
+ } else {
+ return colTypes;
+ }
+ }
}
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiConnection.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiConnection.java
index c5e80efe3..00d561cea 100644
--- a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiConnection.java
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiConnection.java
@@ -111,6 +111,7 @@ public class KyuubiConnection implements SQLConnection, KyuubiLoggable {
private String engineUrl = "";
private boolean isBeeLineMode;
+ private String resultCodec = "simple";
/** Get all direct HiveServer2 URLs from a ZooKeeper based HiveServer2 URL */
public static List<JdbcConnectionParams> getAllUrls(String zookeeperBasedHS2Url)
@@ -146,6 +147,19 @@ public class KyuubiConnection implements SQLConnection, KyuubiLoggable {
}
port = connParams.getPort();
+ resultCodec =
+ connParams
+ .getSessionVars()
+ .getOrDefault(
+ "kyuubi.operation.result.codec",
+ connParams
+ .getHiveVars()
+ .getOrDefault(
+ "kyuubi.operation.result.codec",
+ connParams
+ .getHiveConfs()
+ .getOrDefault("kyuubi.operation.result.codec", "simple")));
+
setupTimeout();
if (sessConfMap.containsKey(FETCH_SIZE)) {
@@ -1366,4 +1380,8 @@ public class KyuubiConnection implements SQLConnection, KyuubiLoggable {
public String getEngineUrl() {
return engineUrl;
}
+
+ String getResultCodec() {
+ return resultCodec;
+ }
}
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiQueryResultSet.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiQueryResultSet.java
index 13ba6395d..f06ada5d4 100644
--- a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiQueryResultSet.java
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiQueryResultSet.java
@@ -250,7 +250,6 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
} catch (SQLException eS) {
throw eS; // rethrow the SQLException as is
} catch (Exception ex) {
- ex.printStackTrace();
throw new KyuubiSQLException("Could not create ResultSet: " + ex.getMessage(), ex);
}
}
@@ -352,7 +351,6 @@ public class KyuubiQueryResultSet extends KyuubiBaseResultSet {
} catch (SQLException eS) {
throw eS;
} catch (Exception ex) {
- ex.printStackTrace();
throw new KyuubiSQLException("Error retrieving next row", ex);
}
// NOTE: fetchOne doesn't throw new SQLFeatureNotSupportedException("Method not supported").
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiStatement.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiStatement.java
index 7451c8589..3abc36666 100644
--- a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiStatement.java
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiStatement.java
@@ -199,15 +199,32 @@ public class KyuubiStatement implements SQLStatement, KyuubiLoggable {
if (!status.isHasResultSet() && !stmtHandle.isHasResultSet()) {
return false;
}
- resultSet =
- new KyuubiQueryResultSet.Builder(this)
- .setClient(client)
- .setSessionHandle(sessHandle)
- .setStmtHandle(stmtHandle)
- .setMaxRows(maxRows)
- .setFetchSize(fetchSize)
- .setScrollable(isScrollableResultset)
- .build();
+
+ String resultCodec = connection.getResultCodec().toLowerCase(Locale.ROOT);
+ LOG.info("kyuubi.operation.result.codec: " + resultCodec);
+ switch (resultCodec) {
+ case "arrow":
+ resultSet =
+ new KyuubiArrowQueryResultSet.Builder(this)
+ .setClient(client)
+ .setSessionHandle(sessHandle)
+ .setStmtHandle(stmtHandle)
+ .setMaxRows(maxRows)
+ .setFetchSize(fetchSize)
+ .setScrollable(isScrollableResultset)
+ .build();
+ break;
+ default:
+ resultSet =
+ new KyuubiQueryResultSet.Builder(this)
+ .setClient(client)
+ .setSessionHandle(sessHandle)
+ .setStmtHandle(stmtHandle)
+ .setMaxRows(maxRows)
+ .setFetchSize(fetchSize)
+ .setScrollable(isScrollableResultset)
+ .build();
+ }
return true;
}
@@ -231,15 +248,30 @@ public class KyuubiStatement implements SQLStatement, KyuubiLoggable {
if (!status.isHasResultSet()) {
return false;
}
- resultSet =
- new KyuubiQueryResultSet.Builder(this)
- .setClient(client)
- .setSessionHandle(sessHandle)
- .setStmtHandle(stmtHandle)
- .setMaxRows(maxRows)
- .setFetchSize(fetchSize)
- .setScrollable(isScrollableResultset)
- .build();
+ String resultCodec = connection.getResultCodec().toLowerCase(Locale.ROOT);
+ LOG.info("kyuubi.operation.result.codec: " + resultCodec);
+ switch (resultCodec) {
+ case "arrow":
+ resultSet =
+ new KyuubiArrowQueryResultSet.Builder(this)
+ .setClient(client)
+ .setSessionHandle(sessHandle)
+ .setStmtHandle(stmtHandle)
+ .setMaxRows(maxRows)
+ .setFetchSize(fetchSize)
+ .setScrollable(isScrollableResultset)
+ .build();
+ default:
+ resultSet =
+ new KyuubiQueryResultSet.Builder(this)
+ .setClient(client)
+ .setSessionHandle(sessHandle)
+ .setStmtHandle(stmtHandle)
+ .setMaxRows(maxRows)
+ .setFetchSize(fetchSize)
+ .setScrollable(isScrollableResultset)
+ .build();
+ }
return true;
}
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowColumnVector.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowColumnVector.java
new file mode 100644
index 000000000..79c84adb9
--- /dev/null
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowColumnVector.java
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.jdbc.hive.arrow;
+
+import java.math.BigDecimal;
+import org.apache.arrow.vector.*;
+import org.apache.arrow.vector.holders.NullableVarCharHolder;
+
+/** A column vector backed by Apache Arrow. */
+public class ArrowColumnVector {
+
+ ArrowVectorAccessor accessor;
+ ArrowColumnVector[] childColumns;
+
+ public ValueVector getValueVector() {
+ return accessor.vector;
+ }
+
+ public boolean hasNull() {
+ return accessor.getNullCount() > 0;
+ }
+
+ public int numNulls() {
+ return accessor.getNullCount();
+ }
+
+ public void close() {
+ if (childColumns != null) {
+ for (int i = 0; i < childColumns.length; i++) {
+ childColumns[i].close();
+ childColumns[i] = null;
+ }
+ childColumns = null;
+ }
+ accessor.close();
+ }
+
+ public boolean isNullAt(int rowId) {
+ return accessor.isNullAt(rowId);
+ }
+
+ public boolean getBoolean(int rowId) {
+ return accessor.getBoolean(rowId);
+ }
+
+ public byte getByte(int rowId) {
+ return accessor.getByte(rowId);
+ }
+
+ public short getShort(int rowId) {
+ return accessor.getShort(rowId);
+ }
+
+ public int getInt(int rowId) {
+ return accessor.getInt(rowId);
+ }
+
+ public long getLong(int rowId) {
+ return accessor.getLong(rowId);
+ }
+
+ public float getFloat(int rowId) {
+ return accessor.getFloat(rowId);
+ }
+
+ public double getDouble(int rowId) {
+ return accessor.getDouble(rowId);
+ }
+
+ public BigDecimal getDecimal(int rowId, int precision, int scale) {
+ if (isNullAt(rowId)) return null;
+ return accessor.getDecimal(rowId, precision, scale);
+ }
+
+ public BigDecimal getDecimal(int rowId) {
+ if (isNullAt(rowId)) return null;
+ return accessor.getDecimal(rowId);
+ }
+
+ public String getString(int rowId) {
+ if (isNullAt(rowId)) return null;
+ return accessor.getString(rowId);
+ }
+
+ public byte[] getBinary(int rowId) {
+ if (isNullAt(rowId)) return null;
+ return accessor.getBinary(rowId);
+ }
+
+ public ArrowColumnVector(ValueVector vector) {
+ initAccessor(vector);
+ }
+
+ void initAccessor(ValueVector vector) {
+ if (vector instanceof BitVector) {
+ accessor = new BooleanAccessor((BitVector) vector);
+ } else if (vector instanceof TinyIntVector) {
+ accessor = new ByteAccessor((TinyIntVector) vector);
+ } else if (vector instanceof SmallIntVector) {
+ accessor = new ShortAccessor((SmallIntVector) vector);
+ } else if (vector instanceof IntVector) {
+ accessor = new IntAccessor((IntVector) vector);
+ } else if (vector instanceof BigIntVector) {
+ accessor = new LongAccessor((BigIntVector) vector);
+ } else if (vector instanceof Float4Vector) {
+ accessor = new FloatAccessor((Float4Vector) vector);
+ } else if (vector instanceof Float8Vector) {
+ accessor = new DoubleAccessor((Float8Vector) vector);
+ } else if (vector instanceof DecimalVector) {
+ accessor = new DecimalAccessor((DecimalVector) vector);
+ } else if (vector instanceof VarCharVector) {
+ accessor = new StringAccessor((VarCharVector) vector);
+ } else if (vector instanceof VarBinaryVector) {
+ accessor = new BinaryAccessor((VarBinaryVector) vector);
+ } else if (vector instanceof DateDayVector) {
+ accessor = new DateAccessor((DateDayVector) vector);
+ } else if (vector instanceof TimeStampMicroTZVector) {
+ accessor = new TimestampAccessor((TimeStampMicroTZVector) vector);
+ } else if (vector instanceof TimeStampMicroVector) {
+ accessor = new TimestampNTZAccessor((TimeStampMicroVector) vector);
+ } else if (vector instanceof NullVector) {
+ accessor = new NullAccessor((NullVector) vector);
+ } else if (vector instanceof IntervalYearVector) {
+ accessor = new IntervalYearAccessor((IntervalYearVector) vector);
+ } else if (vector instanceof DurationVector) {
+ accessor = new DurationAccessor((DurationVector) vector);
+ } else {
+ throw new UnsupportedOperationException();
+ }
+ }
+
+ abstract static class ArrowVectorAccessor {
+
+ final ValueVector vector;
+
+ ArrowVectorAccessor(ValueVector vector) {
+ this.vector = vector;
+ }
+
+ final boolean isNullAt(int rowId) {
+ return vector.isNull(rowId);
+ }
+
+ final int getNullCount() {
+ return vector.getNullCount();
+ }
+
+ final void close() {
+ vector.close();
+ }
+
+ boolean getBoolean(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ byte getByte(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ short getShort(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ int getInt(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ long getLong(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ float getFloat(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ double getDouble(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ BigDecimal getDecimal(int rowId, int precision, int scale) {
+ throw new UnsupportedOperationException();
+ }
+
+ BigDecimal getDecimal(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ String getString(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ byte[] getBinary(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ Object getArray(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+
+ Object getMap(int rowId) {
+ throw new UnsupportedOperationException();
+ }
+ }
+
+ static class BooleanAccessor extends ArrowVectorAccessor {
+
+ private final BitVector accessor;
+
+ BooleanAccessor(BitVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final boolean getBoolean(int rowId) {
+ return accessor.get(rowId) == 1;
+ }
+ }
+
+ static class ByteAccessor extends ArrowVectorAccessor {
+
+ private final TinyIntVector accessor;
+
+ ByteAccessor(TinyIntVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final byte getByte(int rowId) {
+ return accessor.get(rowId);
+ }
+ }
+
+ static class ShortAccessor extends ArrowVectorAccessor {
+
+ private final SmallIntVector accessor;
+
+ ShortAccessor(SmallIntVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final short getShort(int rowId) {
+ return accessor.get(rowId);
+ }
+ }
+
+ static class IntAccessor extends ArrowVectorAccessor {
+
+ private final IntVector accessor;
+
+ IntAccessor(IntVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final int getInt(int rowId) {
+ return accessor.get(rowId);
+ }
+ }
+
+ static class LongAccessor extends ArrowVectorAccessor {
+
+ private final BigIntVector accessor;
+
+ LongAccessor(BigIntVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final long getLong(int rowId) {
+ return accessor.get(rowId);
+ }
+ }
+
+ static class FloatAccessor extends ArrowVectorAccessor {
+
+ private final Float4Vector accessor;
+
+ FloatAccessor(Float4Vector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final float getFloat(int rowId) {
+ return accessor.get(rowId);
+ }
+ }
+
+ static class DoubleAccessor extends ArrowVectorAccessor {
+
+ private final Float8Vector accessor;
+
+ DoubleAccessor(Float8Vector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final double getDouble(int rowId) {
+ return accessor.get(rowId);
+ }
+ }
+
+ static class DecimalAccessor extends ArrowVectorAccessor {
+
+ private final DecimalVector accessor;
+
+ DecimalAccessor(DecimalVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final BigDecimal getDecimal(int rowId, int precision, int scale) {
+ if (isNullAt(rowId)) return null;
+ return accessor.getObject(rowId);
+ }
+
+ @Override
+ final BigDecimal getDecimal(int rowId) {
+ if (isNullAt(rowId)) return null;
+ return accessor.getObject(rowId);
+ }
+ }
+
+ static class StringAccessor extends ArrowVectorAccessor {
+
+ private final VarCharVector accessor;
+ private final NullableVarCharHolder stringResult = new NullableVarCharHolder();
+
+ StringAccessor(VarCharVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final String getString(int rowId) {
+ accessor.get(rowId, stringResult);
+ if (stringResult.isSet == 0) {
+ return null;
+ } else {
+ return new String(accessor.get(rowId));
+ }
+ }
+ }
+
+ static class BinaryAccessor extends ArrowVectorAccessor {
+
+ private final VarBinaryVector accessor;
+
+ BinaryAccessor(VarBinaryVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final byte[] getBinary(int rowId) {
+ return accessor.getObject(rowId);
+ }
+ }
+
+ static class DateAccessor extends ArrowVectorAccessor {
+
+ private final DateDayVector accessor;
+
+ DateAccessor(DateDayVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final int getInt(int rowId) {
+ return accessor.get(rowId);
+ }
+ }
+
+ static class TimestampAccessor extends ArrowVectorAccessor {
+
+ private final TimeStampMicroTZVector accessor;
+
+ TimestampAccessor(TimeStampMicroTZVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final long getLong(int rowId) {
+ return accessor.get(rowId);
+ }
+ }
+
+ static class TimestampNTZAccessor extends ArrowVectorAccessor {
+
+ private final TimeStampMicroVector accessor;
+
+ TimestampNTZAccessor(TimeStampMicroVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final long getLong(int rowId) {
+ return accessor.get(rowId);
+ }
+ }
+
+ static class NullAccessor extends ArrowVectorAccessor {
+
+ NullAccessor(NullVector vector) {
+ super(vector);
+ }
+ }
+
+ static class IntervalYearAccessor extends ArrowVectorAccessor {
+
+ private final IntervalYearVector accessor;
+
+ IntervalYearAccessor(IntervalYearVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ int getInt(int rowId) {
+ return accessor.get(rowId);
+ }
+ }
+
+ static class DurationAccessor extends ArrowVectorAccessor {
+
+ private final DurationVector accessor;
+
+ DurationAccessor(DurationVector vector) {
+ super(vector);
+ this.accessor = vector;
+ }
+
+ @Override
+ final long getLong(int rowId) {
+ return DurationVector.get(accessor.getDataBuffer(), rowId);
+ }
+ }
+}
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowColumnarBatch.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowColumnarBatch.java
new file mode 100644
index 000000000..2ab0c94dd
--- /dev/null
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowColumnarBatch.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.jdbc.hive.arrow;
+
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+
+/**
+ * This class wraps multiple ColumnVectors as a row-wise table. It provides a row view of this batch
+ * so that Spark can access the data row by row. Instance of it is meant to be reused during the
+ * entire data loading process. A data source may extend this class with customized logic.
+ */
+public class ArrowColumnarBatch implements AutoCloseable {
+ protected int numRows;
+ protected final ArrowColumnVector[] columns;
+
+ // Staging row returned from `getRow`.
+ protected final ArrowColumnarBatchRow row;
+
+ /**
+ * Called to close all the columns in this batch. It is not valid to access the data after calling
+ * this. This must be called at the end to clean up memory allocations.
+ */
+ @Override
+ public void close() {
+ for (ArrowColumnVector c : columns) {
+ c.close();
+ }
+ }
+
+ /** Returns an iterator over the rows in this batch. */
+ public Iterator<ArrowColumnarBatchRow> rowIterator() {
+ final int maxRows = numRows;
+ final ArrowColumnarBatchRow row = new ArrowColumnarBatchRow(columns);
+ return new Iterator<ArrowColumnarBatchRow>() {
+ int rowId = 0;
+
+ @Override
+ public boolean hasNext() {
+ return rowId < maxRows;
+ }
+
+ @Override
+ public ArrowColumnarBatchRow next() {
+ if (rowId >= maxRows) {
+ throw new NoSuchElementException();
+ }
+ row.rowId = rowId++;
+ return row;
+ }
+
+ @Override
+ public void remove() {
+ throw new UnsupportedOperationException();
+ }
+ };
+ }
+
+ /** Sets the number of rows in this batch. */
+ public void setNumRows(int numRows) {
+ this.numRows = numRows;
+ }
+
+ /** Returns the number of columns that make up this batch. */
+ public int numCols() {
+ return columns.length;
+ }
+
+ /** Returns the number of rows for read, including filtered rows. */
+ public int numRows() {
+ return numRows;
+ }
+
+ /** Returns the column at `ordinal`. */
+ public ArrowColumnVector column(int ordinal) {
+ return columns[ordinal];
+ }
+
+ /** Returns the row in this batch at `rowId`. Returned row is reused across calls. */
+ public ArrowColumnarBatchRow getRow(int rowId) {
+ assert (rowId >= 0 && rowId < numRows);
+ row.rowId = rowId;
+ return row;
+ }
+
+ public ArrowColumnarBatch(ArrowColumnVector[] columns) {
+ this(columns, 0);
+ }
+
+ /**
+ * Create a new batch from existing column vectors.
+ *
+ * @param columns The columns of this batch
+ * @param numRows The number of rows in this batch
+ */
+ public ArrowColumnarBatch(ArrowColumnVector[] columns, int numRows) {
+ this.columns = columns;
+ this.numRows = numRows;
+ this.row = new ArrowColumnarBatchRow(columns);
+ }
+}
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowColumnarBatchRow.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowColumnarBatchRow.java
new file mode 100644
index 000000000..20ed55a1d
--- /dev/null
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowColumnarBatchRow.java
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.jdbc.hive.arrow;
+
+import java.math.BigDecimal;
+import java.sql.Timestamp;
+import org.apache.hive.service.rpc.thrift.TTypeId;
+import org.apache.kyuubi.jdbc.hive.common.DateUtils;
+import org.apache.kyuubi.jdbc.hive.common.HiveIntervalDayTime;
+import org.apache.kyuubi.jdbc.hive.common.HiveIntervalYearMonth;
+
+public class ArrowColumnarBatchRow {
+ public int rowId;
+ private final ArrowColumnVector[] columns;
+
+ public ArrowColumnarBatchRow(ArrowColumnVector[] columns) {
+ this.columns = columns;
+ }
+
+ public int numFields() {
+ return columns.length;
+ }
+
+ public boolean anyNull() {
+ throw new UnsupportedOperationException();
+ }
+
+ public boolean isNullAt(int ordinal) {
+ return columns[ordinal].isNullAt(rowId);
+ }
+
+ public boolean getBoolean(int ordinal) {
+ return columns[ordinal].getBoolean(rowId);
+ }
+
+ public byte getByte(int ordinal) {
+ return columns[ordinal].getByte(rowId);
+ }
+
+ public short getShort(int ordinal) {
+ return columns[ordinal].getShort(rowId);
+ }
+
+ public int getInt(int ordinal) {
+ return columns[ordinal].getInt(rowId);
+ }
+
+ public long getLong(int ordinal) {
+ return columns[ordinal].getLong(rowId);
+ }
+
+ public float getFloat(int ordinal) {
+ return columns[ordinal].getFloat(rowId);
+ }
+
+ public double getDouble(int ordinal) {
+ return columns[ordinal].getDouble(rowId);
+ }
+
+ public BigDecimal getDecimal(int ordinal, int precision, int scale) {
+ return columns[ordinal].getDecimal(rowId, precision, scale);
+ }
+
+ public BigDecimal getDecimal(int ordinal) {
+ return columns[ordinal].getDecimal(rowId);
+ }
+
+ public String getString(int ordinal) {
+ return columns[ordinal].getString(rowId);
+ }
+
+ public byte[] getBinary(int ordinal) {
+ return columns[ordinal].getBinary(rowId);
+ }
+
+ public Object getInterval(int ordinal) {
+ throw new UnsupportedOperationException();
+ }
+
+ public Object getStruct(int ordinal, int numFields) {
+ throw new UnsupportedOperationException();
+ }
+
+ public Object getArray(int ordinal) {
+ throw new UnsupportedOperationException();
+ }
+
+ public Object getMap(int ordinal) {
+ throw new UnsupportedOperationException();
+ }
+
+ public Object get(int ordinal, TTypeId dataType) {
+ switch (dataType) {
+ case BOOLEAN_TYPE:
+ return getBoolean(ordinal);
+ case TINYINT_TYPE:
+ return getByte(ordinal);
+ case SMALLINT_TYPE:
+ return getShort(ordinal);
+ case INT_TYPE:
+ return getInt(ordinal);
+ case BIGINT_TYPE:
+ return getLong(ordinal);
+ case BINARY_TYPE:
+ return getBinary(ordinal);
+ case FLOAT_TYPE:
+ return getFloat(ordinal);
+ case DOUBLE_TYPE:
+ return getDouble(ordinal);
+ case DECIMAL_TYPE:
+ return getDecimal(ordinal);
+ case STRING_TYPE:
+ return getString(ordinal);
+ case TIMESTAMP_TYPE:
+ return new Timestamp(getLong(ordinal) / 1000);
+ case DATE_TYPE:
+ return DateUtils.internalToDate(getInt(ordinal));
+ case INTERVAL_DAY_TIME_TYPE:
+ long microseconds = getLong(ordinal);
+ long seconds = microseconds / 1000000;
+ int nanos = (int) (microseconds % 1000000) * 1000;
+ return new HiveIntervalDayTime(seconds, nanos);
+ case INTERVAL_YEAR_MONTH_TYPE:
+ return new HiveIntervalYearMonth(getInt(ordinal));
+ case ARRAY_TYPE:
+ return getString(ordinal);
+ case MAP_TYPE:
+ return getString(ordinal);
+ case STRUCT_TYPE:
+ return getString(ordinal);
+ default:
+ throw new UnsupportedOperationException("Datatype not supported " + dataType);
+ }
+ }
+}
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowUtils.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowUtils.java
new file mode 100644
index 000000000..1d1587b54
--- /dev/null
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowUtils.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.jdbc.hive.arrow;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.types.DateUnit;
+import org.apache.arrow.vector.types.FloatingPointPrecision;
+import org.apache.arrow.vector.types.IntervalUnit;
+import org.apache.arrow.vector.types.TimeUnit;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hive.service.rpc.thrift.TTypeId;
+import org.apache.kyuubi.jdbc.hive.JdbcColumnAttributes;
+
+public class ArrowUtils {
+
+ public static final RootAllocator rootAllocator = new RootAllocator(Long.MAX_VALUE);
+
+ public static Schema toArrowSchema(
+ List<String> columnNames, List<TTypeId> ttypes, List<JdbcColumnAttributes> columnAttributes) {
+ List<Field> fields =
+ IntStream.range(0, columnNames.size())
+ .mapToObj(i -> toArrowField(columnNames.get(i), ttypes.get(i), columnAttributes.get(i)))
+ .collect(Collectors.toList());
+ return new Schema(fields);
+ }
+
+ public static Field toArrowField(
+ String name, TTypeId ttype, JdbcColumnAttributes jdbcColumnAttributes) {
+ boolean nullable = true;
+ return new Field(
+ name,
+ new FieldType(nullable, toArrowType(ttype, jdbcColumnAttributes), null),
+ Collections.emptyList());
+ }
+
+ public static ArrowType toArrowType(TTypeId ttype, JdbcColumnAttributes jdbcColumnAttributes) {
+ switch (ttype) {
+ case NULL_TYPE:
+ return ArrowType.Null.INSTANCE;
+ case BOOLEAN_TYPE:
+ return ArrowType.Bool.INSTANCE;
+ case TINYINT_TYPE:
+ return new ArrowType.Int(8, true);
+ case SMALLINT_TYPE:
+ return new ArrowType.Int(8 * 2, true);
+ case INT_TYPE:
+ return new ArrowType.Int(8 * 4, true);
+ case BIGINT_TYPE:
+ return new ArrowType.Int(8 * 8, true);
+ case FLOAT_TYPE:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
+ case DOUBLE_TYPE:
+ return new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
+ case STRING_TYPE:
+ return ArrowType.Utf8.INSTANCE;
+ case DECIMAL_TYPE:
+ if (jdbcColumnAttributes != null) {
+ return ArrowType.Decimal.createDecimal(
+ jdbcColumnAttributes.precision, jdbcColumnAttributes.scale, null);
+ }
+ case DATE_TYPE:
+ return new ArrowType.Date(DateUnit.DAY);
+ case TIMESTAMP_TYPE:
+ if (jdbcColumnAttributes != null) {
+ return new ArrowType.Timestamp(TimeUnit.MICROSECOND, jdbcColumnAttributes.timeZone);
+ } else {
+ throw new IllegalStateException("Missing timezoneId where it is mandatory.");
+ }
+ case BINARY_TYPE:
+ return ArrowType.Binary.INSTANCE;
+ case INTERVAL_DAY_TIME_TYPE:
+ return new ArrowType.Duration(TimeUnit.MICROSECOND);
+ case INTERVAL_YEAR_MONTH_TYPE:
+ return new ArrowType.Interval(IntervalUnit.YEAR_MONTH);
+ default:
+ throw new IllegalArgumentException("Unrecognized type name: " + ttype.name());
+ }
+ }
+}
diff --git a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/common/DateUtils.java b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/common/DateUtils.java
index f2897bfb1..fe181346e 100644
--- a/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/common/DateUtils.java
+++ b/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/common/DateUtils.java
@@ -17,6 +17,8 @@
package org.apache.kyuubi.jdbc.hive.common;
+import java.util.TimeZone;
+
/** DateUtils. Thread-safe class */
public class DateUtils {
public static int parseNumericValueWithRange(
@@ -65,4 +67,21 @@ public class DateUtils {
public static String getFieldName(int field) {
return FIELD_NAME[field];
}
+
+ /**
+ * Converts the internal representation of a SQL DATE (int) to the Java type ({@link
+ * java.sql.Date}).
+ */
+ public static java.sql.Date internalToDate(int v) {
+ // note that, in this case, can't handle Daylight Saving Time
+ final long t = v * MILLIS_PER_DAY;
+ return new java.sql.Date(t - TimeZone.getDefault().getOffset(t));
+ }
+
+ /**
+ * The number of milliseconds in a day.
+ *
+ * <p>This is the modulo 'mask' used when converting TIMESTAMP values to DATE and TIME values.
+ */
+ private static final long MILLIS_PER_DAY = 86400000L; // = 24 * 60 * 60 * 1000
}
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/BackendServiceMetric.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/BackendServiceMetric.scala
index 684f2114c..86fc9679b 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/BackendServiceMetric.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/BackendServiceMetric.scala
@@ -183,6 +183,7 @@ trait BackendServiceMetric extends BackendService {
fetchLog: Boolean): TRowSet = {
MetricsSystem.timerTracing(MetricsConstants.BS_FETCH_RESULTS) {
val rowSet = super.fetchResults(operationHandle, orientation, maxRows, fetchLog)
+ // TODO: the statistics are wrong when we enabled the arrow.
val rowsSize =
if (rowSet.getColumnsSize > 0) {
rowSet.getColumns.get(0).getFieldValue match {
diff --git a/pom.xml b/pom.xml
index c5fef8ed7..d61afafa9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -115,6 +115,7 @@
<scala.binary.version>2.12</scala.binary.version>
<scala-collection-compat.version>2.8.1</scala-collection-compat.version>
+ <arrow.version>10.0.1</arrow.version>
<antlr4.version>4.8</antlr4.version>
<apache.archive.dist>https://archive.apache.org/dist</apache.archive.dist>
<bouncycastle.version>1.67</bouncycastle.version>
@@ -253,6 +254,16 @@
<dependencyManagement>
<dependencies>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-vector</artifactId>
+ <version>${arrow.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-memory-netty</artifactId>
+ <version>${arrow.version}</version>
+ </dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>