You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Maksim Lixakov (Jira)" <ji...@apache.org> on 2024/08/08 10:53:00 UTC
[jira] [Updated] (SPARK-49161) Add support for StructType and MapType in JdbcUtils
[ https://issues.apache.org/jira/browse/SPARK-49161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maksim Lixakov updated SPARK-49161:
-----------------------------------
Description:
Currently, in Apache Spark's {{JdbcUtils}} (across all versions, including Spark 4), there is no support for {{{}StructType{}}}, {{MapType.}}
The current implementation in {{JdbcUtils}} supports [{{ArrayType}}|https://github.com/apache/spark/blob/6246b18635f073e872e32dd05a699007446aee94/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L687]
To extend the functionality and enable users to read and write {{StructType}} and {{MapType}} from their databases analog, it is proposed to add similar handling for these complex types in the {{makeGetter}} and {{makeSetter}} methods.
I suppose to see it something like:
{code:java}
case StructType(fields) =>
(rs: ResultSet, row: InternalRow, pos: Int) =>
val structData = rs.getObject(pos + 1).asInstanceOf[Struct]
if (structData == null) {
row.setNullAt(pos)
} else {
val values = new GenericInternalRow(fields.length)
fields.zipWithIndex.foreach { case (field, i) =>
val fieldValue = structData.getAttributes()(i)
values.update(i, getCatalystType(field.dataType).getJavaValue(fieldValue))
}
row.update(pos, values)
}
case MapType(keyType, valueType, valueContainsNull) =>
(rs: ResultSet, row: InternalRow, pos: Int) =>
val mapData = rs.getObject(pos + 1).asInstanceOf[Map[Any, Any]]
if (mapData == null) {
row.setNullAt(pos)
} else {
val keys = mapData.keys.map(getCatalystType(keyType).getJavaValue).toArray
val values = mapData.values.map(getCatalystType(valueType).getJavaValue).toArray
row.update(pos, new ArrayBasedMapData(new GenericArrayData(keys), new GenericArrayData(values)))
{code}
was:
Currently, in Apache Spark's {{JdbcUtils}} (across all versions, including Spark 4), there is no support for {{{}StructType{}}}, {{MapType.}}
The current implementation in {{JdbcUtils}} supports [{{ArrayType}}|https://github.com/apache/spark/blob/6246b18635f073e872e32dd05a699007446aee94/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L687]
To extend the functionality and enable users to read and write {{StructType}} and {{MapType}} from their databases analog, it is proposed to add similar handling for these complex types in the {{makeGetter}} and {{makeSetter}} methods.
I suppose to see it something like:
{code:java}
case StructType(fields) =>
(rs: ResultSet, row: InternalRow, pos: Int) =>
val structData = rs.getObject(pos + 1).asInstanceOf[Struct]
if (structData == null) {
row.setNullAt(pos)
} else {
val values = new GenericInternalRow(fields.length)
fields.zipWithIndex.foreach { case (field, i) =>
val fieldValue = structData.getAttributes()(i)
values.update(i, getCatalystType(field.dataType).getJavaValue(fieldValue))
}
row.update(pos, values)
}case MapType(keyType, valueType, valueContainsNull) =>
(rs: ResultSet, row: InternalRow, pos: Int) =>
val mapData = rs.getObject(pos + 1).asInstanceOf[Map[Any, Any]]
if (mapData == null) {
row.setNullAt(pos)
} else {
val keys = mapData.keys.map(getCatalystType(keyType).getJavaValue).toArray
val values = mapData.values.map(getCatalystType(valueType).getJavaValue).toArray
row.update(pos, new ArrayBasedMapData(new GenericArrayData(keys), new GenericArrayData(values)))
{code}
> Add support for StructType and MapType in JdbcUtils
> ---------------------------------------------------
>
> Key: SPARK-49161
> URL: https://issues.apache.org/jira/browse/SPARK-49161
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
> Affects Versions: 4.0.0
> Reporter: Maksim Lixakov
> Priority: Minor
>
> Currently, in Apache Spark's {{JdbcUtils}} (across all versions, including Spark 4), there is no support for {{{}StructType{}}}, {{MapType.}}
> The current implementation in {{JdbcUtils}} supports [{{ArrayType}}|https://github.com/apache/spark/blob/6246b18635f073e872e32dd05a699007446aee94/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L687]
> To extend the functionality and enable users to read and write {{StructType}} and {{MapType}} from their databases analog, it is proposed to add similar handling for these complex types in the {{makeGetter}} and {{makeSetter}} methods.
> I suppose to see it something like:
> {code:java}
> case StructType(fields) =>
> (rs: ResultSet, row: InternalRow, pos: Int) =>
> val structData = rs.getObject(pos + 1).asInstanceOf[Struct]
> if (structData == null) {
> row.setNullAt(pos)
> } else {
> val values = new GenericInternalRow(fields.length)
> fields.zipWithIndex.foreach { case (field, i) =>
> val fieldValue = structData.getAttributes()(i)
> values.update(i, getCatalystType(field.dataType).getJavaValue(fieldValue))
> }
> row.update(pos, values)
> }
> case MapType(keyType, valueType, valueContainsNull) =>
> (rs: ResultSet, row: InternalRow, pos: Int) =>
> val mapData = rs.getObject(pos + 1).asInstanceOf[Map[Any, Any]]
> if (mapData == null) {
> row.setNullAt(pos)
> } else {
> val keys = mapData.keys.map(getCatalystType(keyType).getJavaValue).toArray
> val values = mapData.values.map(getCatalystType(valueType).getJavaValue).toArray
> row.update(pos, new ArrayBasedMapData(new GenericArrayData(keys), new GenericArrayData(values)))
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org