[GitHub] [spark] maropu commented on a change in pull request #30957: [SPARK-31937][SQL] Support processing ArrayType/MapType/StructType data using no-serde mode script transform

[GitBox <gi...@apache.org>]

maropu commented on a change in pull request #30957:
URL: https://github.com/apache/spark/pull/30957#discussion_r551738592



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala
##########
@@ -329,14 +332,45 @@ case class ScriptTransformationIOSchema(
     schemaLess: Boolean) extends Serializable {
   import ScriptTransformationIOSchema._
 
-  val inputRowFormatMap = inputRowFormat.toMap.withDefault((k) => defaultFormat(k))
-  val outputRowFormatMap = outputRowFormat.toMap.withDefault((k) => defaultFormat(k))
+  val inputRowFormatMap = inputRowFormat.toMap.withDefault(k => defaultFormat(k))
+  val outputRowFormatMap = outputRowFormat.toMap.withDefault(k => defaultFormat(k))
+
+  val separators = (getByte(inputRowFormatMap("TOK_TABLEROWFORMATFIELD"), 0.toByte) ::
+    getByte(inputRowFormatMap("TOK_TABLEROWFORMATCOLLITEMS"), 1.toByte) ::
+    getByte(inputRowFormatMap("TOK_TABLEROWFORMATMAPKEYS"), 2.toByte) :: Nil) ++
+    (4 to 8).map(_.toByte)

Review comment:
       I'm still not sure about why the no-serde case needs to follow the `LazySimpleSerde` behaviour.  As for the migration case you suggested above, is it not enough just to use `LazySimpleSerde` in Spark? Most users who care about the hive compatibility just use Spark with `-Phive` enabled, I think.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org