You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "attilapiros (via GitHub)" <gi...@apache.org> on 2026/01/23 18:00:16 UTC

Re: [PR] [SPARK-54586][SQL] Validate UTF-8 when casting Binary to String [spark]

attilapiros commented on code in PR #53458:
URL: https://github.com/apache/spark/pull/53458#discussion_r2722270293


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:
##########
@@ -639,10 +633,15 @@ case class Cast(
   }
 
   override def nullable: Boolean = if (!isTryCast) {
-    child.nullable || Cast.forceNullable(child.dataType, dataType)
+    // In LEGACY mode with validation enabled, casting BinaryType to StringType can return null
+    val binaryToStringInLegacy = !ansiEnabled && validateUtf8 &&
+      child.dataType == BinaryType && dataType.isInstanceOf[StringType]

Review Comment:
   Here we basically matching on the types of the `from` and `to` which also done in `Cast.forceNullable()`. 
   So I think we should consider moving this logic there.
   Especially as `Cast.forceNullable()` used in several places even around complex types like 'ArrayType' ,`MapType`, 'StructType'. 
   
   If those are relevant we need to test those cases too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org