You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2024/07/12 10:49:47 UTC

[PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

wForget opened a new pull request, #47325:
URL: https://github.com/apache/spark/pull/47325

   Backports #47303 to 3.5
   
   ### What changes were proposed in this pull request?
   
   [[SPARK-47307](https://issues.apache.org/jira/browse/SPARK-47307)] Add a config to optionally chunk base64 strings
   
   ### Why are the changes needed?
   
   In #35110, it was incorrectly asserted that:
   
   > ApacheCommonBase64 obeys http://www.ietf.org/rfc/rfc2045.txt
   
   This is not true as the previous code called:
   
   ```java
   public static byte[] encodeBase64(byte[] binaryData)
   ```
   
   Which states:
   
   > Encodes binary data using the base64 algorithm but does not chunk the output.
   
   However, the RFC 2045 (MIME) base64 encoder does chunk by default. This now means that any Spark encoded base64 strings cannot be decoded by encoders that do not implement RFC 2045. The docs state RFC 4648.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No
   
   ### How was this patch tested?
   
   Existing test suite.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
    No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2226726577

   Should we backport https://github.com/apache/spark/pull/47017
   +1 @wForget 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2229888038

   Merged to branch 3.5 for 3.5.2, thanks @wForget 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #47325:
URL: https://github.com/apache/spark/pull/47325#discussion_r1676826574


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3229,6 +3229,15 @@ object SQLConf {
     .booleanConf
     .createWithDefault(false)
 
+  val CHUNK_BASE64_STRING_ENABLED = buildConf("spark.sql.legacy.chunkBase64String.enabled")
+    .internal()
+    .doc("Whether to truncate string generated by the `Base64` function. When true, base64" +
+      " strings generated by the base64 function are chunked into lines of at most 76" +
+      " characters. When false, the base64 strings are not chunked.")
+    .version("3.5.2")

Review Comment:
   Thank you @dongjoon-hyun. 
   
   Not related to this PR, maybe we shall add multiple fixed version in this field, such as `3.4.4, 3.5.2, 4.0.0`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2226711301

   > ```
   > [info]   Cause: org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot evaluate expression: base64(0x61616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161)
   > [info]   at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
   > [info]   at org.apache.spark.SparkException$.internalError(SparkException.scala:96)
   > ```
   > 
   > can you check these test failures?
   
   Looks like it's missing https://github.com/apache/spark/commit/fb5697d6873ce8a08a2f5b6e4dd7539e557a69a2#diff-fcaa047af256f3afb055c3e5d466d5a0fe94851bd6cc8f96ac04673d52e1a321
   
   Should we backport #47017 or introduce this change in this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2227463911

   However, SPARK-48658 was merged as an improvement JIRA, @yaooqinn . Do you mean we need to convert it as a bug fix?
   
   ![Screenshot 2024-07-14 at 12 45 23](https://github.com/user-attachments/assets/5f956d64-4ff9-4e9a-91b6-ed11d35189da)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #47325: [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings
URL: https://github.com/apache/spark/pull/47325


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #47325:
URL: https://github.com/apache/spark/pull/47325#discussion_r1676306296


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3229,6 +3229,15 @@ object SQLConf {
     .booleanConf
     .createWithDefault(false)
 
+  val CHUNK_BASE64_STRING_ENABLED = buildConf("spark.sql.legacy.chunkBase64String.enabled")
+    .internal()
+    .doc("Whether to truncate string generated by the `Base64` function. When true, base64" +
+      " strings generated by the base64 function are chunked into lines of at most 76" +
+      " characters. When false, the base64 strings are not chunked.")
+    .version("3.5.2")

Review Comment:
   Hmm. Got it. I saw this comment from Apache Spark 3.5.2 release manager, @yaooqinn .
   - https://github.com/apache/spark/pull/47303/files#r1675645743
   
   If then, are we going to update these values from `master` and `branch-3.5` when we do the release of Apache Spark 3.4.4? I'm fine if we are going to do in that way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #47325:
URL: https://github.com/apache/spark/pull/47325#discussion_r1676304171


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3229,6 +3229,15 @@ object SQLConf {
     .booleanConf
     .createWithDefault(false)
 
+  val CHUNK_BASE64_STRING_ENABLED = buildConf("spark.sql.legacy.chunkBase64String.enabled")
+    .internal()
+    .doc("Whether to truncate string generated by the `Base64` function. When true, base64" +
+      " strings generated by the base64 function are chunked into lines of at most 76" +
+      " characters. When false, the base64 strings are not chunked.")
+    .version("3.5.2")

Review Comment:
   IIUC, if we start to backport SPARK-47307, it will go to Apache Spark 3.4.4 together, right? In that case, I'm curious if 3.5.2 is correct.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2227774699

   > @wForget Can you raise another PR to 'master' to add a migration guide for migrating 3.5.1 to 3.5.2?
   
   Sure, I will do it later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2227464217

   If we need to change the issue type, please comment on your initial PR to get a consensus.
   - https://github.com/apache/spark/pull/47017 .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "Kimahriman (via GitHub)" <gi...@apache.org>.
Kimahriman commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2321342164

   FYI this changed the nullability of base64 to always be nullable instead of depending on the input, this broke some of our stateful streams with a schema mismatch error when upgrading from 3.5.1 to 3.5.2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2225936197

   ```
   [info]   Cause: org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot evaluate expression: base64(0x61616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161)
   [info]   at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
   [info]   at org.apache.spark.SparkException$.internalError(SparkException.scala:96)
   ```
   
   can you check these test failures?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #47325:
URL: https://github.com/apache/spark/pull/47325#discussion_r1677194257


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3229,6 +3229,15 @@ object SQLConf {
     .booleanConf
     .createWithDefault(false)
 
+  val CHUNK_BASE64_STRING_ENABLED = buildConf("spark.sql.legacy.chunkBase64String.enabled")
+    .internal()
+    .doc("Whether to truncate string generated by the `Base64` function. When true, base64" +
+      " strings generated by the base64 function are chunked into lines of at most 76" +
+      " characters. When false, the base64 strings are not chunked.")
+    .version("3.5.2")

Review Comment:
   Theoretically, it's possible, but it will enforce us to update all the existing configurations and documentations. So, we had better not to because it could be too much.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2227614856

   Okay, based on the information provided by @dongjoon-hyun and the [Policy of backporting bugfiexes](https://www.mail-archive.com/dev@spark.apache.org/msg10284.html), I think we can only fix the test errors instead of backporting [SPARK-48658](https://issues.apache.org/jira/browse/SPARK-48658)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2230214873

   Thank you, @wForget and @yaooqinn !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2323946020

   @Kimahriman thanks for reporting! I'm fixing it at https://github.com/apache/spark/pull/47952


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47307][SQL][3.5] Add a config to optionally chunk base64 strings [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #47325:
URL: https://github.com/apache/spark/pull/47325#issuecomment-2226129231

   cc @cloud-fan , too


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org