[GitHub] [spark] southernriver commented on a change in pull request #20739: [SPARK-23603][SQL]When the length of the json is in a range, get_json_object will result in missing tail data

[GitBox <gi...@apache.org>]

southernriver commented on a change in pull request #20739: [SPARK-23603][SQL]When the length of the json is in a range,get_json_object will result in missing tail data
URL: https://github.com/apache/spark/pull/20739#discussion_r282805338
 
 

 ##########
 File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
 ##########
 @@ -215,12 +215,20 @@ case class GetJsonObject(json: Expression, path: Expression)
       path: List[PathInstruction]): Boolean = {
     (p.getCurrentToken, path) match {
       case (VALUE_STRING, Nil) if style == RawStyle =>
-        // there is no array wildcard or slice parent, emit this string without quotes
-        if (p.hasTextCharacters) {
-          g.writeRaw(p.getTextCharacters, p.getTextOffset, p.getTextLength)
-        } else {
-          g.writeRaw(p.getText)
-        }
+
+        // Jackson(>=2.7.7) fixes the possibility of missing tail data
+        // when the length of the value is in a range
+        // Now we use the jackson version is 2.6.x
+        // So comment calls the code for this method ( writeRaw(char[] text, int offset, int len) )
+        // Although using writeRaw(String text) will lose some performance
+        g.writeRaw(p.getText)
 
 Review comment:
   What is the different between  ` writeRaw(String text)`  and  `writeRaw(String text, int offset, int len)`,I have seen the source code of both functions,  but I don't know why this action could solve this? thank you !
   code for 2.6.X
   ```
   public void writeRaw(String text)
           throws IOException, JsonGenerationException
       {
           int start = 0;
           int len = text.length();
           while (len > 0) {
               char[] buf = _charBuffer;
               final int blen = buf.length;
               final int len2 = (len < blen) ? len : blen;
               text.getChars(start, start+len2, buf, 0);
               writeRaw(buf, 0, len2);
               start += len2;
               len -= len2;
           }
       }
   @Override
       public void writeRaw(String text, int offset, int len)
           throws IOException, JsonGenerationException
       {
           while (len > 0) {
               char[] buf = _charBuffer;
               final int blen = buf.length;
               final int len2 = (len < blen) ? len : blen;
               text.getChars(offset, offset+len2, buf, 0);
               writeRaw(buf, 0, len2);
               offset += len2;
               len -= len2;
           }
       }
   ```
   
   I debug that the value of `p.getTextOffset` is not zero and which is exactly the missing length of string!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org