You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "eric-maynard (via GitHub)" <gi...@apache.org> on 2024/05/06 22:21:04 UTC
[PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
eric-maynard opened a new pull request, #46408:
URL: https://github.com/apache/spark/pull/46408
### What changes were proposed in this pull request?
Currently, when reading a JSON like this:
```
{"a": {"b": -999.99999999999999999999999999999999995}}
```
With the schema:
```
a STRING
```
Spark will yield a result like this:
```
{"b": -1000.0}
```
### Why are the changes needed?
Currently, when reading a JSON like this:
```
{"a": {"b": -999.99999999999999999999999999999999995}}
```
With the schema:
```
a STRING
```
Spark will yield a result like this:
```
{"b": -1000.0}
```
Other changes such as changes to the input string's whitespace may also occur.
### Does this PR introduce _any_ user-facing change?
Yes, when reading non-string fields from a JSON object using the STRING type, we will now extract the field exactly as it appears.
### How was this patch tested?
Added a test in `JsonSuite.scala`
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "sandip-db (via GitHub)" <gi...@apache.org>.
sandip-db commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2225838233
I did some tests and it looks like that this PR doesn't address the same issue with from_json. Json expressions including from_json use [CreateJacksonParser.utf8String](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L706) to create jackson parser, which in turn converts UTF8String to an InputStreamReader. The latter doesn't allow to copy buffers from random offset. I tried to create jackson parser using the underlying byte array of UTF8String, but ran into [SPARK-16548](https://issues.apache.org/jira/browse/SPARK-16548) (that was fixed by https://github.com/apache/spark/pull/17693).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #46408:
URL: https://github.com/apache/spark/pull/46408#discussion_r1594623260
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -280,13 +280,32 @@ class JacksonParser(
case VALUE_STRING =>
UTF8String.fromString(parser.getText)
- case _ =>
+ case other =>
// Note that it always tries to convert the data as string without the case of failure.
- val writer = new ByteArrayOutputStream()
- Utils.tryWithResource(factory.createGenerator(writer, JsonEncoding.UTF8)) {
- generator => generator.copyCurrentStructure(parser)
+ val startLocation = parser.getTokenLocation
+ startLocation.contentReference().getRawContent match {
Review Comment:
I was not able to find such an existing API -- there is `JacksonParser.getText` but that appears to simply get the current value if it's a string value.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -280,13 +280,32 @@ class JacksonParser(
case VALUE_STRING =>
UTF8String.fromString(parser.getText)
- case _ =>
+ case other =>
// Note that it always tries to convert the data as string without the case of failure.
- val writer = new ByteArrayOutputStream()
- Utils.tryWithResource(factory.createGenerator(writer, JsonEncoding.UTF8)) {
- generator => generator.copyCurrentStructure(parser)
+ val startLocation = parser.getTokenLocation
+ startLocation.contentReference().getRawContent match {
Review Comment:
wrt. multiline JSON, I have added a test to cover this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2230017438
Hm, I think this PR also doesn't cover multiline cases.
Another problem is that it seems `parser.currentLocation.getByteOffset` doesn't report correctly too so we cannot really get the current byte offset ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2103174421
Hey @HyukjinKwon, can you take another look and possible re-trigger tests? I believe multiline should be working now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2230017862
This is what I tried:
```diff
ff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/CreateJacksonParser.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/cat
alyst/json/CreateJacksonParser.scala
index ba7b54fc04e8..721d611bc414 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/CreateJacksonParser.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/CreateJacksonParser.scala
@@ -28,6 +28,10 @@ import org.apache.hadoop.io.Text
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.unsafe.types.UTF8String
+private[sql] trait WithByteArrayInputStream {
+ val source: ByteArrayInputStream
+}
+
object CreateJacksonParser extends Serializable {
def string(jsonFactory: JsonFactory, record: String): JsonParser = {
jsonFactory.createParser(record)
@@ -40,7 +44,10 @@ object CreateJacksonParser extends Serializable {
val bain = new ByteArrayInputStream(
bb.array(), bb.arrayOffset() + bb.position(), bb.remaining())
- jsonFactory.createParser(new InputStreamReader(bain, StandardCharsets.UTF_8))
+ jsonFactory.createParser(
+ new InputStreamReader(bain, StandardCharsets.UTF_8) with WithByteArrayInputStream {
+ override val source: ByteArrayInputStream = bain
+ })
}
def text(jsonFactory: JsonFactory, record: Text): JsonParser = {
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/
json/JacksonParser.scala
index 32a1731a93d4..d98a14302d5a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.catalyst.json
-import java.io.{ByteArrayOutputStream, CharConversionException}
+import java.io.{ByteArrayOutputStream, CharConversionException, InputStreamReader}
import java.nio.charset.MalformedInputException
import scala.collection.mutable.ArrayBuffer
@@ -332,6 +332,14 @@ class JacksonParser(
val buffer = new Array[Byte](size)
positionedReadable.read(startLocation.getByteOffset, buffer, 0, size)
UTF8String.fromBytes(buffer, 0, size)
+ case inputStream: InputStreamReader with WithByteArrayInputStream =>
+ skipAhead()
+ val endLocation = parser.currentLocation.getByteOffset
+
+ val size = endLocation.toInt - (startLocation.getByteOffset.toInt)
+ val buffer = new Array[Byte](size)
+ inputStream.source.read(buffer, startLocation.getByteOffset.toInt, size)
+ UTF8String.fromBytes(buffer, 0, size)
case _ =>
val writer = new ByteArrayOutputStream()
Utils.tryWithResource(factory.createGenerator(writer, JsonEncoding.UTF8)) {
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark
/sql/catalyst/expressions/JsonExpressionsSuite.scala
index a23e7f44a48d..12cc8165a39e 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala
@@ -432,6 +432,16 @@ class JsonExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper with
)
}
+ test("from_json - nested object to string") {
+ val jsonData = """{"a": {"b": -999.99999999999999999999999999999999995}}"""
+ val schema = StructType(StructField("a", StringType) :: Nil)
+ checkEvaluation(
+ JsonToStructs(schema, Map.empty, Literal(jsonData), UTC_OPT),
+ InternalRow("""{"b": -999.99999999999999999999999999999999995}""")
+ )
+ }
+
+
test("from_json - invalid data") {
val jsonData = """{"a" 1}"""
val schema = StructType(StructField("a", IntegerType) :: Nil)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2229681590
Let me take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #46408:
URL: https://github.com/apache/spark/pull/46408#discussion_r1594851880
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
Review Comment:
```suggestion
expectedExactData = Seq("""{"white": "space"}""")
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
Review Comment:
```suggestion
expectedInexactData = Seq("""{"white":"space"}"""),
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
Review Comment:
```suggestion
"""{"data": {"white": "space"}}""",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "revans2 (via GitHub)" <gi...@apache.org>.
revans2 commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2222973633
@eric-maynard and @HyukjinKwon any update on fixing at least from_json if not also get_json_object? If you need me to try and try and debug this and put up a patch I am happy to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #46408:
URL: https://github.com/apache/spark/pull/46408#discussion_r1595695553
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3900,26 +3900,26 @@ abstract class JsonSuite
}
}
extractData(
- s"""{"data": {"white": "space"}}""",
- expectedInexactData = Seq(s"""{"white":"space"}"""),
- expectedExactData = Seq(s"""{"white": "space"}""")
+ """{"data": {"white": "space"}}""",
+ expectedInexactData = Seq("""{"white":"space"}"""),
+ expectedExactData = Seq("""{"white": "space"}""")
)
extractData(
- s"""{"data": ["white", "space"]}""",
- expectedInexactData = Seq(s"""["white","space"]"""),
- expectedExactData = Seq(s"""["white", "space"]""")
+ """{"data": ["white", "space"]}""",
+ expectedInexactData = Seq("""["white","space"]"""),
+ expectedExactData = Seq("""["white", "space"]""")
)
val granularFloat = "-999.99999999999999999999999999999999995"
extractData(
s"""{"data": {"v": ${granularFloat}}}""",
- expectedInexactData = Seq(s"""{"v":-1000.0}"""),
+ expectedInexactData = Seq("""{"v":-1000.0}"""),
expectedExactData = Seq(s"""{"v": ${granularFloat}}""")
)
// In multiLine, we fall back to the inexact method:
extractData(
- s"""{"data": {"white":\n"space"}}""",
Review Comment:
This makes `\n` no longer function as a newline here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #46408:
URL: https://github.com/apache/spark/pull/46408#discussion_r1594852087
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
+ expectedExactData = Seq(s"""["white", "space"]""")
Review Comment:
```suggestion
expectedExactData = Seq("""["white", "space"]""")
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
Review Comment:
```suggestion
expectedInexactData = Seq("""["white","space"]"""),
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
Review Comment:
```suggestion
"""{"data": ["white", "space"]}""",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2103609100
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2223500412
ping @eric-maynard
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #46408:
URL: https://github.com/apache/spark/pull/46408#discussion_r1594851819
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
Review Comment:
```suggestion
expectedInexactData = Seq("""{"white":"space"}"""),
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
Review Comment:
```suggestion
"""{"data": ["white", "space"]}""",
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
Review Comment:
```suggestion
expectedExactData = Seq("""{"white": "space"}""")
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
Review Comment:
```suggestion
"""{"data": {"white": "space"}}""",
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
Review Comment:
```suggestion
expectedInexactData = Seq("""["white","space"]"""),
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
+ expectedExactData = Seq(s"""["white", "space"]""")
+ )
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ extractData(
+ s"""{"data": {"v": ${granularFloat}}}""",
+ expectedInexactData = Seq(s"""{"v":-1000.0}"""),
+ expectedExactData = Seq(s"""{"v": ${granularFloat}}""")
+ )
+ // In multiLine, we fall back to the inexact method:
+ extractData(
+ s"""{"data": {"white":\n"space"}}""",
Review Comment:
```suggestion
"""{"data": {"white":\n"space"}}""",
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
+ expectedExactData = Seq(s"""["white", "space"]""")
+ )
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ extractData(
+ s"""{"data": {"v": ${granularFloat}}}""",
+ expectedInexactData = Seq(s"""{"v":-1000.0}"""),
+ expectedExactData = Seq(s"""{"v": ${granularFloat}}""")
+ )
+ // In multiLine, we fall back to the inexact method:
+ extractData(
+ s"""{"data": {"white":\n"space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
Review Comment:
```suggestion
expectedInexactData = Seq("""{"white":"space"}"""),
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
+ expectedExactData = Seq(s"""["white", "space"]""")
Review Comment:
```suggestion
expectedExactData = Seq("""["white", "space"]""")
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
+ expectedExactData = Seq(s"""["white", "space"]""")
+ )
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ extractData(
+ s"""{"data": {"v": ${granularFloat}}}""",
+ expectedInexactData = Seq(s"""{"v":-1000.0}"""),
+ expectedExactData = Seq(s"""{"v": ${granularFloat}}""")
+ )
+ // In multiLine, we fall back to the inexact method:
+ extractData(
+ s"""{"data": {"white":\n"space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white":"space"}"""),
Review Comment:
```suggestion
expectedExactData = Seq("""{"white":"space"}"""),
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
+ expectedExactData = Seq(s"""["white", "space"]""")
+ )
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ extractData(
+ s"""{"data": {"v": ${granularFloat}}}""",
+ expectedInexactData = Seq(s"""{"v":-1000.0}"""),
Review Comment:
```suggestion
expectedInexactData = Seq("""{"v":-1000.0}"""),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #46408:
URL: https://github.com/apache/spark/pull/46408#discussion_r1594852326
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
+ expectedExactData = Seq(s"""["white", "space"]""")
+ )
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ extractData(
+ s"""{"data": {"v": ${granularFloat}}}""",
+ expectedInexactData = Seq(s"""{"v":-1000.0}"""),
+ expectedExactData = Seq(s"""{"v": ${granularFloat}}""")
+ )
+ // In multiLine, we fall back to the inexact method:
+ extractData(
+ s"""{"data": {"white":\n"space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white":"space"}"""),
Review Comment:
```suggestion
expectedExactData = Seq("""{"white":"space"}"""),
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
+ expectedExactData = Seq(s"""["white", "space"]""")
+ )
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ extractData(
+ s"""{"data": {"v": ${granularFloat}}}""",
+ expectedInexactData = Seq(s"""{"v":-1000.0}"""),
+ expectedExactData = Seq(s"""{"v": ${granularFloat}}""")
+ )
+ // In multiLine, we fall back to the inexact method:
+ extractData(
+ s"""{"data": {"white":\n"space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
Review Comment:
```suggestion
expectedInexactData = Seq("""{"white":"space"}"""),
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
+ expectedExactData = Seq(s"""["white", "space"]""")
+ )
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ extractData(
+ s"""{"data": {"v": ${granularFloat}}}""",
+ expectedInexactData = Seq(s"""{"v":-1000.0}"""),
+ expectedExactData = Seq(s"""{"v": ${granularFloat}}""")
+ )
+ // In multiLine, we fall back to the inexact method:
+ extractData(
+ s"""{"data": {"white":\n"space"}}""",
Review Comment:
```suggestion
"""{"data": {"white":\n"space"}}""",
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,65 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: values are unchanged when read as string") {
+ withTempPath { path =>
+ def extractData(
+ jsonString: String,
+ expectedInexactData: Seq[String],
+ expectedExactData: Seq[String],
+ multiLine: Boolean = false): Unit = {
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .mode("overwrite")
+ .text(path.getAbsolutePath)
+
+ withClue("Exact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "true") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedExactData.map(d => Row(d)))
+ }
+ }
+
+ withClue("Inexact string parsing") {
+ withSQLConf(SQLConf.JSON_EXACT_STRING_PARSING.key -> "false") {
+ val df = spark.read
+ .schema("data STRING")
+ .option("multiLine", multiLine.toString)
+ .json(path.getAbsolutePath)
+ checkAnswer(df, expectedInexactData.map(d => Row(d)))
+ }
+ }
+ }
+ extractData(
+ s"""{"data": {"white": "space"}}""",
+ expectedInexactData = Seq(s"""{"white":"space"}"""),
+ expectedExactData = Seq(s"""{"white": "space"}""")
+ )
+ extractData(
+ s"""{"data": ["white", "space"]}""",
+ expectedInexactData = Seq(s"""["white","space"]"""),
+ expectedExactData = Seq(s"""["white", "space"]""")
+ )
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ extractData(
+ s"""{"data": {"v": ${granularFloat}}}""",
+ expectedInexactData = Seq(s"""{"v":-1000.0}"""),
Review Comment:
```suggestion
expectedInexactData = Seq("""{"v":-1000.0}"""),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #46408:
URL: https://github.com/apache/spark/pull/46408#discussion_r1591615927
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -280,13 +280,32 @@ class JacksonParser(
case VALUE_STRING =>
UTF8String.fromString(parser.getText)
- case _ =>
+ case other =>
// Note that it always tries to convert the data as string without the case of failure.
- val writer = new ByteArrayOutputStream()
- Utils.tryWithResource(factory.createGenerator(writer, JsonEncoding.UTF8)) {
- generator => generator.copyCurrentStructure(parser)
+ val startLocation = parser.getTokenLocation
+ startLocation.contentReference().getRawContent match {
+ case byteArray: Array[Byte] =>
+ other match {
+ case START_OBJECT =>
+ parser.skipChildren()
+ case START_ARRAY =>
+ parser.skipChildren()
+ case _ =>
+ // Do nothing in this case; we've already read the token
+ }
+ val endLocation = parser.currentLocation.getByteOffset
+
+ UTF8String.fromBytes(
+ byteArray,
+ startLocation.getByteOffset.toInt,
+ endLocation.toInt - (startLocation.getByteOffset.toInt))
+ case _ =>
Review Comment:
It's not clear to me when or if this branch would occur, but I have preserved the existing code here to cover cases where the input may not be readily indexed by a byte offset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #46408:
URL: https://github.com/apache/spark/pull/46408#discussion_r1594767881
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,24 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: decimal precision is preserved when object is read as string") {
+ withTempPath { path =>
+
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ val jsonString = s"""{"data": {"v": ${granularFloat}}}, {"data": {"v": ${granularFloat}}}]"""
+
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .text(path.getAbsolutePath)
+
+ val df = spark.read.schema("data STRING").json(path.getAbsolutePath)
+
+ val expected = s"""{"v": ${granularFloat}}"""
Review Comment:
Added more tests -- can you clarify the last example and what we expect that to do? It seems like invalid JSON
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "sadikovi (via GitHub)" <gi...@apache.org>.
sadikovi commented on code in PR #46408:
URL: https://github.com/apache/spark/pull/46408#discussion_r1594580639
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -280,13 +280,32 @@ class JacksonParser(
case VALUE_STRING =>
UTF8String.fromString(parser.getText)
- case _ =>
+ case other =>
// Note that it always tries to convert the data as string without the case of failure.
- val writer = new ByteArrayOutputStream()
- Utils.tryWithResource(factory.createGenerator(writer, JsonEncoding.UTF8)) {
- generator => generator.copyCurrentStructure(parser)
+ val startLocation = parser.getTokenLocation
+ startLocation.contentReference().getRawContent match {
Review Comment:
Is there an existing API to get the remaining content as string? Also, would it work with multi-line JSON?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,24 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: decimal precision is preserved when object is read as string") {
+ withTempPath { path =>
+
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ val jsonString = s"""{"data": {"v": ${granularFloat}}}, {"data": {"v": ${granularFloat}}}]"""
Review Comment:
The JSON string appears to be invalid, for example, it ends with `]` but I don't see any opening bracket for it.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,24 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: decimal precision is preserved when object is read as string") {
+ withTempPath { path =>
+
Review Comment:
nit: You can remove the new line on L3871.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -3865,6 +3865,24 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-48148: decimal precision is preserved when object is read as string") {
+ withTempPath { path =>
+
+ val granularFloat = "-999.99999999999999999999999999999999995"
+ val jsonString = s"""{"data": {"v": ${granularFloat}}}, {"data": {"v": ${granularFloat}}}]"""
+
+ Seq(jsonString).toDF()
+ .repartition(1)
+ .write
+ .text(path.getAbsolutePath)
+
+ val df = spark.read.schema("data STRING").json(path.getAbsolutePath)
+
+ val expected = s"""{"v": ${granularFloat}}"""
Review Comment:
Can you add more test cases for the following?
- {"data": {"v": "abc"}}, expected: "{"v": "abc"}"
- {"data":{"v": "0.999"}}, expected: "{"v": "0.999"}"
- {"data": [1, 2, 3]}, expected: "[1, 2, 3]"
- {"data": <deeply-nested-object>}, expected the object as string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #46408: [SPARK-48148][CORE] JSON objects should not be modified when read as STRING
URL: https://github.com/apache/spark/pull/46408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2101842606
```
SPARK-48148: values are unchanged when read as string *** FAILED *** (134 milliseconds)
```
seems it fails
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2103609498
btw you can trigger on your own https://github.com/eric-maynard/spark/runs/24789350525 I can't trigger :-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #46408:
URL: https://github.com/apache/spark/pull/46408#discussion_r1594623695
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -280,13 +280,32 @@ class JacksonParser(
case VALUE_STRING =>
UTF8String.fromString(parser.getText)
- case _ =>
+ case other =>
// Note that it always tries to convert the data as string without the case of failure.
- val writer = new ByteArrayOutputStream()
- Utils.tryWithResource(factory.createGenerator(writer, JsonEncoding.UTF8)) {
- generator => generator.copyCurrentStructure(parser)
+ val startLocation = parser.getTokenLocation
+ startLocation.contentReference().getRawContent match {
Review Comment:
wrt. multiline JSON, I have added a test to cover this. It seems that the content reference is not a byte array when using multiline mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "sadikovi (via GitHub)" <gi...@apache.org>.
sadikovi commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2101372332
cc @dongjoon-hyun @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48148][CORE] JSON objects should not be modified when read as STRING [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46408:
URL: https://github.com/apache/spark/pull/46408#issuecomment-2223501345
sent offline ping too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org