You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "tdcmeehan (via GitHub)" <gi...@apache.org> on 2026/09/10 15:29:06 UTC
[PR] [SQL] Stream multiline top-level JSON arrays [spark]
tdcmeehan opened a new pull request, #58704:
URL: https://github.com/apache/spark/pull/58704
### What changes were proposed in this pull request?
This PR changes multiline JSON reading so that a top-level array of structs is consumed lazily from a single Jackson parser instead of materializing the entire array before returning rows.
It also extends `FailureSafeParser` to handle failures raised while advancing a lazy parser iterator, preserves rows emitted before later structural corruption, and adds benchmark cases for both many small elements and fewer large elements.
### Why are the changes needed?
The existing parser materializes every element of a multiline top-level JSON array in memory. Large arrays can therefore require memory proportional to the complete document even though Spark consumes the result as an iterator. Streaming elements bounds parser-side memory while retaining the existing valid-row and parse-mode behavior.
### Does this PR introduce _any_ user-facing change?
Yes. Reading a multiline top-level JSON array no longer materializes the complete array before producing rows. If malformed structure is encountered after valid rows have already been emitted, those rows remain available and the malformed document is then handled according to the configured parse mode.
### How was this patch tested?
Added regression coverage for lazy top-level array parsing and malformed input after partial output, and extended the existing SPARK-3308 top-level-array coverage.
Ran:
- `catalyst/compile`
- `sql/Test/compile`
- `sql/testOnly org.apache.spark.sql.execution.datasources.json.JsonV1Suite` (160 tests)
- `sql/testOnly org.apache.spark.sql.execution.datasources.json.JsonV2Suite` (161 tests)
- `catalyst/scalastyle`
- `sql/scalastyle`
- `sql/Test/scalastyle`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4016956145
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +745,86 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = createParser(factory, record)
+ def fail(error: Throwable): Nothing = {
+ try jsonParser.close() catch {
+ case NonFatal(closeError) => error.addSuppressed(closeError)
+ }
+ throw badRecord(error, () => recordLiteral(record))
+ }
+ def handleFailure[T](operation: => T): T = {
+ try operation catch {
+ case e: SparkUpgradeException => fail(e)
+ case e: CharConversionException if options.encoding.isEmpty => fail(e)
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
Review Comment:
**Non-blocking (P2):** A `PartialResultException` here is recoverable at the array-element boundary: the current object has been consumed and later elements remain in the parser. Closing the parser and replacing the source iterator causes inputs such as `[{"a":"bad"},{"a":2}]` (with `a int`) to emit only the partial/corrupt first row and silently drop the valid second row. Please preserve the live source iterator while emitting the recovery row, then resume with the next element; terminal structural failures should still close it.
See **Shared repair plan 1** in the review body.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:
##########
@@ -414,8 +414,15 @@ object MultiLineJsonDataSource extends JsonDataSource {
schema,
parser.options.columnNameOfCorruptRecord)
- safeParser.parse(
- CodecStreams.createInputStreamWithCloseResource(conf, file.toPath))
+ val input = CodecStreams.createInputStreamWithCloseResource(conf, file.toPath)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit](_ => input.close()))
Review Comment:
**Nit (P3):** `CodecStreams.createInputStreamWithCloseResource` already registers this returned stream for task-completion cleanup. Adding another listener here retains a redundant callback for every file and closes the same stream twice; please rely on the helper's existing ownership.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +745,86 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = createParser(factory, record)
Review Comment:
**Non-blocking (P2):** `createParser` can fail before `nextToken` while detecting an unsupported encoding (for example, Jackson throws `CharConversionException` for bytes `00 00 ff fe`). Because this call is outside `handleFailure`, that exception escapes `FailureSafeParser` even in PERMISSIVE or DROPMALFORMED mode. Please include parser acquisition in the same bad-record conversion boundary as iterator advancement.
See **Shared repair plan 1** in the review body.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -1066,6 +1069,97 @@ abstract class JsonSuite
}
}
+ gridTest("SPARK-3308 Read multiline top level JSON arrays")(
+ Seq(false, true)) { enabled =>
+ withSQLConf(SQLConf.JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY.key -> enabled.toString) {
+ withTempPath { file =>
+ Files.write(file.toPath, """[{"a":1},{"a":2}]""".getBytes(StandardCharsets.UTF_8))
+ checkAnswer(
+ spark.read.option("multiLine", true).schema("a int").json(file.getCanonicalPath),
+ Seq(Row(1), Row(2)))
+ }
+ }
+ }
+
+ test("multiline top level JSON arrays are parsed lazily") {
+ val schema = StructType(Seq(StructField("a", IntegerType)))
+ val options = new JSONOptions(Map("multiLine" -> "true"), SQLConf.get.sessionLocalTimeZone)
+ val parser = new JacksonParser(schema, options, allowArrayAsStructs = true)
+ val input = new ByteArrayInputStream(
+ s"""[{"a":1},{"a":2,"payload":"${"x" * 200000}"}]""".getBytes(StandardCharsets.UTF_8))
+ val rows = parser.parseIterator[InputStream](
+ input,
+ CreateJacksonParser.inputStream(_: JsonFactory, _: InputStream),
+ stream => UTF8String.fromBytes(stream.readAllBytes()))
+
+ assert(rows.next().getInt(0) === 1)
+ assert(input.available() > 0)
+ }
+
+ test("non-array JSON is parsed eagerly by parseIterator") {
+ val schema = StructType(Seq(StructField("a", IntegerType)))
+ val options = new JSONOptions(Map("multiLine" -> "true"), SQLConf.get.sessionLocalTimeZone)
+ val parser = new JacksonParser(schema, options, allowArrayAsStructs = true)
+ var closed = false
+ val input = new ByteArrayInputStream("""{"a":1}""".getBytes(StandardCharsets.UTF_8)) {
+ override def close(): Unit = {
+ closed = true
+ super.close()
+ }
+ }
+
+ val rows = parser.parseIterator[InputStream](
+ input,
+ CreateJacksonParser.inputStream(_: JsonFactory, _: InputStream),
+ stream => UTF8String.fromBytes(stream.readAllBytes()))
+
+ assert(closed)
+ assert(rows.next().getInt(0) === 1)
+ assert(!rows.hasNext)
+ }
+
+ gridTest("multiline top level JSON array keeps rows emitted before malformed input")(
Review Comment:
**Non-blocking (P2):** This matrix covers terminal JSON syntax failure, but it does not enter the recoverable `PartialResultException` state or a failure during `createParser`; existing partial-result tests use the eager parser. Archive tests also leave the new setting disabled, so the new `readStream` branch is never exercised. Please add enabled-path coverage for a partial element followed by valid rows, parser-construction failure under each parse mode, and an archive entry.
See **Shared repair plan 1** in the review body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4057250910
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +746,107 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = try {
+ createParser(factory, record)
+ } catch {
+ case e: SparkUpgradeException => throw e
+ case e: CharConversionException if options.encoding.isEmpty =>
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ // Abandoning the iterator before END_ARRAY (e.g. a LIMIT) skips finish()/fail(), so close the
+ // parser at task completion; close() is idempotent with those eager closes.
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit](_ => jsonParser.close()))
+ def fail(error: Throwable): Nothing = {
+ try jsonParser.close() catch {
+ case NonFatal(closeError) => error.addSuppressed(closeError)
+ }
+ throw badRecord(error, () => recordLiteral(record))
+ }
+ def handleFailure[T](recoverPartialResult: Boolean)(operation: => T): T = {
+ try operation catch {
+ case e: SparkUpgradeException => fail(e)
+ case e: CharConversionException if options.encoding.isEmpty => fail(e)
+ case e: PartialResultException
+ if recoverPartialResult && options.parseMode != FailFastMode =>
+ throw badRecord(e, () => recordLiteral(record)).copy(recoverable = true)
+ case e: PartialResultException =>
+ fail(e)
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
Review Comment:
**Non-blocking (P2):** An incompatible scalar element reaches this terminal RuntimeException arm even though its token is already at a safe top-level array-element boundary. FailureSafeParser then replaces the source iterator with only the recovery result, so PERMISSIVE and DROPMALFORMED reads of an input such as [{"a":1},42,{"a":2}] silently lose the final valid row. Please classify this scalar-boundary failure as recoverable while keeping nested object and array mismatches terminal.
**Recommended change:** Classify only scalar element-type conversion failures that remain at the current top-level element boundary as recoverable in non-FAILFAST modes, and cover scalars before, between, and after valid objects across parse modes and archive/file paths.
**Why this works:** Inspect the current element token and error category at the array-loop owner. Convert a safe scalar mismatch into a recoverable BadRecordException so FailureSafeParser emits or drops that element and resumes the preserved iterator; keep nested containers and structurally ambiguous failures terminal.
**Scope:** Complete element-boundary recovery without broadening recovery to undefined Jackson token positions.
**Compatibility:** Existing partial-object recovery, terminal malformed-document handling, corrupt-record text, and the disabled eager path remain unchanged.
**Risks:** An overly broad classification could resume from inside a malformed nested container and duplicate or misparse later tokens. An overly narrow classification could continue dropping later valid objects after other safely consumed scalar forms.
**Constraints:** Resume only when parser position is known to be at a complete top-level element boundary. FAILFAST must still fail immediately. Terminal structural corruption must still close and stop the iterator.
**Success:** In PERMISSIVE mode, an incompatible scalar produces one recovery row and later valid objects exactly once. In DROPMALFORMED mode, the incompatible scalar is omitted and later valid objects remain. In FAILFAST mode, the first incompatible scalar still raises the established malformed-record error. Nested or structurally ambiguous failures remain terminal and close the parser.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +746,107 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = try {
+ createParser(factory, record)
+ } catch {
+ case e: SparkUpgradeException => throw e
+ case e: CharConversionException if options.encoding.isEmpty =>
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ // Abandoning the iterator before END_ARRAY (e.g. a LIMIT) skips finish()/fail(), so close the
+ // parser at task completion; close() is idempotent with those eager closes.
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit](_ => jsonParser.close()))
+ def fail(error: Throwable): Nothing = {
+ try jsonParser.close() catch {
Review Comment:
**Non-blocking (P2):** The malformed and truncated-array tests do not observe this terminal cleanup side effect. If fail() stopped closing the parser, they would still return the same rows and errors while retaining the input until task completion. Please add a regression assertion that terminal advancement failures close the parser and underlying input synchronously.
**Recommended change:** Introduce one clearable parser-lifecycle owner captured by TaskContext, route every eager close through it, and add resource-sensitive JSON lifecycle coverage for empty input, empty arrays, normal exhaustion, terminal failure, and abandonment, including repeated archive entries.
**Why this works:** The task listener captures only a small owner that atomically consumes its current parser. Normal completion, terminal failure, and eager fallback use the same close-and-clear operation so a completed parser becomes unreachable immediately; task completion consumes and closes only a parser still owned because its iterator was abandoned.
**Scope:** Unify streaming parser ownership and verify every lifecycle transition without changing row or parse-mode semantics.
**Compatibility:** The enabled path remains lazy; disabled and excluded option paths remain eager; recovery and terminal error semantics do not change.
**Risks:** Clearing ownership before close completes could lose the only cleanup backstop after a close failure. Independent close routes could race or mask the original parse exception if they do not share one consume operation.
**Constraints:** Normal exhaustion and terminal failures must still close synchronously. Task completion must still close an iterator abandoned before END_ARRAY. Existing recoverable-element and parse-mode row behavior must remain unchanged.
**Success:** After normal, zero-row, eager-fallback, or terminal completion, TaskContext no longer retains the completed parser. An abandoned iterator still closes its current parser when the task completes. Repeated completed archive entries do not accumulate parser or entry-buffer ownership until task completion. Rows, corrupt-record values, and parse-mode errors remain unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #58704:
URL: https://github.com/apache/spark/pull/58704#issuecomment-5627808419
Mind taking a look at https://github.com/apache/spark/pull/58704/checks?check_run_id=103066275835?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4053512430
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +746,104 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = try {
+ createParser(factory, record)
+ } catch {
+ case e: SparkUpgradeException => throw e
+ case e: CharConversionException if options.encoding.isEmpty =>
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ // Abandoning the iterator before END_ARRAY (e.g. a LIMIT) skips finish()/fail(), so close the
+ // parser at task completion; close() is idempotent with those eager closes.
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit](_ => jsonParser.close()))
Review Comment:
**Non-blocking (P2):** The task-completion listener captures this JsonParser directly, and closing the parser does not clear that captured reference. On archive reads, the parser can retain the ByteArrayInputStream and complete byte array for its entry, so consuming many entries retains all completed entry buffers until task completion and can OOM despite the streaming path. Please make the fallback cleanup stop retaining a parser once its eager close path has run.
**Recommended change:** Replace the per-parser task-listener capture with one clearable parser-lifecycle owner used by normal completion, terminal failure, eager fallback, and task completion, and add lifecycle and archive-entry retention regression coverage.
**Why this works:** The task listener captures only a clearable ownership handle. Every eager close atomically consumes or clears that handle before releasing the parser, while task completion consumes the same handle and closes it only when an iterator was abandoned. Completed parsers therefore cease to be reachable from TaskContext without weakening abandonment cleanup.
**Scope:** Unify streaming JsonParser close ownership across eager and task-completion paths and verify that archive entries do not accumulate completed parser state.
**Compatibility:** Streaming remains lazy, terminal and recoverable failures retain their current parse-mode semantics, and task completion remains a cleanup backstop for early iterator abandonment.
**Risks:** A close route could clear ownership before the parser is actually closed, weakening cleanup after a close failure. Independent close paths could race or mask the original parse failure if they do not consume the shared owner consistently.
**Constraints:** Normal completion and terminal parse failures must still close eagerly. Task completion must still close the parser when a consumer abandons the iterator. Parse-mode recovery and lazy element delivery must remain unchanged.
**Success:** After an entry iterator finishes or fails, TaskContext no longer retains its parser or backing entry bytes. An iterator abandoned before completion still has its current parser closed at task completion. Successful, recoverable, and terminal parsing retain their current rows and parse-mode behavior.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +746,104 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
Review Comment:
**Non-blocking (P2):** JacksonParser construction has already called makeRootConverter(schema), whose struct branch builds the equivalent element converter. Calling makeConverter(schema) again here duplicates recursive converter construction for every file or archive entry, and the work is wasted entirely for non-array roots. Please reuse the existing converter graph or defer this construction until an array root has actually been observed.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -7034,6 +7034,16 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY =
+ buildConf("spark.sql.json.enableStreamingTopLevelArray")
+ .internal()
+ .doc("When true, multiline JSON reads stream the elements of a top-level array one at a " +
Review Comment:
**Nit (P3):** This description makes the setting sound unconditional, but the parser only selects the streaming path when neither singleVariantColumn nor explodeEmbeddedArray is active. With either option, the top-level array is still converted eagerly. Please qualify the description so it states the actual option scope.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -1066,6 +1070,282 @@ abstract class JsonSuite
}
}
+ gridTest("SPARK-3308 Read multiline top level JSON arrays")(
+ Seq(false, true)) { enabled =>
+ withSQLConf(SQLConf.JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY.key -> enabled.toString) {
+ withTempPath { file =>
+ Files.write(file.toPath, """[{"a":1},{"a":2}]""".getBytes(StandardCharsets.UTF_8))
Review Comment:
**Non-blocking (P2):** The new enabled-path cases do not exercise either zero-row branch: an empty document closes before the streaming iterator is constructed, while an empty array closes from the iterator's first prepare call. They also do not assert synchronous closure after normal array exhaustion. Please add cases that assert both zero returned rows and parser closure for the empty inputs, plus closure after fully consuming a normal array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4048017523
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +745,100 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = try {
+ createParser(factory, record)
+ } catch {
+ case e: SparkUpgradeException => throw e
+ case e: CharConversionException if options.encoding.isEmpty =>
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ def fail(error: Throwable): Nothing = {
+ try jsonParser.close() catch {
+ case NonFatal(closeError) => error.addSuppressed(closeError)
+ }
+ throw badRecord(error, () => recordLiteral(record))
+ }
+ def handleFailure[T](operation: => T): T = {
+ try operation catch {
+ case e: SparkUpgradeException => fail(e)
+ case e: CharConversionException if options.encoding.isEmpty => fail(e)
+ case e: PartialResultException if options.parseMode != FailFastMode =>
Review Comment:
**Non-blocking (P2):** This handler also runs while eagerly converting a non-array root. In that state a `PartialResultException` is raised before any source iterator can be returned, so marking it recoverable bypasses `fail()` and leaves the parser/input stream open after the recovery rows finish. A partition with many such files can accumulate descriptors until task completion. Please mark partial failures recoverable only inside the top-level-array iterator and close the parser for this fallback state.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONArchiveReadBase.scala:
##########
@@ -166,6 +166,16 @@ trait JSONArchiveReadBase extends ArchiveReadSuiteBase {
extraOptions = Map("multiLine" -> "true"))
}
+ test("JSON: streaming multi-line top-level arrays match a directory read") {
Review Comment:
**Non-blocking (P2):** This proves that the enabled archive path handles valid arrays, but it does not exercise `readStream`'s separate failure wiring or pre-buffered corrupt-record literal. The existing malformed archive test leaves the switch disabled, and the mode matrix in `JsonSuite` goes through `readFile`. Please add enabled archive cases for partial-element recovery and terminal malformed input across the parse modes, including an assertion that PERMISSIVE returns the complete entry text in `_corrupt_record`.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:
##########
@@ -414,8 +414,14 @@ object MultiLineJsonDataSource extends JsonDataSource {
schema,
parser.options.columnNameOfCorruptRecord)
- safeParser.parse(
- CodecStreams.createInputStreamWithCloseResource(conf, file.toPath))
+ val input = CodecStreams.createInputStreamWithCloseResource(conf, file.toPath)
+ if (parser.options.streamMultilineTopLevelArray) {
+ safeParser.parseIterator(
+ input,
+ input => parser.parseIterator[InputStream](input, streamParser, partitionedFileString))
Review Comment:
**Non-blocking (P2):** A recoverable partial element invokes the bad-record thunk when `_corrupt_record` is populated, and this thunk reopens and `readAllBytes()` the complete file each time. Multiple malformed elements therefore reread and rematerialize the same large document once per element, defeating much of the streaming benefit. Please memoize the file literal lazily within this `readFile` invocation and reuse it for every recovery row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4078726161
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -7034,6 +7034,27 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY =
+ buildConf("spark.sql.json.enableStreamingTopLevelArray")
+ .internal()
Review Comment:
**Non-blocking (P2):** The public JSON option table tells users that `spark.sql.json.enableStreamingTopLevelArray` is this option's session default, but `.internal()` removes the key from public configuration discovery. That gives the same key conflicting support signals. Since the option docs expose it as user-settable, please make the ConfigEntry public (or stop documenting the key as a supported default).
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala:
##########
@@ -239,6 +239,10 @@ class JSONOptions(
val useUnsafeRow: Boolean = parameters.get(USE_UNSAFE_ROW).map(_.toBoolean).getOrElse(
SQLConf.get.getConf(SQLConf.JSON_USE_UNSAFE_ROW))
+ val streamMultilineTopLevelArray: Boolean =
+ parameters.get(ENABLE_STREAMING_TOP_LEVEL_ARRAY).map(_.toBoolean).getOrElse(
Review Comment:
**Nit (P3):** Please add the inverse precedence case: session config true with per-read option false, and assert eager document-scoped malformed handling. The current test only covers option true over session false, so an OR-based implementation or one that ignores explicit false would still pass.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -7034,6 +7034,27 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY =
+ buildConf("spark.sql.json.enableStreamingTopLevelArray")
+ .internal()
+ .doc("When true, multiline JSON reads stream the elements of a top-level array one at a " +
Review Comment:
**Nit (P3):** `DataFrameReader.json(Dataset[String])` builds the same `JSONOptions`, so this setting is accepted there, but that path still calls `rawParser.parse` and materializes the array eagerly. Please scope this help to multiline file and archive reads, matching the option table, unless the Dataset path is also wired to `parseIterator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "tdcmeehan (via GitHub)" <gi...@apache.org>.
tdcmeehan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4066297643
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +745,86 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = createParser(factory, record)
+ def fail(error: Throwable): Nothing = {
+ try jsonParser.close() catch {
+ case NonFatal(closeError) => error.addSuppressed(closeError)
+ }
+ throw badRecord(error, () => recordLiteral(record))
+ }
+ def handleFailure[T](operation: => T): T = {
+ try operation catch {
+ case e: SparkUpgradeException => fail(e)
+ case e: CharConversionException if options.encoding.isEmpty => fail(e)
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
Review Comment:
Done in 488871bc3b8, generalized in dd76c16adef. A recoverable element failure no longer replaces the source iterator: `handleFailure` marks the `BadRecordException` recoverable and `FailureSafeParser` splices the recovery row ahead of the surviving source -- `delegate = if (e.recoverable) recovery ++ source else recovery` (`FailureSafeParser.scala:83`) -- so later valid elements still come through. Terminal structural failures still go through `fail()`, which closes the parser first.
Resumability is now decided by where the parser was left rather than by the exception alone: `resumableAfter` (`JacksonParser.scala:797-802`) requires `PartialResultException` for a container start, since `convertObject` raises it only after consuming through `END_OBJECT`. The `partial object` row of the parse-mode grid (`JsonSuite.scala:1309-1320`) pins it on `[{"a":1},{"a":"bad"},{"a":2}]` -- `(1,null)`, `(null,D)`, `(2,null)` under PERMISSIVE and `(1,null)`, `(2,null)` under DROPMALFORMED -- the shape you cited plus the preceding valid element.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:
##########
@@ -414,8 +414,15 @@ object MultiLineJsonDataSource extends JsonDataSource {
schema,
parser.options.columnNameOfCorruptRecord)
- safeParser.parse(
- CodecStreams.createInputStreamWithCloseResource(conf, file.toPath))
+ val input = CodecStreams.createInputStreamWithCloseResource(conf, file.toPath)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit](_ => input.close()))
Review Comment:
Done in 488871bc3b8 -- the extra `addTaskCompletionListener[Unit](_ => input.close())` is gone. At head the only completion listener in this file closes a `linesReader` (`JsonDataSource.scala:214`), and the four `CodecStreams.createInputStreamWithCloseResource` call sites (316, 381, 401, 418) add none, so the helper owns those streams alone.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +745,86 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = createParser(factory, record)
Review Comment:
Done in 488871bc3b8. `createParser` is now inside `parseIterator`'s own try (`JacksonParser.scala:768-778`), which catches `CharConversionException` when `options.encoding` is empty along with the `RuntimeException` / `JsonProcessingException` / `MalformedInputException` / `Partial*` family and rethrows `badRecord(...)`, so an acquisition failure reaches `FailureSafeParser` as a `BadRecordException` and honours the parse mode. `SparkUpgradeException` is still rethrown as-is. Covered by `multiline JSON parser creation failures honor parse mode` (`JsonSuite.scala:1420`), which drives the `00 00 ff fe` encoding case through PERMISSIVE, DROPMALFORMED and FAILFAST.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -1066,6 +1069,97 @@ abstract class JsonSuite
}
}
+ gridTest("SPARK-3308 Read multiline top level JSON arrays")(
+ Seq(false, true)) { enabled =>
+ withSQLConf(SQLConf.JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY.key -> enabled.toString) {
+ withTempPath { file =>
+ Files.write(file.toPath, """[{"a":1},{"a":2}]""".getBytes(StandardCharsets.UTF_8))
+ checkAnswer(
+ spark.read.option("multiLine", true).schema("a int").json(file.getCanonicalPath),
+ Seq(Row(1), Row(2)))
+ }
+ }
+ }
+
+ test("multiline top level JSON arrays are parsed lazily") {
+ val schema = StructType(Seq(StructField("a", IntegerType)))
+ val options = new JSONOptions(Map("multiLine" -> "true"), SQLConf.get.sessionLocalTimeZone)
+ val parser = new JacksonParser(schema, options, allowArrayAsStructs = true)
+ val input = new ByteArrayInputStream(
+ s"""[{"a":1},{"a":2,"payload":"${"x" * 200000}"}]""".getBytes(StandardCharsets.UTF_8))
+ val rows = parser.parseIterator[InputStream](
+ input,
+ CreateJacksonParser.inputStream(_: JsonFactory, _: InputStream),
+ stream => UTF8String.fromBytes(stream.readAllBytes()))
+
+ assert(rows.next().getInt(0) === 1)
+ assert(input.available() > 0)
+ }
+
+ test("non-array JSON is parsed eagerly by parseIterator") {
+ val schema = StructType(Seq(StructField("a", IntegerType)))
+ val options = new JSONOptions(Map("multiLine" -> "true"), SQLConf.get.sessionLocalTimeZone)
+ val parser = new JacksonParser(schema, options, allowArrayAsStructs = true)
+ var closed = false
+ val input = new ByteArrayInputStream("""{"a":1}""".getBytes(StandardCharsets.UTF_8)) {
+ override def close(): Unit = {
+ closed = true
+ super.close()
+ }
+ }
+
+ val rows = parser.parseIterator[InputStream](
+ input,
+ CreateJacksonParser.inputStream(_: JsonFactory, _: InputStream),
+ stream => UTF8String.fromBytes(stream.readAllBytes()))
+
+ assert(closed)
+ assert(rows.next().getInt(0) === 1)
+ assert(!rows.hasNext)
+ }
+
+ gridTest("multiline top level JSON array keeps rows emitted before malformed input")(
Review Comment:
Done -- all three gaps.
`multiline top level JSON array parse modes with a malformed element` (`JsonSuite.scala:1309-1320`) runs `[{"a":1},<malformed>,{"a":2}]` over three element shapes -- a partial object, a scalar, a nested array -- times PERMISSIVE/DROPMALFORMED/FAILFAST, times the setting off and on, so the enabled path's element granularity is asserted against the disabled path's document granularity in the same case. `multiline JSON parser creation failures honor parse mode` (line 1420) covers parser construction across the three modes. Both from 488871bc3b8, extended in dd76c16adef and 75561b8daae.
Archive coverage with the setting enabled came in 5999f6b5974: `JSON: streaming archive arrays resume after a partial element` and `JSON: streaming archive arrays handle terminal malformed input` (`JSONArchiveReadBase.scala:180`, `:205`), both under `withSQLConf(JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY -> "true")`.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +745,100 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = try {
+ createParser(factory, record)
+ } catch {
+ case e: SparkUpgradeException => throw e
+ case e: CharConversionException if options.encoding.isEmpty =>
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ def fail(error: Throwable): Nothing = {
+ try jsonParser.close() catch {
+ case NonFatal(closeError) => error.addSuppressed(closeError)
+ }
+ throw badRecord(error, () => recordLiteral(record))
+ }
+ def handleFailure[T](operation: => T): T = {
+ try operation catch {
+ case e: SparkUpgradeException => fail(e)
+ case e: CharConversionException if options.encoding.isEmpty => fail(e)
+ case e: PartialResultException if options.parseMode != FailFastMode =>
Review Comment:
Done in 5999f6b5974, tightened in dd76c16adef. The non-array root is converted with `handleFailure(elementStart = None)` (`JacksonParser.scala:865`) and `resumableAfter` returns `false` for `None` unconditionally (`:801`), so that state always routes to `fail()`, which closes the parser before throwing. Recoverability is reachable only from inside the top-level-array iterator, where an element start token exists, so the retention is excluded by construction rather than by case analysis.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONArchiveReadBase.scala:
##########
@@ -166,6 +166,16 @@ trait JSONArchiveReadBase extends ArchiveReadSuiteBase {
extraOptions = Map("multiLine" -> "true"))
}
+ test("JSON: streaming multi-line top-level arrays match a directory read") {
Review Comment:
Done in 5999f6b5974. Two enabled archive gridTests over PERMISSIVE/DROPMALFORMED/FAILFAST: `JSON: streaming archive arrays resume after a partial element` (`JSONArchiveReadBase.scala:180`) and `JSON: streaming archive arrays handle terminal malformed input` (`:205`). Both go through `readStream`, so its own `FailureSafeParser` wiring and pre-buffered literal are what get exercised, not `readFile`'s. The PERMISSIVE case asserts the complete entry text in `_corrupt_record`: `checkAnswer(df, Seq(Row(1, "Alice", null), Row(null, null, document)))`, where `document` is the whole entry.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:
##########
@@ -414,8 +414,14 @@ object MultiLineJsonDataSource extends JsonDataSource {
schema,
parser.options.columnNameOfCorruptRecord)
- safeParser.parse(
- CodecStreams.createInputStreamWithCloseResource(conf, file.toPath))
+ val input = CodecStreams.createInputStreamWithCloseResource(conf, file.toPath)
+ if (parser.options.streamMultilineTopLevelArray) {
+ safeParser.parseIterator(
+ input,
+ input => parser.parseIterator[InputStream](input, streamParser, partitionedFileString))
Review Comment:
Done in 5999f6b5974, with the archive side in 5072e171a5c. `fileLiteral` is a `lazy val` inside `readFile` (`JsonDataSource.scala:398-404`), so the open and `readAllBytes()` happen at most once per invocation however many elements recover; `partitionedFileString` just returns it. The archive path's `documentLiteral` (`:435`) is a `lazy val` over bytes already in hand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4107507137
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -7034,6 +7034,26 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY =
+ buildConf("spark.sql.json.enableStreamingTopLevelArray")
+ .doc("When true, multiline JSON file reads stream the elements of a top-level array one at " +
+ "a time instead of materializing the entire array before returning rows. This applies " +
+ "only to reads into a struct schema that take top-level arrays as structs, and has no " +
+ "effect on reads using the `singleVariantColumn` or `explodeEmbeddedArray` option. " +
+ "Streaming also makes an array element, rather than the whole document, the record " +
+ "that a parse mode applies to, since rows already emitted cannot be withdrawn: " +
+ "PERMISSIVE fills the corrupt record column for the malformed element only, leaving " +
+ "it null on the valid rows of the same document, and DROPMALFORMED drops that " +
+ "element rather than the whole document. An element whose failure leaves the parser " +
+ "at an unknown position, such as a nested value of the wrong shape, still ends the " +
+ "document, as does a failure outside any element, such as a syntax error between two " +
+ "elements or a missing closing bracket. It can be overwritten by the JSON option " +
+ "`enableStreamingTopLevelArray`.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .booleanConf
+ .createWithDefault(false)
Review Comment:
Thanks, that rationale makes sense. The explicit false/true grids already pin both semantic branches, and the declaration makes the compatibility default visible, so I’m fine leaving out the separate unset case.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4105396987","thread_id":"inline:4105396987","verdict_sha256":"fbfd6156813f99061dd3aa8e7b04d155ef214ed6f88f7eb390b7ddbab21cebbd"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "tdcmeehan (via GitHub)" <gi...@apache.org>.
tdcmeehan commented on PR #58704:
URL: https://github.com/apache/spark/pull/58704#issuecomment-5690756357
Thanks @cloud-fan, addressed your comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "tdcmeehan (via GitHub)" <gi...@apache.org>.
tdcmeehan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4061266335
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -7034,6 +7034,16 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY =
+ buildConf("spark.sql.json.enableStreamingTopLevelArray")
+ .internal()
+ .doc("When true, multiline JSON reads stream the elements of a top-level array one at a " +
Review Comment:
Done. The doc now states the scope: streaming applies only to reads into a struct schema that take top-level arrays as structs, and reads using `singleVariantColumn` or `explodeEmbeddedArray` are never streamed.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +746,104 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
Review Comment:
Done. `arrayElementConverter` is a single `private lazy val` (`JacksonParser.scala:67`) that both the eager root converter and the streaming iterator use, so the graph is built once per parser. Being lazy, it is not built at all for a non-array root.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +746,104 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = try {
+ createParser(factory, record)
+ } catch {
+ case e: SparkUpgradeException => throw e
+ case e: CharConversionException if options.encoding.isEmpty =>
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ // Abandoning the iterator before END_ARRAY (e.g. a LIMIT) skips finish()/fail(), so close the
+ // parser at task completion; close() is idempotent with those eager closes.
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit](_ => jsonParser.close()))
Review Comment:
Done in dd76c16adef. The listener captures an `AtomicReference` cell rather than the parser: `closeParser()` does `getAndSet(null)` and closes only what it took, and all four eager close paths go through it alongside the listener. Exactly one close therefore happens, and a completed read leaves nothing reachable from the `TaskContext`. Clearing happens before the close, so a close that throws still clears — the alternative loses the parser for good if the clear is what fails.
Covered by `multiline top level JSON array stops retaining a closed parser`, which counts `close()` calls through a `JsonParserDelegate` over the empty, empty-array, valid and truncated documents and asserts `markTaskCompleted` adds none. A stream-level counter cannot observe this: jackson's own `close()` is idempotent, so the regression shows up only as a second *call*.
On the archive-entry coverage you asked for: one parser per entry is this same mechanism repeated, so I have pinned the per-parser invariant instead. A multi-entry test could only fail for the same reason this one does.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +746,107 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = try {
+ createParser(factory, record)
+ } catch {
+ case e: SparkUpgradeException => throw e
+ case e: CharConversionException if options.encoding.isEmpty =>
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ // Abandoning the iterator before END_ARRAY (e.g. a LIMIT) skips finish()/fail(), so close the
+ // parser at task completion; close() is idempotent with those eager closes.
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit](_ => jsonParser.close()))
+ def fail(error: Throwable): Nothing = {
+ try jsonParser.close() catch {
Review Comment:
Done in dd76c16adef, through the same cell. The truncated-array case of `multiline top level JSON array stops retaining a closed parser` is exactly this path: `prepare()` raises from `nextToken()`, `fail()` closes, and the test asserts one close *before* `markTaskCompleted` and none added by it. If `fail()` stopped closing, that first assertion would see 0 — the listener would close it later, which is the retention this is meant to catch.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -1066,6 +1070,282 @@ abstract class JsonSuite
}
}
+ gridTest("SPARK-3308 Read multiline top level JSON arrays")(
+ Seq(false, true)) { enabled =>
+ withSQLConf(SQLConf.JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY.key -> enabled.toString) {
+ withTempPath { file =>
+ Files.write(file.toPath, """[{"a":1},{"a":2}]""".getBytes(StandardCharsets.UTF_8))
Review Comment:
Done in dd76c16adef and 75561b8daae. The lifecycle grid runs over `""`, `"[]"`, a valid two-element array and the truncated array, asserting both the rows each yields (0, 0, 2, 1) and one `close()` in each. That covers the empty document's close before the iterator is constructed, the empty array's close from the first `prepare()`, and closure after normal exhaustion, with zero rows asserted for both empty inputs.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -717,43 +746,107 @@ class JacksonParser(
}
} catch {
case e: SparkUpgradeException => throw e
- case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException) =>
- // JSON parser currently doesn't support partial results for corrupted records.
- // For such records, all fields other than the field configured by
- // `columnNameOfCorruptRecord` are set to `null`.
- throw BadRecordException(() => recordLiteral(record), () => Array.empty, e)
case e: CharConversionException if options.encoding.isEmpty =>
- val msg =
- """JSON parser cannot handle a character in its input.
- |Specifying encoding as an input option explicitly might help to resolve the issue.
- |""".stripMargin + e.getMessage
- val wrappedCharException = new CharConversionException(msg)
- wrappedCharException.initCause(e)
- throw BadRecordException(() => recordLiteral(record), () => Array.empty,
- wrappedCharException)
- case PartialResultException(row, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(row),
- convertCauseForPartialResult(cause))
- case PartialResultArrayException(rows, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => rows,
- cause)
- // These exceptions should never be thrown outside of JacksonParser.
- // They are used for the control flow in the parser. We add them here for completeness
- // since they also indicate a bad record.
- case PartialArrayDataResultException(arrayData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(arrayData)),
- convertCauseForPartialResult(cause))
- case PartialMapDataResultException(mapData, cause) =>
- throw BadRecordException(
- record = () => recordLiteral(record),
- partialResults = () => Array(InternalRow(mapData)),
- convertCauseForPartialResult(cause))
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ }
+
+ private[sql] def parseIterator[T](
+ record: T,
+ createParser: (JsonFactory, T) => JsonParser,
+ recordLiteral: T => UTF8String): Iterator[InternalRow] = {
+ val streamArray = allowArrayAsStructs && schema.isInstanceOf[StructType] &&
+ options.singleVariantColumn.isEmpty && options.explodeEmbeddedArray.isEmpty
+ val elementConverter = if (streamArray) makeConverter(schema) else null
+ val jsonParser = try {
+ createParser(factory, record)
+ } catch {
+ case e: SparkUpgradeException => throw e
+ case e: CharConversionException if options.encoding.isEmpty =>
+ throw badRecord(e, () => recordLiteral(record))
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
+ _: PartialResultException | _: PartialResultArrayException |
+ _: PartialArrayDataResultException | _: PartialMapDataResultException) =>
+ throw badRecord(e, () => recordLiteral(record))
+ }
+ // Abandoning the iterator before END_ARRAY (e.g. a LIMIT) skips finish()/fail(), so close the
+ // parser at task completion; close() is idempotent with those eager closes.
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit](_ => jsonParser.close()))
+ def fail(error: Throwable): Nothing = {
+ try jsonParser.close() catch {
+ case NonFatal(closeError) => error.addSuppressed(closeError)
+ }
+ throw badRecord(error, () => recordLiteral(record))
+ }
+ def handleFailure[T](recoverPartialResult: Boolean)(operation: => T): T = {
+ try operation catch {
+ case e: SparkUpgradeException => fail(e)
+ case e: CharConversionException if options.encoding.isEmpty => fail(e)
+ case e: PartialResultException
+ if recoverPartialResult && options.parseMode != FailFastMode =>
+ throw badRecord(e, () => recordLiteral(record)).copy(recoverable = true)
+ case e: PartialResultException =>
+ fail(e)
+ case e @ (_: RuntimeException | _: JsonProcessingException | _: MalformedInputException |
Review Comment:
Done in dd76c16adef. Resumability is now decided by where the parser is left rather than by the exception alone. `resumableAfter` takes the element's start token: a scalar start makes any `RuntimeException` recoverable, since the parser is still on that scalar's own token; a container start still requires `PartialResultException`, which is the only container failure that provably drained to `END_OBJECT`. `FailFastMode` is excluded as before.
`[{"a":1},42,{"a":2}]` now yields `(1,null)`, `(null,D)`, `(2,null)` under PERMISSIVE and `(1,null)`, `(2,null)` under DROPMALFORMED.
The parse-mode grid is parameterized over the malformed element's shape — a bad object field, a scalar, a nested array — against one document `[{"a":1},<malformed>,{"a":2}]`, with the nested array as the negative control that has to stay terminal. A classification broad enough to resume from inside a container fails there.
Two notes on the framing. First, one consequence reaches beyond the input you cited: an all-scalar array such as `[1,2,3]` is malformed in every element against a struct schema, so it now reports one corrupt record per element rather than one per document. The existing SPARK-18352 multiline expectations are updated and made configuration-dependent for that. Object elements already behaved this way, through the partial-result path the eager reader has always had.
Second, on "silently lose the final valid row": for that input the disabled path loses more, not less — it yields a single `(null,D)` and drops *both* valid rows, because the scalar failure escapes `convertArray` before any element is accumulated. So the enabled path was already ahead of the one it replaces. The real defect was that element granularity was promised for every malformed element and delivered only for objects, which is what this fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "tdcmeehan (via GitHub)" <gi...@apache.org>.
tdcmeehan commented on PR #58704:
URL: https://github.com/apache/spark/pull/58704#issuecomment-5793157322
Thank you @cloud-fan, addressed your feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #58704: [SPARK-59417][SQL] Stream multiline top-level JSON arrays
URL: https://github.com/apache/spark/pull/58704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "tdcmeehan (via GitHub)" <gi...@apache.org>.
tdcmeehan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4106843648
##########
docs/sql-data-sources-json.md:
##########
@@ -237,6 +237,12 @@ Data source options of JSON can be set via:
<td>Parse one record, which may span multiple lines, per file. JSON built-in functions ignore this option.</td>
<td>read</td>
</tr>
+ <tr>
+ <td><code>enableStreamingTopLevelArray</code></td>
+ <td>(value of <code>spark.sql.json.enableStreamingTopLevelArray</code> configuration)</td>
+ <td>When <code>multiLine</code> is enabled and a file holds a top-level JSON array, read the array's elements one at a time instead of materializing the whole array before returning rows. It applies to reads into a struct schema, and has no effect on reads using <code>singleVariantColumn</code> or <code>explodeEmbeddedArray</code>. While streaming, <code>mode</code> applies to an individual element rather than the whole document: <code>PERMISSIVE</code> fills <code>columnNameOfCorruptRecord</code> for the malformed element alone, leaving it null on the valid rows of the same document, and <code>DROPMALFORMED</code> drops that element rather than the document. An element whose failure leaves the parser at an unknown position, such as a nested value of the wrong shape, still ends the document, as does a failure outside any element, such as a syntax error between two elements or a missing closing bracket.</td>
Review Comment:
Done in `763bf149a25` — it now reads "PERMISSIVE fills the field configured by `columnNameOfCorruptRecord`", with the rest of the element-level explanation unchanged.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:
##########
@@ -414,8 +415,14 @@ object MultiLineJsonDataSource extends JsonDataSource {
schema,
parser.options.columnNameOfCorruptRecord)
- safeParser.parse(
- CodecStreams.createInputStreamWithCloseResource(conf, file.toPath))
+ val input = CodecStreams.createInputStreamWithCloseResource(conf, file.toPath)
+ if (parser.options.streamMultilineTopLevelArray) {
+ safeParser.parseIterator(
+ input,
+ input => parser.parseIterator[InputStream](input, streamParser, partitionedFileString))
Review Comment:
Good catch. `763bf149a25` adds explicit non-UTF-8 coverage on both routes. The SPARK-23723 mismatched-encoding test now runs with the conf disabled and enabled (the directory route), and the archive suites stream an ISO-8859-1 top-level array and check it against a directory read of the same bytes (the archive route) — ISO-8859-1 because Jackson auto-detects UTF-16/32, so only a charset it can't guess shows the encoding is honored. I verified it the way you described: replacing the directory `streamParser` with the default parser turns the SPARK-23723 enabled arms red, and replacing the archive `streamParser` turns the archive comparison red, so neither argument can be dropped silently now.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -7034,6 +7034,26 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY =
+ buildConf("spark.sql.json.enableStreamingTopLevelArray")
+ .doc("When true, multiline JSON file reads stream the elements of a top-level array one at " +
+ "a time instead of materializing the entire array before returning rows. This applies " +
+ "only to reads into a struct schema that take top-level arrays as structs, and has no " +
+ "effect on reads using the `singleVariantColumn` or `explodeEmbeddedArray` option. " +
+ "Streaming also makes an array element, rather than the whole document, the record " +
+ "that a parse mode applies to, since rows already emitted cannot be withdrawn: " +
+ "PERMISSIVE fills the corrupt record column for the malformed element only, leaving " +
+ "it null on the valid rows of the same document, and DROPMALFORMED drops that " +
+ "element rather than the whole document. An element whose failure leaves the parser " +
+ "at an unknown position, such as a nested value of the wrong shape, still ends the " +
+ "document, as does a failure outside any element, such as a syntax error between two " +
+ "elements or a missing closing bracket. It can be overwritten by the JSON option " +
+ "`enableStreamingTopLevelArray`.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .booleanConf
+ .createWithDefault(false)
Review Comment:
Thanks — and I follow the concern: the grids set the key explicitly on both sides, so on their own they wouldn't catch a change to the default. I'd lean toward leaving this one out, though. Both sides of the recovery boundary are already pinned by the disabled/enabled grids — a malformed array gives the whole-document result with it off and per-element recovery with it on — so the contract itself is covered. A case with neither set would assert the current default value specifically, and flipping that default later would be a deliberate change a reviewer would see, best pinned alongside that change rather than guarded here. Since it's non-blocking I've left it as-is for now, but I'm glad to add the explicit default-path case if you'd still prefer it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58704:
URL: https://github.com/apache/spark/pull/58704#issuecomment-5732048131
Thanks, the earlier comments are addressed. I found one related cleanup issue in the revised path: when the enabled iterator parser falls back to a non-array document and root conversion produces a partial result, it returns recovery output without closing the non-resumable parser.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:5690756357","thread_id":"conversation:5690756357","verdict_sha256":"a0fba4655453eaab81b014b5dd1dc543243518ab81244e894668ce35e355295f"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "tdcmeehan (via GitHub)" <gi...@apache.org>.
tdcmeehan commented on PR #58704:
URL: https://github.com/apache/spark/pull/58704#issuecomment-5628334240
@HyukjinKwon done, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4105396991
##########
docs/sql-data-sources-json.md:
##########
@@ -237,6 +237,12 @@ Data source options of JSON can be set via:
<td>Parse one record, which may span multiple lines, per file. JSON built-in functions ignore this option.</td>
<td>read</td>
</tr>
+ <tr>
+ <td><code>enableStreamingTopLevelArray</code></td>
+ <td>(value of <code>spark.sql.json.enableStreamingTopLevelArray</code> configuration)</td>
+ <td>When <code>multiLine</code> is enabled and a file holds a top-level JSON array, read the array's elements one at a time instead of materializing the whole array before returning rows. It applies to reads into a struct schema, and has no effect on reads using <code>singleVariantColumn</code> or <code>explodeEmbeddedArray</code>. While streaming, <code>mode</code> applies to an individual element rather than the whole document: <code>PERMISSIVE</code> fills <code>columnNameOfCorruptRecord</code> for the malformed element alone, leaving it null on the valid rows of the same document, and <code>DROPMALFORMED</code> drops that element rather than the document. An element whose failure leaves the parser at an unknown position, such as a nested value of the wrong shape, still ends the document, as does a failure outside any element, such as a syntax error between two elements or a missing closing bracket.</td>
Review Comment:
**Nit (P3):** `columnNameOfCorruptRecord` is the option that names the destination field, not the field itself. Please say that PERMISSIVE fills the field configured by `columnNameOfCorruptRecord`; the rest of the element-level explanation can stay unchanged.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:
##########
@@ -414,8 +415,14 @@ object MultiLineJsonDataSource extends JsonDataSource {
schema,
parser.options.columnNameOfCorruptRecord)
- safeParser.parse(
- CodecStreams.createInputStreamWithCloseResource(conf, file.toPath))
+ val input = CodecStreams.createInputStreamWithCloseResource(conf, file.toPath)
+ if (parser.options.streamMultilineTopLevelArray) {
+ safeParser.parseIterator(
+ input,
+ input => parser.parseIterator[InputStream](input, streamParser, partitionedFileString))
Review Comment:
**Non-blocking (P2):** This new iterator route has no enabled case with an explicit non-UTF-8 encoding, and the archive route has the same independent wiring. Replacing either `streamParser` argument with the default parser would leave the UTF-8 streaming tests and eager encoding tests green. Please cover an explicitly encoded non-UTF-8 top-level array through both directory and archive reads.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -7034,6 +7034,26 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY =
+ buildConf("spark.sql.json.enableStreamingTopLevelArray")
+ .doc("When true, multiline JSON file reads stream the elements of a top-level array one at " +
+ "a time instead of materializing the entire array before returning rows. This applies " +
+ "only to reads into a struct schema that take top-level arrays as structs, and has no " +
+ "effect on reads using the `singleVariantColumn` or `explodeEmbeddedArray` option. " +
+ "Streaming also makes an array element, rather than the whole document, the record " +
+ "that a parse mode applies to, since rows already emitted cannot be withdrawn: " +
+ "PERMISSIVE fills the corrupt record column for the malformed element only, leaving " +
+ "it null on the valid rows of the same document, and DROPMALFORMED drops that " +
+ "element rather than the whole document. An element whose failure leaves the parser " +
+ "at an unknown position, such as a nested value of the wrong shape, still ends the " +
+ "document, as does a failure outside any element, such as a syntax error between two " +
+ "elements or a missing closing bracket. It can be overwritten by the JSON option " +
+ "`enableStreamingTopLevelArray`.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .booleanConf
+ .createWithDefault(false)
Review Comment:
**Non-blocking (P2):** The compatibility boundary here is the behavior when callers leave both settings unset, but every malformed-array test currently forces this key to true or false. A future default flip would change ordinary reads from document-level to element-level recovery without failing those grids. Please add a malformed multiline top-level-array case with neither the session key nor per-read option set and assert the eager whole-document result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59417][SQL] Stream multiline top-level JSON arrays [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58704:
URL: https://github.com/apache/spark/pull/58704#issuecomment-5842546192
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/f4327af76197334dbf893a6f7ae34e2d4a096c5f
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org