You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2024/07/31 09:04:42 UTC
(spark) branch master updated: [SPARK-48964][SQL][DOCS] Fix the discrepancy between implementation, comment and documentation of option `recursive.fields.max.depth` in ProtoBuf connector
This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5954ed19b9dd [SPARK-48964][SQL][DOCS] Fix the discrepancy between implementation, comment and documentation of option `recursive.fields.max.depth` in ProtoBuf connector
5954ed19b9dd is described below
commit 5954ed19b9dd1e7e741e37a689ae741722f7d5b6
Author: Wei Guo <gu...@gmail.com>
AuthorDate: Wed Jul 31 17:04:26 2024 +0800
[SPARK-48964][SQL][DOCS] Fix the discrepancy between implementation, comment and documentation of option `recursive.fields.max.depth` in ProtoBuf connector
### What changes were proposed in this pull request?
This PR aims to fix the discrepancy between implementation, comment and documentation of option `recursive.fields.max.depth` in ProtoBuf connector
### Why are the changes needed?
Unify code implementation and documentation description.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47458 from wayneguow/SPARK-48964.
Authored-by: Wei Guo <gu...@gmail.com>
Signed-off-by: yangjie01 <ya...@baidu.com>
---
.../utils/src/main/resources/error/error-conditions.json | 2 +-
.../apache/spark/sql/protobuf/utils/SchemaConverters.scala | 14 +++++++-------
docs/sql-data-sources-protobuf.md | 11 ++++++-----
3 files changed, 14 insertions(+), 13 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index 15b851a78d62..de127d4a7bf0 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -3640,7 +3640,7 @@
},
"RECURSIVE_PROTOBUF_SCHEMA" : {
"message" : [
- "Found recursive reference in Protobuf schema, which can not be processed by Spark by default: <fieldDescriptor>. try setting the option `recursive.fields.max.depth` 0 to 10. Going beyond 10 levels of recursion is not allowed."
+ "Found recursive reference in Protobuf schema, which can not be processed by Spark by default: <fieldDescriptor>. try setting the option `recursive.fields.max.depth` 1 to 10. Going beyond 10 levels of recursion is not allowed."
],
"sqlState" : "42K0G"
},
diff --git a/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala b/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala
index feb5aed03451..56c1f8185061 100644
--- a/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala
+++ b/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala
@@ -176,16 +176,16 @@ object SchemaConverters extends Logging {
}
case MESSAGE =>
// If the `recursive.fields.max.depth` value is not specified, it will default to -1,
- // and recursive fields are not permitted. Setting it to 0 drops all recursive fields,
- // 1 allows it to be recursed once, and 2 allows it to be recursed twice and so on.
- // A value greater than 10 is not allowed, and if a protobuf record has more depth for
- // recursive fields than the allowed value, it will be truncated and some fields may be
- // discarded.
+ // and recursive fields are not permitted. Setting it to 1 drops all recursive fields,
+ // 2 allows it to be recursed once, and 3 allows it to be recursed twice and so on.
+ // A value less than or equal to 0 or greater than 10 is not allowed, and if a protobuf
+ // record has more depth for recursive fields than the allowed value, it will be truncated
+ // and some fields may be discarded.
// SQL Schema for protob2uf `message Person { string name = 1; Person bff = 2;}`
// will vary based on the value of "recursive.fields.max.depth".
// 1: struct<name: string>
- // 2: struct<name string, bff: struct<name: string>>
- // 3: struct<name string, bff: struct<name string, bff: struct<name: string>>>
+ // 2: struct<name: string, bff: struct<name: string>>
+ // 3: struct<name: string, bff: struct<name: string, bff: struct<name: string>>>
// and so on.
// TODO(rangadi): A better way to terminate would be replace the remaining recursive struct
// with the byte array of corresponding protobuf. This way no information is lost.
diff --git a/docs/sql-data-sources-protobuf.md b/docs/sql-data-sources-protobuf.md
index 28e3e83bef7c..34cb1d4997d2 100644
--- a/docs/sql-data-sources-protobuf.md
+++ b/docs/sql-data-sources-protobuf.md
@@ -402,9 +402,9 @@ Spark supports the writing of all Spark SQL types into Protobuf. For most types,
## Handling circular references protobuf fields
One common issue that can arise when working with Protobuf data is the presence of circular references. In Protobuf, a circular reference occurs when a field refers back to itself or to another field that refers back to the original field. This can cause issues when parsing the data, as it can result in infinite loops or other unexpected behavior.
-To address this issue, the latest version of spark-protobuf introduces a new feature: the ability to check for circular references through field types. This allows users use the `recursive.fields.max.depth` option to specify the maximum number of levels of recursion to allow when parsing the schema. By default, `spark-protobuf` will not permit recursive fields by setting `recursive.fields.max.depth` to -1. However, you can set this option to 0 to 10 if needed.
+To address this issue, the latest version of spark-protobuf introduces a new feature: the ability to check for circular references through field types. This allows users use the `recursive.fields.max.depth` option to specify the maximum number of levels of recursion to allow when parsing the schema. By default, `spark-protobuf` will not permit recursive fields by setting `recursive.fields.max.depth` to -1. However, you can set this option to 1 to 10 if needed.
-Setting `recursive.fields.max.depth` to 0 drops all recursive fields, setting it to 1 allows it to be recursed once, and setting it to 2 allows it to be recursed twice. A `recursive.fields.max.depth` value greater than 10 is not allowed, as it can lead to performance issues and even stack overflows.
+Setting `recursive.fields.max.depth` to 1 drops all recursive fields, setting it to 2 allows it to be recursed once, and setting it to 3 allows it to be recursed twice. A `recursive.fields.max.depth` value greater than 10 is not allowed, as it can lead to performance issues and even stack overflows.
SQL Schema for the below protobuf message will vary based on the value of `recursive.fields.max.depth`.
@@ -425,9 +425,10 @@ message Person {
// The protobuf schema defined above, would be converted into a Spark SQL columns with the following
// structure based on `recursive.fields.max.depth` value.
-0: struct<name: string, bff: null>
-1: struct<name string, bff: <name: string, bff: null>>
-2: struct<name string, bff: <name: string, bff: struct<name: string, bff: null>>> ...
+1: struct<name: string>
+2: struct<name: string, bff: struct<name: string>>
+3: struct<name: string, bff: struct<name: string, bff: struct<name: string>>>
+...
{% endhighlight %}
<div class="d-none">
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org