You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2022/04/26 01:58:12 UTC
[spark] branch branch-3.3 updated: [SPARK-39001][SQL][DOCS][FOLLOW-UP] Revert the doc changes for dropFieldIfAllNull, prefersDecimal and primitivesAsString (schema_of_json)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new ff9b163c760 [SPARK-39001][SQL][DOCS][FOLLOW-UP] Revert the doc changes for dropFieldIfAllNull, prefersDecimal and primitivesAsString (schema_of_json)
ff9b163c760 is described below
commit ff9b163c760a109972a3ca6334fa2254e8096be1
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Apr 26 10:57:11 2022 +0900
[SPARK-39001][SQL][DOCS][FOLLOW-UP] Revert the doc changes for dropFieldIfAllNull, prefersDecimal and primitivesAsString (schema_of_json)
This PR is a followup of https://github.com/apache/spark/pull/36339. Actually `schema_of_json` expression supports `dropFieldIfAllNull`, `prefersDecimal` an `primitivesAsString`:
```scala
scala> spark.range(1).selectExpr("""schema_of_json("{'a': null}", map('dropFieldIfAllNull', 'true'))""").show()
+---------------------------+
|schema_of_json({'a': null})|
+---------------------------+
| STRUCT<>|
+---------------------------+
scala> spark.range(1).selectExpr("""schema_of_json("{'b': 1.0}", map('prefersDecimal', 'true'))""").show()
+--------------------------+
|schema_of_json({'b': 1.0})|
+--------------------------+
| STRUCT<b: DECIMAL...|
+--------------------------+
scala> spark.range(1).selectExpr("""schema_of_json("{'b': 1.0}", map('primitivesAsString', 'true'))""").show()
+--------------------------+
|schema_of_json({'b': 1.0})|
+--------------------------+
| STRUCT<b: STRING>|
+--------------------------+
```
For correct documentation.
To end users, no because it's a partial revert of the docs unreleased yet.
Partial logical revert so I did not add a test also since this is just a doc change.
Closes #36346 from HyukjinKwon/SPARK-39001-followup.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 5056c6cc333982d39546f2acf9a889d102cc4ab3)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
docs/sql-data-sources-json.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/docs/sql-data-sources-json.md b/docs/sql-data-sources-json.md
index 8128e779ace..ed4f7866bd0 100644
--- a/docs/sql-data-sources-json.md
+++ b/docs/sql-data-sources-json.md
@@ -127,13 +127,13 @@ Data source options of JSON can be set via:
<tr>
<td><code>primitivesAsString</code></td>
<td><code>false</code></td>
- <td>Infers all primitive values as a string type. JSON built-in functions ignore this option.</td>
+ <td>Infers all primitive values as a string type.</td>
<td>read</td>
</tr>
<tr>
<td><code>prefersDecimal</code></td>
<td><code>false</code></td>
- <td>Infers all floating-point values as a decimal type. If the values do not fit in decimal, then it infers them as doubles. JSON built-in functions ignore this option.</td>
+ <td>Infers all floating-point values as a decimal type. If the values do not fit in decimal, then it infers them as doubles.</td>
<td>read</td>
</tr>
<tr>
@@ -235,7 +235,7 @@ Data source options of JSON can be set via:
<tr>
<td><code>dropFieldIfAllNull</code></td>
<td><code>false</code></td>
- <td>Whether to ignore column of all null values or empty array/struct during schema inference. JSON built-in functions ignore this option.</td>
+ <td>Whether to ignore column of all null values or empty array/struct during schema inference.</td>
<td>read</td>
</tr>
<tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org