You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Stuart White (Jira)" <ji...@apache.org> on 2020/10/29 21:52:00 UTC
[jira] [Updated] (SPARK-33291) Inconsistent NULL conversions to
strings redux
[ https://issues.apache.org/jira/browse/SPARK-33291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stuart White updated SPARK-33291:
---------------------------------
Description:
The changes in [SPARK-32501 Inconsistent NULL conversions to strings|https://issues.apache.org/jira/browse/SPARK-32501] introduced some behavior that I'd like to clean up a bit.
Here's sample code to illustrate the behavior I'd like to clean up:
{noformat}
val rows = Seq[String](null)
.toDF("value")
.withColumn("struct1", struct('value as "value1"))
.withColumn("struct2", struct('value as "value1", 'value as "value2"))
.withColumn("array1", array('value))
.withColumn("array2", array('value, 'value))
// Show the DataFrame using the "first" codepath.
rows.show(truncate=false)
+-----+-------+-------------+------+--------+
|value|struct1|struct2 |array1|array2 |
+-----+-------+-------------+------+--------+
|null |{ null}|{ null, null}|[] |[, null]|
+-----+-------+-------------+------+--------+
// Write the DataFrame to disk, then read it back and show it to trigger the "codegen" code path:
rows.write.parquet("rows")
spark.read.parquet("rows").show(truncate=false)
+-----+-------+-------------+-------+-------------+
|value|struct1|struct2 |array1 |array2 |
+-----+-------+-------------+-------+-------------+
|null |{ null}|{ null, null}|[ null]|[ null, null]|
+-----+-------+-------------+-------+-------------+
{noformat}
Notice:
1. If the first element of a struct is null, it is printed with a leading space (e.g. "\{ null\}"). I think it's preferable to print it without the leading space (e.g. "\{null\}"). This is consistent with how non-null values are printed inside a struct.
2. If the first element of an array is null, it is not printed at all in the first code path, and the "codegen" code path prints it with a leading space. I think both code paths should be consistent and print it without a leading space (e.g. "[null]").
The desired result of this ticket is to product the following output via both code paths:
{noformat}
+-----+-------+------------+------+------------+
|value|struct1|struct2 |array1|array2 |
+-----+-------+------------+------+------------+
|null |{null} |{null, null}|[null]|[null, null]|
+-----+-------+------------+------+------------+
{noformat}
was:
The changes in [SPARK-32501 Inconsistent NULL conversions to strings|https://issues.apache.org/jira/browse/SPARK-32501] introduced some behavior that I'd like to clean up a bit.
Here's sample code to illustrate the behavior I'd like to clean up:
{noformat}
val rows = Seq[String](null)
.toDF("value")
.withColumn("struct1", struct('value as "value1"))
.withColumn("struct2", struct('value as "value1", 'value as "value2"))
.withColumn("array1", array('value))
.withColumn("array2", array('value, 'value))
.withColumn("map1", map(lit("value1"), 'value))
.withColumn("map2", map(lit("value1"), 'value, lit("value2"), 'value))
// Show the DataFrame using the "first" codepath.
rows.show(truncate=false)
+-----+-------+-------------+------+--------+----------------+--------------------------------+
|value|struct1|struct2 |array1|array2 |map1 |map2 |
+-----+-------+-------------+------+--------+----------------+--------------------------------+
|null |{ null}|{ null, null}|[] |[, null]|{value1 -> null}|{value1 -> null, value2 -> null}|
+-----+-------+-------------+------+--------+----------------+--------------------------------+
// Write the DataFrame to disk, then read it back and show it to trigger the "codegen" code path:
rows.write.parquet("rows")
spark.read.parquet("rows").show(truncate=false)
+-----+-------+-------------+-------+-------------+----------------+--------------------------------+
|value|struct1|struct2 |array1 |array2 |map1 |map2 |
+-----+-------+-------------+-------+-------------+----------------+--------------------------------+
|null |{ null}|{ null, null}|[ null]|[ null, null]|{value1 -> null}|{value1 -> null, value2 -> null}|
+-----+-------+-------------+-------+-------------+----------------+--------------------------------+
{noformat}
Notice:
1. If the first element of a struct is null, it is printed with a leading space (e.g. "\{ null\}"). I think it's preferable to print it without the leading space (e.g. "\{null\}"). This is consistent with how non-null values are printed inside a struct.
2. If the first element of an array is null, it is not printed at all in the first code path, and the "codegen" code path prints it with a leading space. I think both code paths should be consistent and print it without a leading space (e.g. "[null]").
The desired result of this ticket is to product the following output via both code paths:
{noformat}
+-----+-------+------------+------+------------+----------------+--------------------------------+
|value|struct1|struct2 |array1|array2 |map1 |map2 |
+-----+-------+------------+------+------------+----------------+--------------------------------+
|null |{null} |{null, null}|[null]|[null, null]|{value1 -> null}|{value1 -> null, value2 -> null}|
+-----+-------+------------+------+------------+----------------+--------------------------------+
{noformat}
> Inconsistent NULL conversions to strings redux
> ----------------------------------------------
>
> Key: SPARK-33291
> URL: https://issues.apache.org/jira/browse/SPARK-33291
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Stuart White
> Priority: Minor
>
> The changes in [SPARK-32501 Inconsistent NULL conversions to strings|https://issues.apache.org/jira/browse/SPARK-32501] introduced some behavior that I'd like to clean up a bit.
> Here's sample code to illustrate the behavior I'd like to clean up:
> {noformat}
> val rows = Seq[String](null)
> .toDF("value")
> .withColumn("struct1", struct('value as "value1"))
> .withColumn("struct2", struct('value as "value1", 'value as "value2"))
> .withColumn("array1", array('value))
> .withColumn("array2", array('value, 'value))
> // Show the DataFrame using the "first" codepath.
> rows.show(truncate=false)
> +-----+-------+-------------+------+--------+
> |value|struct1|struct2 |array1|array2 |
> +-----+-------+-------------+------+--------+
> |null |{ null}|{ null, null}|[] |[, null]|
> +-----+-------+-------------+------+--------+
> // Write the DataFrame to disk, then read it back and show it to trigger the "codegen" code path:
> rows.write.parquet("rows")
> spark.read.parquet("rows").show(truncate=false)
> +-----+-------+-------------+-------+-------------+
> |value|struct1|struct2 |array1 |array2 |
> +-----+-------+-------------+-------+-------------+
> |null |{ null}|{ null, null}|[ null]|[ null, null]|
> +-----+-------+-------------+-------+-------------+
> {noformat}
> Notice:
> 1. If the first element of a struct is null, it is printed with a leading space (e.g. "\{ null\}"). I think it's preferable to print it without the leading space (e.g. "\{null\}"). This is consistent with how non-null values are printed inside a struct.
> 2. If the first element of an array is null, it is not printed at all in the first code path, and the "codegen" code path prints it with a leading space. I think both code paths should be consistent and print it without a leading space (e.g. "[null]").
> The desired result of this ticket is to product the following output via both code paths:
> {noformat}
> +-----+-------+------------+------+------------+
> |value|struct1|struct2 |array1|array2 |
> +-----+-------+------------+------+------------+
> |null |{null} |{null, null}|[null]|[null, null]|
> +-----+-------+------------+------+------------+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org