You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dilip Biswal (Jira)" <ji...@apache.org> on 2020/07/15 10:24:00 UTC
[jira] [Assigned] (SPARK-31480) Improve the EXPLAIN FORMATTED's
output for DSV2's Scan Node
[ https://issues.apache.org/jira/browse/SPARK-31480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dilip Biswal reassigned SPARK-31480:
------------------------------------
Assignee: Dilip Biswal
> Improve the EXPLAIN FORMATTED's output for DSV2's Scan Node
> -----------------------------------------------------------
>
> Key: SPARK-31480
> URL: https://issues.apache.org/jira/browse/SPARK-31480
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Xiao Li
> Assignee: Dilip Biswal
> Priority: Major
>
> Below is the EXPLAIN OUTPUT when using the *DSV2*
> *Output of EXPLAIN EXTENDED*
> {code:java}
> +- BatchScan[col.dots#39L] JsonScan DataFilters: [isnotnull(col.dots#39L), (col.dots#39L = 500)], Location: InMemoryFileIndex[file:/private/var/folders/nr/j6hw4kr51wv0zynvr6srwgr00000gp/T/spark-7dad6f63-dc..., PartitionFilters: [], ReadSchema: struct<col.dots:bigint>
> {code}
> *Output of EXPLAIN FORMATTED*
> {code:java}
> (1) BatchScan
> Output [1]: [col.dots#39L]
> Arguments: [col.dots#39L], JsonScan(org.apache.spark.sql.test.TestSparkSession@45eab322,org.apache.spark.sql.execution.datasources.InMemoryFileIndex@72065f16,StructType(StructField(col.dots,LongType,true)),StructType(StructField(col.dots,LongType,true)),StructType(),org.apache.spark.sql.util.CaseInsensitiveStringMap@8822c5e0,Vector(),List(isnotnull(col.dots#39L), (col.dots#39L = 500)))
> {code}
> When using *DSV1*, the output is much cleaner than the output of DSV2, especially for EXPLAIN FORMATTED.
> *Output of EXPLAIN EXTENDED*
> {code:java}
> +- FileScan json [col.dots#37L] Batched: false, DataFilters: [isnotnull(col.dots#37L), (col.dots#37L = 500)], Format: JSON, Location: InMemoryFileIndex[file:/private/var/folders/nr/j6hw4kr51wv0zynvr6srwgr00000gp/T/spark-89021d76-59..., PartitionFilters: [], PushedFilters: [IsNotNull(`col.dots`), EqualTo(`col.dots`,500)], ReadSchema: struct<col.dots:bigint>
> {code}
> *Output of EXPLAIN FORMATTED*
> {code:java}
> (1) Scan json
> Output [1]: [col.dots#37L]
> Batched: false
> Location: InMemoryFileIndex [file:/private/var/folders/nr/j6hw4kr51wv0zynvr6srwgr00000gp/T/spark-89021d76-5971-4a96-bf10-0730873f6ce0]
> PushedFilters: [IsNotNull(`col.dots`), EqualTo(`col.dots`,500)]
> ReadSchema: struct<col.dots:bigint>{code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org