You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2019/09/18 03:56:00 UTC
[jira] [Assigned] (SPARK-28930) Spark DESC FORMATTED TABLENAME
information display issues
[ https://issues.apache.org/jira/browse/SPARK-28930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon reassigned SPARK-28930:
------------------------------------
Assignee: Sujith Chacko
> Spark DESC FORMATTED TABLENAME information display issues
> ---------------------------------------------------------
>
> Key: SPARK-28930
> URL: https://issues.apache.org/jira/browse/SPARK-28930
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.3
> Reporter: jobit mathew
> Assignee: Sujith Chacko
> Priority: Minor
>
> Spark DESC FORMATTED TABLENAME information display issues.Showing incorrect *Last Access time and* feeling some information displays can make it better.
> Test steps:
> 1. Open spark sql
> 2. Create table with partition
> CREATE EXTERNAL TABLE IF NOT EXISTS employees_info_extended ( id INT, name STRING, usd_flag STRING, salary DOUBLE, deductions MAP<STRING, DOUBLE>, address STRING ) PARTITIONED BY (entrytime STRING) STORED AS TEXTFILE location 'hdfs://hacluster/user/sparkhive/warehouse';
> 3. from spark sql check the table description
> desc formatted tablename;
> 4. From scala shell check the table description
> sql("desc formatted tablename").show()
> *Issue1:*
> If there is no comment for spark scala shell shows *"null" in small letters* but all other places Hive beeline/Spark beeline/Spark SQL it is showing in *CAPITAL "NULL*". Better to show same in all places.
>
> {code:java}
> *scala>* sql("desc formatted employees_info_extended").show(false);
> +-----------------------------+---------------------------++-------
> |col_name|data_type|*comment*|
> +-----------------------------+---------------------------++-------
> |id|int|*null*|
> |name|string|*null*|
> |usd_flag|string|*null*|
> |salary|double|*null*|
> |deductions|map<string,double>|*null*|
> |address|string|null|
> |entrytime|string|null|
> | # Partition Information| | |
> | # col_name|data_type|comment|
> |entrytime|string|null|
> | | | |
> | # Detailed Table Information| | |
> |Database|sparkdb__| |
> |Table|employees_info_extended| |
> |Owner|root| |
> *|Created Time |Tue Aug 20 13:42:06 CST 2019| |*
> *|Last Access |Thu Jan 01 08:00:00 CST 1970| |*
> |Created By|Spark 2.4.3| |
> |Type|EXTERNAL| |
> |Provider|hive| |
> +-----------------------------+---------------------------++-------
> only showing top 20 rows
> *scala>*
> {code}
> *Issue 2:*
> Spark SQL "desc formatted tablename" is not showing the header [# col_name,data_type,comment|#col_name,data_type,comment] in the top of the query result.But header is showing on top of partition description. For Better understanding show the header on Top of the query result.Other than in spark sql ,we are able to see the header like [# col_name,data_type,comment|#col_name,data_type,comment] in spark-beeline & hive beeline .
> {code:java}
> *spark-sql>* desc formatted employees_info_extended1;
> id int *NULL*
> name string *NULL*
> usd_flag string NULL
> salary double NULL
> deductions map<string,double> NULL
> address string NULL
> entrytime string NULL
> *
> ## Partition Information*
> ## col_name data_type comment*
> entrytime string *NULL*
>
> *spark-sql>* | | |
> {code}
>
> *This is Scala shell showing the headers "|col_name|data_type|*comment*| "*
>
> *scala>* sql("desc formatted employees_info_extended").show(false); +-----------------------------+---------------------------++------- *|col_name|data_type|*comment*|* +-----------------------------+---------------------------++------- |id|int|*null*| |name|string|*null*| |usd_flag|string|*null*|
>
> *This is the spark-beeline which is showing the headers*
> 0: jdbc:hive2://10.186.60.158:23040/default> desc formatted employees; +-------------------------------+---------------------------------------------------------------------------------+------------------+--+ | *col_name | data_type | comment | +-*------------------------------+---------------------------------------------------------------------------------+------------------+--+ | name | string | Employee name | | salary | float | Employee salary | | | | |
>
> *This is Hive Beeline showing headers*
> *0: jdbc:hive2://10.18.98.147:21066/> desc formatted mytest1;*
> INFO : Concurrency mode is disabled, not creating a lock manager
> +-------------------------------+------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> *| col_name | data_type | comment |*
> +-------------------------------+------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | # col_name | data_type | comment |
> | col | array<string> | from deserializer |
> | value | string
> *Issue 3:*
> I created the table on Aug 20.So it is showing created time correct .*But Last access time showing 1970 Jan 01*. It is not good to show Last access time earlier time than the created time.Better to show the correct date and time else show UNKNOWN.
> *[Created Time,Tue Aug 20 13:42:06 CST 2019,]*
> *[Last Access,Thu Jan 01 08:00:00 CST 1970,]*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org