You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/08/09 07:32:26 UTC
[GitHub] [spark] wangyum opened a new pull request #25395:
[SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with
SQLConf.get.redactOptions
wangyum opened a new pull request #25395: [SPARK-28675][SQL] Replace CatalogUtils.maskCredentials with SQLConf.get.redactOptions
URL: https://github.com/apache/spark/pull/25395
## What changes were proposed in this pull request?
This PR replaces `CatalogUtils.maskCredentials` with `SQLConf.get.redactOptions` to consistent with redact logic.
## How was this patch tested?
manual tests:
Before this PR:
```sql
spark-sql> desc extended mysql_federated_sample;
TBL_ID bigint NULL
CREATE_TIME int NULL
DB_ID bigint NULL
LAST_ACCESS_TIME int NULL
OWNER string NULL
RETENTION int NULL
SD_ID bigint NULL
TBL_NAME string NULL
TBL_TYPE string NULL
VIEW_EXPANDED_TEXT string NULL
VIEW_ORIGINAL_TEXT string NULL
IS_REWRITE_ENABLED boolean NULL
# Detailed Table Information
Database default
Table mysql_federated_sample
Owner root
Created Time Thu Aug 08 16:53:43 GMT-07:00 2019
Last Access Wed Dec 31 17:00:00 GMT-07:00 1969
Created By Spark 3.0.0-SNAPSHOT
Type MANAGED
Provider org.apache.spark.sql.jdbc
Location file:/user/hive/warehouse/mysql_federated_sample
Serde Library org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat org.apache.hadoop.mapred.SequenceFileInputFormat
OutputFormat org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Storage Properties [url=###, driver=com.mysql.jdbc.Driver, dbtable=TBLS]
spark-sql> SHOW TABLE EXTENDED LIKE 'mysql_federated_sample';
19/08/08 17:04:16 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
default mysql_federated_sample false Database: default
Table: mysql_federated_sample
Owner: root
Created Time: Thu Aug 08 16:53:43 GMT-07:00 2019
Last Access: Wed Dec 31 17:00:00 GMT-07:00 1969
Created By: Spark 3.0.0-SNAPSHOT
Type: MANAGED
Provider: org.apache.spark.sql.jdbc
Location: file:/user/hive/warehouse/mysql_federated_sample
Serde Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.SequenceFileInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Storage Properties: [url=###, driver=com.mysql.jdbc.Driver, dbtable=TBLS]
Schema: root
|-- TBL_ID: long (nullable = true)
|-- CREATE_TIME: integer (nullable = true)
|-- DB_ID: long (nullable = true)
|-- LAST_ACCESS_TIME: integer (nullable = true)
|-- OWNER: string (nullable = true)
|-- RETENTION: integer (nullable = true)
|-- SD_ID: long (nullable = true)
|-- TBL_NAME: string (nullable = true)
|-- TBL_TYPE: string (nullable = true)
|-- VIEW_EXPANDED_TEXT: string (nullable = true)
|-- VIEW_ORIGINAL_TEXT: string (nullable = true)
|-- IS_REWRITE_ENABLED: boolean (nullable = true)
```
After this PR:
```sql
spark-sql> desc formatted mysql_federated_sample;
TBL_ID bigint NULL
CREATE_TIME int NULL
DB_ID bigint NULL
LAST_ACCESS_TIME int NULL
OWNER string NULL
RETENTION int NULL
SD_ID bigint NULL
TBL_NAME string NULL
TBL_TYPE string NULL
VIEW_EXPANDED_TEXT string NULL
VIEW_ORIGINAL_TEXT string NULL
IS_REWRITE_ENABLED boolean NULL
# Detailed Table Information
Database default
Table mysql_federated_sample
Owner root
Created Time Thu Aug 08 17:35:35 GMT-07:00 2019
Last Access Wed Dec 31 17:00:00 GMT-07:00 1969
Created By Spark 3.0.0-SNAPSHOT
Type MANAGED
Provider org.apache.spark.sql.jdbc
Location file:/user/hive/warehouse/mysql_federated_sample
Serde Library org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat org.apache.hadoop.mapred.SequenceFileInputFormat
OutputFormat org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Storage Properties [url=*********(redacted), driver=com.mysql.jdbc.Driver, dbtable=TBLS]
spark-sql> SHOW TABLE EXTENDED LIKE 'mysql_federated_sample';
19/08/08 17:36:07 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
default mysql_federated_sample false Database: default
Table: mysql_federated_sample
Owner: root
Created Time: Thu Aug 08 17:35:35 GMT-07:00 2019
Last Access: Wed Dec 31 17:00:00 GMT-07:00 1969
Created By: Spark 3.0.0-SNAPSHOT
Type: MANAGED
Provider: org.apache.spark.sql.jdbc
Location: file:/user/hive/warehouse/mysql_federated_sample
Serde Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.SequenceFileInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Storage Properties: [url=*********(redacted), driver=com.mysql.jdbc.Driver, dbtable=TBLS]
Schema: root
|-- TBL_ID: long (nullable = true)
|-- CREATE_TIME: integer (nullable = true)
|-- DB_ID: long (nullable = true)
|-- LAST_ACCESS_TIME: integer (nullable = true)
|-- OWNER: string (nullable = true)
|-- RETENTION: integer (nullable = true)
|-- SD_ID: long (nullable = true)
|-- TBL_NAME: string (nullable = true)
|-- TBL_TYPE: string (nullable = true)
|-- VIEW_EXPANDED_TEXT: string (nullable = true)
|-- VIEW_ORIGINAL_TEXT: string (nullable = true)
|-- IS_REWRITE_ENABLED: boolean (nullable = true)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org