You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Michael Smith (Jira)" <ji...@apache.org> on 2022/11/16 00:00:00 UTC
[jira] [Commented] (IMPALA-9487) SHOW and DESCRIBE statements should display EC policies
[ https://issues.apache.org/jira/browse/IMPALA-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17634577#comment-17634577 ]
Michael Smith commented on IMPALA-9487:
---------------------------------------
h2. SHOW FILES
Add a new column to SHOW FILES output. Only add this column when at least one file is erasure coded to avoid changing the format for existing users.
{code:java}
Query: show files in ect
+---------------------------------------------------------------------------------------------------------------------+------+---------------------+--------------+
| Path | Size | Partition | EC Policy |
+---------------------------------------------------------------------------------------------------------------------+------+---------------------+--------------+
| hdfs://localhost:20500/ec-warehouse/ect/date_col=2009-01-01/1d46b6cbf747c7a8-a818238e00000000_1347890556_data.0.txt | 338B | date_col=2009-01-01 | RS-3-2-1024k |
| hdfs://localhost:20500/ec-warehouse/ect/date_col=2009-01-02/1d46b6cbf747c7a8-a818238e00000000_588711848_data.0.txt | 353B | date_col=2009-01-02 | RS-3-2-1024k |
| hdfs://localhost:20500/ec-warehouse/ect/date_col=2009-01-03/1d46b6cbf747c7a8-a818238e00000000_140483252_data.0.txt | 353B | date_col=2009-01-03 | RS-3-2-1024k |
| hdfs://localhost:20500/ec-warehouse/ect/date_col=2009-01-04/1d46b6cbf747c7a8-a818238e00000000_795616328_data.0.txt | 353B | date_col=2009-01-04 | RS-3-2-1024k |
| hdfs://localhost:20500/ec-warehouse/ect/date_col=2009-01-05/1d46b6cbf747c7a8-a818238e00000000_234666509_data.0.txt | 353B | date_col=2009-01-05 | RS-3-2-1024k |
+---------------------------------------------------------------------------------------------------------------------+------+---------------------+--------------+
Fetched 5 row(s) in 0.07s
{code}
h2. DESCRIBE EXTENDED
If the table location (a directory) has an erasure coding policy, list it under storage information.
{code:java}
Query: describe extended ect
+------------------------------+------------------------------------------------------------+----------------------+
| name | type | comment |
+------------------------------+------------------------------------------------------------+----------------------+
| # col_name | data_type | comment |
| | NULL | NULL |
| id | int | Add a comment |
| bool_col | boolean | NULL |
| tinyint_col | tinyint | NULL |
| smallint_col | smallint | NULL |
| int_col | int | NULL |
| bigint_col | bigint | NULL |
| float_col | float | NULL |
| double_col | double | NULL |
| string_col | string | NULL |
| timestamp_col | timestamp | NULL |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| | NULL | NULL |
| date_col | date | NULL |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | default | NULL |
| OwnerType: | USER | NULL |
| Owner: | michael | NULL |
| CreateTime: | Tue Nov 15 15:10:21 PST 2022 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Location: | hdfs://localhost:20500/ec-warehouse/ect | NULL |
| Table Type: | EXTERNAL_TABLE | NULL |
| Table Parameters: | NULL | NULL |
| | DO_NOT_UPDATE_STATS | true |
| | EXTERNAL | TRUE |
| | OBJCAPABILITIES | EXTREAD,EXTWRITE |
| | STATS_GENERATED | TASK |
| | TRANSLATED_TO_EXTERNAL | TRUE |
| | external.table.purge | TRUE |
| | impala.lastComputeStatsTime | 1668549634 |
| | numRows | -1 |
| | totalSize | 27923 |
| | transient_lastDdlTime | 1668548361 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL |
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | 0 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| Erasure coding policy: | RS-3-2-1024k | NULL |
| | NULL | NULL |
| # Constraints | NULL | NULL |
+------------------------------+------------------------------------------------------------+----------------------+
Fetched 50 row(s) in 0.02s
{code}
> SHOW and DESCRIBE statements should display EC policies
> -------------------------------------------------------
>
> Key: IMPALA-9487
> URL: https://issues.apache.org/jira/browse/IMPALA-9487
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
> Reporter: Sahil Takiar
> Assignee: Michael Smith
> Priority: Major
> Labels: observability
>
> Since EC policies can be set per-file, the {{show files}} command should display if a file is an EC file, and what the EC policy is.
> EC policies can be set on a table level directory, so it would be useful if 'describe extended [table-name]' indicated if the table had an EC policy set or not.
> For partitioned tables, {{show partitions}} should list out the EC policy of each partition directory (we already do something similar for HDFS cacheing).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org