You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2024/10/31 00:02:00 UTC
[jira] [Resolved] (IMPALA-13340) COPY TESTCASE in LocalCatalog mode doesn't dump the partition and file metadata
[ https://issues.apache.org/jira/browse/IMPALA-13340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Quanlong Huang resolved IMPALA-13340.
-------------------------------------
Fix Version/s: Impala 4.5.0
Resolution: Fixed
Resolving this. Thank [~MikaelSmith] and [~jasonmfehr] for the review!
> COPY TESTCASE in LocalCatalog mode doesn't dump the partition and file metadata
> -------------------------------------------------------------------------------
>
> Key: IMPALA-13340
> URL: https://issues.apache.org/jira/browse/IMPALA-13340
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Fix For: Impala 4.5.0
>
>
> IMPALA-11901 fixes the failures of using COPY TESTCASE statements in LocalCatalog mode. However, only the table metadata is dumped, e.g. table schema, column stats. The partition and file metadata are missing.
> To reproduce the issue locally, start the Impala cluster in LocalCatalog mode.
> {code:bash}
> bin/start-impala-cluster.py --catalogd_args=--catalog_topic_mode=minimal --impalad_args=--use_local_catalog{code}
> Dump the metadata of a query on a partitioned table:
> {noformat}
> copy testcase to '/tmp' select * from functional_parquet.alltypes;
> +--------------------------------------------------------------------------------------+
> | Test case data output path |
> +--------------------------------------------------------------------------------------+
> | hdfs://localhost:20500/tmp/impala-testcase-data-c8316356-6448-4458-acad-c2f72f43c3e1 |
> +--------------------------------------------------------------------------------------+ {noformat}
> Check the metadata from the source cluster
> {noformat}
> show partitions functional_parquet.alltypes
> +-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+
> | year | month | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | EC Policy |
> +-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+
> | 2009 | 1 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=1 | NONE |
> | 2009 | 2 | -1 | 1 | 8.09KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=2 | NONE |
> | 2009 | 3 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=3 | NONE |
> | 2009 | 4 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=4 | NONE |
> | 2009 | 5 | -1 | 1 | 8.55KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=5 | NONE |
> | 2009 | 6 | -1 | 1 | 8.23KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=6 | NONE |
> | 2009 | 7 | -1 | 1 | 8.25KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=7 | NONE |
> | 2009 | 8 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=8 | NONE |
> | 2009 | 9 | -1 | 1 | 8.41KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=9 | NONE |
> | 2009 | 10 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=10 | NONE |
> | 2009 | 11 | -1 | 1 | 8.44KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=11 | NONE |
> | 2009 | 12 | -1 | 1 | 8.59KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=12 | NONE |
> | 2010 | 1 | -1 | 1 | 8.38KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=1 | NONE |
> | 2010 | 2 | -1 | 1 | 7.80KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=2 | NONE |
> | 2010 | 3 | -1 | 1 | 8.29KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=3 | NONE |
> | 2010 | 4 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=4 | NONE |
> | 2010 | 5 | -1 | 1 | 8.62KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=5 | NONE |
> | 2010 | 6 | -1 | 1 | 8.26KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=6 | NONE |
> | 2010 | 7 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=7 | NONE |
> | 2010 | 8 | -1 | 1 | 8.64KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=8 | NONE |
> | 2010 | 9 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=9 | NONE |
> | 2010 | 10 | -1 | 1 | 8.59KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=10 | NONE |
> | 2010 | 11 | -1 | 1 | 8.20KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=11 | NONE |
> | 2010 | 12 | -1 | 1 | 8.60KB | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=12 | NONE |
> | Total | | -1 | 24 | 201.52KB | 0B | | | | | |
> +-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+{noformat}
> Import the metadata
> {noformat}
> copy testcase from 'hdfs://localhost:20500/tmp/impala-testcase-data-c8316356-6448-4458-acad-c2f72f43c3e1';
> +----------------------------------------------------------------------------------------------------------------+
> | summary |
> +----------------------------------------------------------------------------------------------------------------+
> | Testcase generated using Impala version 4.5.0-SNAPSHOT. 1 db(s), 1 table(s) and 0 view(s) imported for query: |
> | |
> | SELECT * FROM functional_parquet.alltypes |
> +----------------------------------------------------------------------------------------------------------------+{noformat}
> Check the partition metadata, they are missing:
> {noformat}
> set PLANNER_TESTCASE_MODE=true;
> show partitions functional_parquet.alltypes;
> Query: show partitions functional_parquet.alltypes
> +-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+
> | year | month | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | EC Policy |
> +-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+
> | Total | | -1 | 0 | 0B | 0B | | | | | |
> +-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org