You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/09/19 07:30:00 UTC
[jira] [Commented] (IMPALA-10076) Reduce logs about partition level
catalog updates
[ https://issues.apache.org/jira/browse/IMPALA-10076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198677#comment-17198677 ]
ASF subversion and git services commented on IMPALA-10076:
----------------------------------------------------------
Commit 8e745a778078b8b0f07573d6c2f53d80bc079a12 in impala's branch refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=8e745a7 ]
IMPALA-10076: Reduce partition level update logs
Catalogd and the legacy mode coordinator log catalog topic items one by
a line. A table can have more than 100K partitions. It's too verbose to
log all partition level updates. This patch aggregates the partition
level updates of each table and only logs the aggregated info in a line.
For instance, here is an aggregated log:
Collected 24 partition update(s):
1:HDFS_PARTITION:test.tbl:(p=1,p=2,...,p=24), version=1451, original
size=(avg=646, min=646, max=648, sum=15516), compressed size=(avg=478,
min=475, max=485, sum=11487)
If there are more than 3 partitions, we only show the lexicographically
smallest, the second smallest and the largest partition names. If there
are only one partition update, the log format is the same as before:
Collected 1 partition update(s): 1:HDFS_PARTITION:test.tbl:p=0,
version=1451, original size=648, compressed size=475
To support these, this patch implements a helper class,
PartitionMetaSummary, to aggregate these information and get a summary
for logging. The compressed size is calculated by BE. So the jni method
for NativeAddPendingTopicItem is extended to return the actual
compressed size.
Tests:
- Added unit tests for PartitionMetaSummary.
- Start the cluster in mixed catalog mode and manually verified the
logs.
Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368
Reviewed-on: http://gerrit.cloudera.org:8080/16375
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Reduce logs about partition level catalog updates
> -------------------------------------------------
>
> Key: IMPALA-10076
> URL: https://issues.apache.org/jira/browse/IMPALA-10076
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> After IMPALA-3127, catalogd will send catalog updates in partition level. So catalogd will log updates in partition level, e.g.
> {code:java}
> I0812 17:12:18.367053 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=6, version=1451, original size=702, compressed size=529
> I0812 17:12:18.367229 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=7, version=1451, original size=702, compressed size=530
> I0812 17:12:18.367377 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=8, version=1451, original size=702, compressed size=528
> I0812 17:12:18.367514 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=9, version=1451, original size=702, compressed size=530
> I0812 17:12:18.367652 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=1, version=1451, original size=702, compressed size=530
> I0812 17:12:18.367787 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=10, version=1451, original size=704, compressed size=527
> I0812 17:12:18.367918 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=11, version=1451, original size=704, compressed size=530
> I0812 17:12:18.368062 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=12, version=1451, original size=704, compressed size=532
> I0812 17:12:18.368203 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=2, version=1451, original size=702, compressed size=529
> I0812 17:12:18.368335 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=3, version=1451, original size=702, compressed size=528
> I0812 17:12:18.368495 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=4, version=1451, original size=702, compressed size=527
> I0812 17:12:18.368629 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=5, version=1451, original size=702, compressed size=525
> I0812 17:12:18.368762 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=6, version=1451, original size=702, compressed size=529
> I0812 17:12:18.368984 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=7, version=1451, original size=702, compressed size=530
> I0812 17:12:18.369117 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=8, version=1451, original size=702, compressed size=529
> I0812 17:12:18.369246 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=9, version=1451, original size=702, compressed size=530
> I0812 17:12:18.369390 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=1, version=1451, original size=702, compressed size=529
> I0812 17:12:18.369525 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=10, version=1451, original size=704, compressed size=525
> I0812 17:12:18.369654 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=11, version=1451, original size=704, compressed size=530
> I0812 17:12:18.369783 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=12, version=1451, original size=704, compressed size=533
> I0812 17:12:18.369912 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=2, version=1451, original size=702, compressed size=528
> I0812 17:12:18.370040 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=3, version=1451, original size=702, compressed size=528
> I0812 17:12:18.370168 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=4, version=1451, original size=702, compressed size=527
> I0812 17:12:18.370322 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=5, version=1451, original size=702, compressed size=526
> I0812 17:12:18.371448 29433 catalog-server.cc:737] Collected update: 1:TABLE:functional.alltypes, version=1451, original size=3944, compressed size=1859
> I0812 17:12:18.373958 29433 catalog-server.cc:737] Collected update: 1:CATALOG_SERVICE_ID, version=1451, original size=60, compressed size=58
> I0812 17:12:20.364428 29443 catalog-server.cc:342] A catalog update with 26 entries is assembled. Catalog version: 1451 Last sent catalog version: 1450{code}
> Similarly, coordinators will log them as well, e.g.
> {code:java}
> I0812 17:12:21.240887 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=6 version: 1451 size: 702
> I0812 17:12:21.241179 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=7 version: 1451 size: 702
> I0812 17:12:21.241423 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=8 version: 1451 size: 702
> I0812 17:12:21.241655 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=9 version: 1451 size: 702
> I0812 17:12:21.241880 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=1 version: 1451 size: 702
> I0812 17:12:21.242110 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=10 version: 1451 size: 704
> I0812 17:12:21.242337 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=11 version: 1451 size: 704
> I0812 17:12:21.242563 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=12 version: 1451 size: 704
> I0812 17:12:21.242816 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=2 version: 1451 size: 702
> I0812 17:12:21.243049 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=3 version: 1451 size: 702
> I0812 17:12:21.243273 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=4 version: 1451 size: 702
> I0812 17:12:21.243499 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=5 version: 1451 size: 702
> I0812 17:12:21.243721 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=6 version: 1451 size: 702
> I0812 17:12:21.243942 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=7 version: 1451 size: 702
> I0812 17:12:21.244164 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=8 version: 1451 size: 702
> I0812 17:12:21.244388 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=9 version: 1451 size: 702
> I0812 17:12:21.244612 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=1 version: 1451 size: 702
> I0812 17:12:21.244838 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=10 version: 1451 size: 704
> I0812 17:12:21.245064 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=11 version: 1451 size: 704
> I0812 17:12:21.245330 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=12 version: 1451 size: 704
> I0812 17:12:21.245561 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=2 version: 1451 size: 702
> I0812 17:12:21.245918 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=3 version: 1451 size: 702
> I0812 17:12:21.246156 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=4 version: 1451 size: 702
> I0812 17:12:21.246387 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=5 version: 1451 size: 702
> I0812 17:12:21.247385 30308 ImpaladCatalog.java:222] Adding: TABLE:functional.alltypes version: 1451 size: 3944
> I0812 17:12:21.247516 30308 ImpaladCatalog.java:222] Adding: CATALOG_SERVICE_ID version: 1451 size: 60
> I0812 17:12:21.248576 30308 impala-server.cc:1770] Catalog topic update applied with version: 1451 new min catalog object version: 3{code}
> These are too verbose. Especially for tables with many partitions. We should find a way to reduce these logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org