You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2020/08/27 09:09:00 UTC
[jira] [Work started] (IMPALA-10076) Reduce logs about partition
level catalog updates
[ https://issues.apache.org/jira/browse/IMPALA-10076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on IMPALA-10076 started by Quanlong Huang.
-----------------------------------------------
> Reduce logs about partition level catalog updates
> -------------------------------------------------
>
> Key: IMPALA-10076
> URL: https://issues.apache.org/jira/browse/IMPALA-10076
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> After IMPALA-3127, catalogd will send catalog updates in partition level. So catalogd will log updates in partition level, e.g.
> {code:java}
> I0812 17:12:18.367053 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=6, version=1451, original size=702, compressed size=529
> I0812 17:12:18.367229 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=7, version=1451, original size=702, compressed size=530
> I0812 17:12:18.367377 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=8, version=1451, original size=702, compressed size=528
> I0812 17:12:18.367514 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=9, version=1451, original size=702, compressed size=530
> I0812 17:12:18.367652 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=1, version=1451, original size=702, compressed size=530
> I0812 17:12:18.367787 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=10, version=1451, original size=704, compressed size=527
> I0812 17:12:18.367918 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=11, version=1451, original size=704, compressed size=530
> I0812 17:12:18.368062 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=12, version=1451, original size=704, compressed size=532
> I0812 17:12:18.368203 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=2, version=1451, original size=702, compressed size=529
> I0812 17:12:18.368335 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=3, version=1451, original size=702, compressed size=528
> I0812 17:12:18.368495 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=4, version=1451, original size=702, compressed size=527
> I0812 17:12:18.368629 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=5, version=1451, original size=702, compressed size=525
> I0812 17:12:18.368762 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=6, version=1451, original size=702, compressed size=529
> I0812 17:12:18.368984 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=7, version=1451, original size=702, compressed size=530
> I0812 17:12:18.369117 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=8, version=1451, original size=702, compressed size=529
> I0812 17:12:18.369246 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=9, version=1451, original size=702, compressed size=530
> I0812 17:12:18.369390 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=1, version=1451, original size=702, compressed size=529
> I0812 17:12:18.369525 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=10, version=1451, original size=704, compressed size=525
> I0812 17:12:18.369654 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=11, version=1451, original size=704, compressed size=530
> I0812 17:12:18.369783 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=12, version=1451, original size=704, compressed size=533
> I0812 17:12:18.369912 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=2, version=1451, original size=702, compressed size=528
> I0812 17:12:18.370040 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=3, version=1451, original size=702, compressed size=528
> I0812 17:12:18.370168 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=4, version=1451, original size=702, compressed size=527
> I0812 17:12:18.370322 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=5, version=1451, original size=702, compressed size=526
> I0812 17:12:18.371448 29433 catalog-server.cc:737] Collected update: 1:TABLE:functional.alltypes, version=1451, original size=3944, compressed size=1859
> I0812 17:12:18.373958 29433 catalog-server.cc:737] Collected update: 1:CATALOG_SERVICE_ID, version=1451, original size=60, compressed size=58
> I0812 17:12:20.364428 29443 catalog-server.cc:342] A catalog update with 26 entries is assembled. Catalog version: 1451 Last sent catalog version: 1450{code}
> Similarly, coordinators will log them as well, e.g.
> {code:java}
> I0812 17:12:21.240887 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=6 version: 1451 size: 702
> I0812 17:12:21.241179 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=7 version: 1451 size: 702
> I0812 17:12:21.241423 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=8 version: 1451 size: 702
> I0812 17:12:21.241655 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=9 version: 1451 size: 702
> I0812 17:12:21.241880 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=1 version: 1451 size: 702
> I0812 17:12:21.242110 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=10 version: 1451 size: 704
> I0812 17:12:21.242337 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=11 version: 1451 size: 704
> I0812 17:12:21.242563 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=12 version: 1451 size: 704
> I0812 17:12:21.242816 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=2 version: 1451 size: 702
> I0812 17:12:21.243049 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=3 version: 1451 size: 702
> I0812 17:12:21.243273 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=4 version: 1451 size: 702
> I0812 17:12:21.243499 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=5 version: 1451 size: 702
> I0812 17:12:21.243721 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=6 version: 1451 size: 702
> I0812 17:12:21.243942 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=7 version: 1451 size: 702
> I0812 17:12:21.244164 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=8 version: 1451 size: 702
> I0812 17:12:21.244388 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=9 version: 1451 size: 702
> I0812 17:12:21.244612 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=1 version: 1451 size: 702
> I0812 17:12:21.244838 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=10 version: 1451 size: 704
> I0812 17:12:21.245064 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=11 version: 1451 size: 704
> I0812 17:12:21.245330 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=12 version: 1451 size: 704
> I0812 17:12:21.245561 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=2 version: 1451 size: 702
> I0812 17:12:21.245918 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=3 version: 1451 size: 702
> I0812 17:12:21.246156 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=4 version: 1451 size: 702
> I0812 17:12:21.246387 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=5 version: 1451 size: 702
> I0812 17:12:21.247385 30308 ImpaladCatalog.java:222] Adding: TABLE:functional.alltypes version: 1451 size: 3944
> I0812 17:12:21.247516 30308 ImpaladCatalog.java:222] Adding: CATALOG_SERVICE_ID version: 1451 size: 60
> I0812 17:12:21.248576 30308 impala-server.cc:1770] Catalog topic update applied with version: 1451 new min catalog object version: 3{code}
> These are too verbose. Especially for tables with many partitions. We should find a way to reduce these logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org