You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/10/21 06:39:00 UTC
[jira] [Commented] (IMPALA-10168) Expose JSON catalog objects in
catalogd's debug page
[ https://issues.apache.org/jira/browse/IMPALA-10168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218137#comment-17218137 ]
ASF subversion and git services commented on IMPALA-10168:
----------------------------------------------------------
Commit ee4043e1a0940ae5711c68336d1ad522631d0e35 in impala's branch refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ee4043e ]
IMPALA-10168: Expose JSON catalog objects in catalogd's debug page
Catalogd has a debug page at '/catalog_object' showing catalog objects
in thrift debug strings. It's inconvenient for tests to parse the thrift
string and get interesting infos.
This patch extends this page to support returning JSON results, which
eases tests to extract complex infos from the catalog objects, e.g.
partition ids of a hdfs table. Just like getting json results from other
pages, the usage is adding a ‘json’ argument in the URL, e.g.
http://localhost:25020/catalog_object?json&object_type=TABLE&object_name=db1.tbl1
Implementation:
Csaba helped to find that Thrift has a protocol, TSimpleJSONProtocol,
which can convert thrift objects to human readable JSON strings. This
simplifies the implementation a lot. However, TSimpleJSONProtocol is not
implemented in cpp yet (THRIFT-2476). So we do the conversion in FE to
use its java implementation.
Tests:
- Add tests to verify json fields existence.
Change-Id: I15f256b4e3f5206c7140746694106e03b0a4ad92
Reviewed-on: http://gerrit.cloudera.org:8080/16449
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Expose JSON catalog objects in catalogd's debug page
> ----------------------------------------------------
>
> Key: IMPALA-10168
> URL: https://issues.apache.org/jira/browse/IMPALA-10168
> Project: IMPALA
> Issue Type: New Feature
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
>
> Catalogd has a debug web page for showing catalog objects. Example URL: [http://localhost:25020/catalog_object?object_type=TABLE&object_name=functional.alltypes]
> Catalog objects are shown in thrift debug strings which are hard to parse for tests. We do have some tests parsing simple things from the thrift debug strings. For instances, we extract the catalog version here:
> [https://github.com/apache/impala/blob/fc19e70cbc94735c38e64573c1e015878f218281/tests/common/impala_service.py#L165]
> We verify whether a table is loaded by checking some substrings:
> [https://github.com/apache/impala/blob/9ecbe7d3dc22d7233e4bf2f91aaab43d0ad4e373/tests/custom_cluster/test_automatic_invalidation.py#L33-L34]
> However, it's hard to parse the partition ids of a hdfs table since the debug string is too complex. It'd be better to support exposing JSON results to ease test codes. JSON format results also help for customer to develop their monitor tools.
> The new URL can be something like [http://localhost:25020/catalog_object?json&object_type=TABLE&object_name=functional.alltypes] (Just adding a "json" parameter).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org