You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Joe McDonnell (Jira)" <ji...@apache.org> on 2023/10/19 04:58:00 UTC

[jira] [Commented] (IMPALA-12191) Add hardware and OS details to runtime profile

    [ https://issues.apache.org/jira/browse/IMPALA-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17777015#comment-17777015 ] 

Joe McDonnell commented on IMPALA-12191:
----------------------------------------

Here are some hints for new devs:

When Impala executes a query, it keeps a runtime profile with a lot of statistics about what happened and how long things took. To see a runtime profile, do the following (with a running Impala):

 
{noformat}
# Start impala-shell
${IMPALA_HOME}/bin/impala-shell.sh

# Create a table and insert a value
create table foo (i int);
insert into foo values (1);


# Run a simple query (this is using some builtin 
select * from foo;
# This will have output like this:
[localhost:21050] default> select * from foo;
Query: select * from foo
Query submitted at: 2023-10-18 21:47:43 (Coordinator: http://hostname:25000)
Query progress can be monitored at: http://hostname:25000/query_plan?query_id=0143a16ae13ffa30:31c5605a00000000
+---+
| i |
+---+
| 1 |
+---+
Fetched 1 row(s) in 0.14s{noformat}
That "Query progress can be monitored at:" URL goes to the runtime profile (served up by the Impala Web UI)

The profile looks something like this:
{noformat}
Query (id=0143a16ae13ffa30:31c5605a00000000):
  DEBUG MODE WARNING: Query profile created while running a DEBUG build of Impala. Use RELEASE builds to measure query performance.
  Summary:
    Session ID: c843e81384684663:108499c34fd8f18b
    Session Type: HIVESERVER2
    HiveServer2 Protocol Version: V6
    Start Time: 2023-10-18 21:47:43.932465000
    End Time: 2023-10-18 21:47:44.070168000
    Duration: 137.703ms (137703 us)
    Query Type: QUERY
    Query State: FINISHED
    Impala Query State: FINISHED
    Query Status: OK
    Impala Version: impalad version 4.4.0-SNAPSHOT DEBUG (build 098ad53f656f0e623574778f13a1bba055e75cfc)
    User: joemcdonnell
    Connected User: joemcdonnell
    Delegated User: 
    Network Address: 127.0.0.1:50186
    Default Db: default
    Sql Statement: select * from foo
    Coordinator: joemcdonnell-22743:27000
... huge amount more text ...{noformat}
This Jira wants to add something to this top portion with information about the CPU info and OS. Basically, we want something like the "Impala Version" listed there. We can use "git grep" on the "be" directory to find this:
{noformat}
$ git grep "Impala Version" be/
be/src/service/client-request-state.cc:  summary_profile_->AddInfoString("Impala Version", GetVersionString(/* compact */ true));{noformat}
[https://github.com/apache/impala/blob/master/be/src/service/client-request-state.cc#L161]

We have some utilities with CPU information in [https://github.com/apache/impala/blob/master/be/src/util/cpu-info.h] . Look at DebugString().

For OS information, see [https://github.com/apache/impala/blob/master/be/src/util/os-util.h]

 

> Add hardware and OS details to runtime profile
> ----------------------------------------------
>
>                 Key: IMPALA-12191
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12191
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: be
>            Reporter: David Rorke
>            Assignee: David Rorke
>            Priority: Major
>
> The runtime profiles are currently lacking any details about the hardware the query ran on (CPU model and core count, cache sizes, etc) OS versions, etc which may all be relevant when analyzing performance issues, comparing performance metrics across different profiles, etc.
> We should add relevant hardware and OS details to the profile.  The information currently displayed at the root Impalad web UI page (Hardware Info, OS Info) would be a good starting point.
> IMPALA-12118 is also relevant if we want to cover ARM processors.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org