[jira] [Resolved] (HIVE-28718) Aggregate storage statistics in Hive LLAP

["László Bodor (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-28718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

László Bodor resolved HIVE-28718.
---------------------------------

> Aggregate storage statistics in Hive LLAP
> -----------------------------------------
>
>                 Key: HIVE-28718
>                 URL: https://issues.apache.org/jira/browse/HIVE-28718
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: László Bodor
>            Priority: Major
>             Fix For: Not Applicable
>
>
> problem: there was TEZ-4451, which introduced IOStatisticsLogging logging, but it isn't sufficient, e.g. when running simple insert on mow-dev it prints like below (per attempt):
> {code}
> query-executor <14>1 2024-11-22T09:26:10.361Z query-executor-0-0 query-executor 1 f77182f8-03d1-4560-94a9-d17ae0b4c685 [mdc@38374 class="task.TaskRunner2Callable" dagId="dag_1732265989444_0000_2" fragmentId="1732265989444_0000_2_01_000000_0" level="INFO" queryId="hive_20241122092606_33b043a5-720a-48f0-8213-e34b34480313" thread="TezTR-989444_0_2_1_0_0"] TaskAttemptId=attempt_1732265989444_0000_2_01_000000_0, counters=((stream_write_bytes=14));\r\rgauges=();\r\rminimums=();\r\rmaximums=();\r\rmeans=();\r
> {code}
> but when I enable logging in hadoop:
> {code}
>        <property>
>          <name>fs.iostatistics.logging.level</name>
>          <value>info</value>
>        </property>
> {code}
> it gives stats like below at the end of the query:
> {code}
> query-executor <14>1 2024-11-22T09:26:14.150Z query-executor-0-0 query-executor 1 f77182f8-03d1-4560-94a9-d17ae0b4c685 [mdc@38374 class="statistics.IOStatisticsLogging" level="INFO" thread="IPC Server handler 3 on 25000"] IOStatistics: counters=((action_http_head_request=11)\r(audit_request_execution=25)\r(audit_span_creation=9)\r(directories_created=2)\r(files_copied=1)\r(files_copied_bytes=213)\r(files_created=1)\r(files_deleted=1)\r(ignored_errors=1)\r(object_copy_requests=1)\r(object_delete_objects=1)\r(object_delete_request=1)\r(object_list_request=9)\r(object_metadata_request=11)\r(object_put_bytes=213)\r(object_put_request=3)\r(object_put_request_completed=3)\r(op_create=1)\r(op_exists=4)\r(op_exists.failures=2)\r(op_mkdirs=1)\r(op_rename=1)\r(store_io_request=25)\r(stream_write_block_uploads=1)\r(stream_write_bytes=213)\r(stream_write_total_data=426));\r\rgauges=((stream_write_block_uploads_pending=1));\r\rminimums=((action_http_head_request.min=10)\r(object_delete_request.min=25)\r(object_list_request.min=13)\r(object_put_request.min=19)\r(op_create.min=83)\r(op_exists.failures.min=28)\r(op_exists.min=24)\r(op_mkdirs.min=64)\r(op_rename.min=209)\r(store_io_rate_limited_duration.min=0));\r\rmaximums=((action_http_head_request.max=539)\r(object_delete_request.max=25)\r(object_list_request.max=59)\r(object_put_request.max=99)\r(op_create.max=83)\r(op_exists.failures.max=621)\r(op_exists.max=41)\r(op_mkdirs.max=64)\r(op_rename.max=209)\r(store_io_rate_limited_duration.max=0));\r\rmeans=((action_http_head_request.mean=(samples=11, sum=680, mean=61.8182))\r(object_delete_request.mean=(samples=1, sum=25, mean=25.0000))\r(object_list_request.mean=(samples=9, sum=190, mean=21.1111))\r(object_put_request.mean=(samples=3, sum=140, mean=46.6667))\r(op_create.mean=(samples=1, sum=83, mean=83.0000))\r(op_exists.failures.mean=(samples=2, sum=649, mean=324.5000))\r(op_exists.mean=(samples=2, sum=65, mean=32.5000))\r(op_mkdirs.mean=(samples=1, sum=64, mean=64.0000))\r(op_rename.mean=(samples=1, sum=209, mean=209.0000))\r(store_io_rate_limited_duration.mean=(samples=1, sum=0, mean=0.0000)));\r
> {code}
> we can see that the latter is much more sufficient, we should enable it by default
> after HIVE-27884, filesystem is shared between attempts of the same query, so the latter is printed after the query, once, which can also be used for aggregation, serving useful counters on the query summary
> we need to experiment with this to become able to point AWS related issues much faster
> default logging is one question, possible aggregation on query level is another one, which is also tracked on TEZ-4590



--
This message was sent by Atlassian Jira
(v8.20.10#820010)