You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Surya Hebbar (Code Review)" <ge...@cloudera.org> on 2024/12/19 14:53:47 UTC
[Impala-ASF-CR] IMPALA-13624: Implement textual representation of aggregate event sequences
Surya Hebbar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22245
Change subject: IMPALA-13624: Implement textual representation of aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation of aggregate event sequences
Currently, within the JSON query profile the event timeline among
fragment instances is expressed in list format, without any form
of summarization.
The representation of event sequences is verbose, and only the first
instance timestamps are printed with default verbosity and all instances
timestamps in case of extended verbosity.
With this format, it becomes difficult to analyze a particular phase
across instances.
With the inclusion of aggregate metrics in the JSON query profile,
it is also possible to display them in a comprehensive format,
which would help in skew analysis or other such cases.
The default verbosity would now print a summary of event sequences in
the following manner instead of just the first instance timestamps.
When no. of instances > 5 -
Node Lifecycle Event Timeline Summary :
- Open Started:
Div. Max Min Avg #Inst. Count
20% 2s133ms 2s000ms 2s050ms 3
40% 2s300ms 2s200ms 2s250ms 4
.... .... .... .... ...
100% 2s850ms 2s900ms 2s875ms 2
- Open Finished: ...
... ... ... ...
When no. of instances <= 5 -
Node Lifecycle Event Timeline :
- Open Started: 2s850ms, 3s850ms, 4s150ms, 5s850ms
- Open Finished: 4s094ms, 5s850ms, 6s150ms, 7s850ms
- First Batch Requested: 8s094ms, 9s850ms, 10s150ms, 11s850ms
- First Batch Returned: ... ... ... ... ... ...
- ... ... ... ... ... ... ... ...
In case of extended verbosity all instance timestamps are printed along
with the summary.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile.cc
1 file changed, 134 insertions(+), 33 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/1
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 1
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Tests:
For test_profile_tool.py, updated the generated outputs for text
and JSON profiles.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Reviewed-on: http://gerrit.cloudera.org:8080/22245
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Riza Suminto <ri...@cloudera.com>
---
M be/src/util/runtime-profile.cc
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.pretty.json
M testdata/workloads/tpch/queries/runtime-profile-aggregated.test
9 files changed, 675 insertions(+), 1,689 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Riza Suminto: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 17
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Tests:
For test_profile_tool.py, updated the generated outputs for text
and JSON profiles.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile.cc
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.pretty.json
M testdata/workloads/tpch/queries/runtime-profile-aggregated.test
9 files changed, 675 insertions(+), 1,689 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/16
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 16
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 11: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 11
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 14:20:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 6:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/22245/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22245/5//COMMIT_MSG@19
PS5, Line 19:
: The su
> Spell it out in commit message: "in histogram format".
Done. I have updated the message now.
http://gerrit.cloudera.org:8080/#/c/22245/5//COMMIT_MSG@37
PS5, Line 37:
: Node Lifecycle Event Timeline:
: - Open Started: 5s885ms, 1s708ms, 3s434ms
> The histogram can be more self explanatory if Interval and Width is also pr
The formats present in PS 3 to 5 were to reduce the number of lines in the profile, as suggested during a sync.
I have implemented it in this format now. Thank you for the suggestion.
http://gerrit.cloudera.org:8080/#/c/22245/5/be/src/util/runtime-profile.cc
File be/src/util/runtime-profile.cc:
http://gerrit.cloudera.org:8080/#/c/22245/5/be/src/util/runtime-profile.cc@2755
PS5, Line 2755: {
> Unnecessary scope?
This is to make sure the copied agg_event_sequences and any other related temporary structures are out of scope once done.
http://gerrit.cloudera.org:8080/#/c/22245/5/be/src/util/runtime-profile.cc@2785
PS5, Line 2785: for (const auto& seq : agg_event_sequences) {
> Why is it need to hold lock again here if you have copy event_sequence_map_
Sorry, had forgotten to remove it, the entire reason for transformation was to avoid this.
http://gerrit.cloudera.org:8080/#/c/22245/5/be/src/util/runtime-profile.cc@2818
PS5, Line 2818: if (event_sequence_json.HasMember("events") &&
> I'm concern about the performance here because it converts into rapidjson f
'AggEventSequence' is the common struct/data structure that holds values efficiently.
For generating the summary, this can either directly be converted to text or to a volatile data structure in between such as JSON.
As this summarized structure may change, and to also cover the possibility of missing events and order, using the volatile JSON format is possibly the best choice.
http://gerrit.cloudera.org:8080/#/c/22245/5/be/src/util/runtime-profile.cc@2832
PS5, Line 2832: ostringstream interval_summary_l
> Reuse this stringstream by moving the declaration before L2816 loop.
Done
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 6
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Apr 2025 09:57:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt@5922
PS13, Line 5922: - Prepare Finished: 3.999ms
: - Open Finished: 3s819ms
: - First Batch Produced: 3s819ms
: - First Batch Sent: 3s819ms
: - ExecInternal Finished: 3s819ms
> Why the delta, number in parentheses, lost in this case?
Fragment Instance Lifecycle Event Timeline :
- Open Started: 2s850ms, 3s850ms, 4s150ms, 5s850ms
- Open Finished: 4s094ms, 5s850ms, 6s150ms, 7s850ms
- First Batch Requested: 8s094ms, 9s850ms, 10s150ms, 11s850ms
- First Batch Returned: ... ... ... ... ... ...
- ... ... ... ... ... ... ... ...
This fragment only has a single instance, when multiple instances are present in a fragment, deltas are not currently printed.
The related code begins from
https://gerrit.cloudera.org/#/c/22245/13/be/src/util/runtime-profile.cc#2888
To represent deltas or derived info, a separate case needs to be carved out for a fragment with a single instance.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 10:14:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 16:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/15/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt:
http://gerrit.cloudera.org:8080/#/c/22245/15/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt@741
PS15, Line 741: Count: 1
> Done
Thanks a lot for recognizing this, it seems we had missed it in JSON profile, better that this was found through text profile. Thank you again.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 16
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 May 2025 16:49:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13: -Code-Review
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json@3024
PS13, Line 3024: 1135987328.0, 1139987328.0, 0.0, 0.0, 1147987200.0
> It is probably better to leave as double and truncate the decimals with for
Yeah, lets switch back to double. I suspect some counters do need that high precisions.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 May 2025 14:54:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 16:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/15/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt:
http://gerrit.cloudera.org:8080/#/c/22245/15/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt@741
PS15, Line 741: Count: 1
> Sorry, I mean the other 4.
Done
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 16
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 May 2025 16:35:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Refactored small parts of info_strings representation into
PrettyPrintInfoStrings as per TODOs mentioned in IMPALA-9846.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
2 files changed, 150 insertions(+), 50 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/8
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 8
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for textual representation of event sequences present
in the aggregated profile(v2).
The patch also tries[WIP] to refactor parts of RuntimeProfileBase
into RuntimeProfile(i.e. info_strings_) as proposed during implementation
of profile V2 in IMPALA-9846.
With the format present in V1 and V2, it becomes difficult to analyze
an event's timestamps across instances.
The event sequences are now displayed in the following comprehensive
format, which would help with an easier view for skew analysis among
other possible use cases.
We achieve this by reusing the generation of aggregated event sequences
within the profile V2, as the extensive considerations of missing events
is already implemented there.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (1s140ms):
Instances - Count: 12, Min: 931.215ms, Avg: 260.020ms, Max: 1s140ms
| O . . . O |
| 8 - - - 4 |
- Open Finished (1s426ms):
Instances - Count: 12, Min: 1s134ms, Avg: 422.490ms, Max: 1s426ms
| O O O . O |
| 2 3 4 - 3 |
- First Batch Requested (1s426ms):
Instances - Count: 12, Min: 1s134ms, Avg: 422.498ms, Max: 1s426ms
| O O O . O |
| 2 3 4 - 3 |
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 727.203ms
- Open Finished: 4s881ms
- First Batch Requested: 4s881ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
2 files changed, 140 insertions(+), 50 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/3
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 3
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18876/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 8
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Apr 2025 12:45:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/22245/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22245/6//COMMIT_MSG@33
PS6, Line 33: HistogramInterval: 2s312ms, 2s826ms, 3s339ms, 3s853ms, 4s366ms
New format looks good but there isn't much value in printing HistogramInterval and HistogramWidth in this case. They can be calculated from Min/Max if needed
http://gerrit.cloudera.org:8080/#/c/22245/6/be/src/util/runtime-profile.cc
File be/src/util/runtime-profile.cc:
http://gerrit.cloudera.org:8080/#/c/22245/6/be/src/util/runtime-profile.cc@2870
PS6, Line 2870: stream << "HistogramInterval: " << interval_summary_line.str() << "\n";
Remove HistogramInterval and HistogramWidth.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 6
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Apr 2025 13:26:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19189/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 16
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 May 2025 17:01:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 16:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11879/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 16
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 May 2025 16:36:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
HistogramInterval: 2s312ms, 2s826ms, 3s339ms, 3s853ms, 4s366ms
HistogramWidth: 513.578ms
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Refactored small parts of info_strings representation into
PrettyPrintInfoStrings as per TODOs mentioned in IMPALA-9846.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
2 files changed, 162 insertions(+), 50 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/6
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 6
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 2:
I will try highlighting instances with unreported events in future patchsets.
Also in another change, will try implementing slow instance identification during aggregated event sequence JSON generation.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 2
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Feb 2025 20:50:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18392/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 2
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Feb 2025 21:17:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19113/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 10
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 May 2025 22:11:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Tests:
For test_profile_tool.py, updated the generated outputs for text
and JSON profiles.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.pretty.json
11 files changed, 691 insertions(+), 1,764 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/11
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 11
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 16: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 16
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 May 2025 22:49:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt@5922
PS13, Line 5922: - Prepare Finished: 3.999ms
: - Open Finished: 3s819ms
: - First Batch Produced: 3s819ms
: - First Batch Sent: 3s819ms
: - ExecInternal Finished: 3s819ms
> Fragment Instance Lifecycle Event Timeline :
Previously, all the instances and their timestamps were ignored, not being printed at all.
Only a single/first instance was being represented with deltas.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 10:17:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation of aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation of aggregate event sequences
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17735/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 1
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Dec 2024 15:19:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Refactored small parts of info_strings representation into
PrettyPrintInfoStrings as per TODOs mentioned in IMPALA-9846.
Tests:
For test_profile_tool.py, updated the generated outputs for text
and JSON profiles.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.pretty.json
11 files changed, 714 insertions(+), 1,791 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/9
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 9
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19104/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 9
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 May 2025 09:27:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18713/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 3
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Mar 2025 14:15:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19152/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 09:09:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt@5922
PS13, Line 5922: - Prepare Finished: 3.999ms
: - Open Finished: 3s819ms
: - First Batch Produced: 3s819ms
: - First Batch Sent: 3s819ms
: - ExecInternal Finished: 3s819ms
> Previously, all the instances and their timestamps were ignored, not being
Ack.
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json@3024
PS13, Line 3024: 1135987328.0, 1139987328.0, 0.0, 0.0, 1147987200.0
Why the average changes here? Is this precision issue that can be inconsistent on each test run?
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 10:23:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt@5922
PS13, Line 5922: - Prepare Finished: 3.999ms
: - Open Finished: 3s819ms
: - First Batch Produced: 3s819ms
: - First Batch Sent: 3s819ms
: - ExecInternal Finished: 3s819ms
> Previously, all the instances and their timestamps were ignored, not being
https://gerrit.cloudera.org/#/c/22245/13/be/src/util/runtime-profile.cc@2888
Fragment Instance Lifecycle Event Timeline[0]: 1s159ms
- Prepare Finished: 0.000ns (0.000ns)
- Open Finished: 1s147ms (1s147ms)
- First Batch Produced: 1s147ms (0.000ns)
- First Batch Sent: 1s147ms (0.000ns)
- ExecInternal Finished: 1s159ms (11.999ms)
The following were not being represented....
Fragment Instance Lifecycle Event Timeline[1]:
Fragment Instance Lifecycle Event Timeline[2]:
.
.
.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 10:21:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Tests:
For test_profile_tool.py, updated the generated outputs for text
and JSON profiles.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile.cc
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.pretty.json
8 files changed, 674 insertions(+), 1,688 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/14
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 14
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation of aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation of aggregate event sequences
......................................................................
Patch Set 1:
After completion of the comprehensive format, the tests and indentation can be added.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 1
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Dec 2024 14:54:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Refactored small parts of info_strings representation into
PrettyPrintInfoStrings as per TODOs mentioned in IMPALA-9846.
Tests:
For test_profile_tool.py, updated the generated outputs for text
and JSON profiles.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.pretty.json
11 files changed, 696 insertions(+), 1,773 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/10
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 10
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 12: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11862/
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 12
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 01:25:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13:
(1 comment)
Thank you for the comments.
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json@3024
PS13, Line 3024: 1135987328.0, 1139987328.0, 0.0, 0.0, 1147987200.0
> Why the average changes here? Is this precision issue that can be inconsist
In the text profile and JSON's visual representations, as 3 digits after milliseconds were being represented mostly, I had switched from double to float for the averaged timestamps.
If the precision of 50 nanoseconds is necessary, for the averaged timestamps, I can switch back to the representation with the double. As these were averaged timestamps, I was thinking this precision is sufficient.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 10:40:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json@3024
PS13, Line 3024: 1135987328.0, 1139987328.0, 0.0, 0.0, 1147987200.0
> Thank you for the comments. Please let me know, if I should switch back to
The values are consistent with each run, the precision was changed from double to float.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 10:44:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/15/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt:
http://gerrit.cloudera.org:8080/#/c/22245/15/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt@741
PS15, Line 741: Count: 8
Why this is only 8? Where are the other 3?
Are the missing 3 simply not exist in EventTimeline, or it is ignored because the value is 0.000ns?
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 15
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 May 2025 23:04:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 10:
The generated profiles have been updated again, after rebasing on top of IMPALA-13487(newly added profile counters).
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 10
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 May 2025 22:04:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18738/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 5
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Mar 2025 14:16:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 16: Code-Review+2
LGTM. Thanks for working on this!
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 16
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 14 May 2025 00:20:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 9: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11837/
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 9
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 May 2025 16:04:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11840/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 10
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 May 2025 22:05:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 7:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/18874/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 7
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Apr 2025 11:53:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for textual representation of event sequences present
in the aggregated profile(v2).
The patch also tries[WIP] to refactor parts of RuntimeProfileBase
into RuntimeProfile(i.e. info_strings_) as proposed during implementation
of profile V2 in IMPALA-9846.
With the format present in V1 and V2, it becomes difficult to analyze
an event's timestamps across instances.
The event sequences are now displayed in the following comprehensive
format, which would help with an easier view for skew analysis among
other possible use cases.
Example, Case 1 - Multiple spans
As instances exceeded the limit, aggregated into spans.
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 3
#instances = 12
Node Lifecycle Event Timeline Summary :
- Open Started:
Span #Inst Min Avg Max
33% 8 1s198ms 1s363ms 1s529ms
100% 4 4s197ms 4s197ms 4s197ms
Example, Case 2 - Single Span
As instances exceeded the limit, aggregated into spans.
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 1
#instances = 12
Node Lifecycle Event Timeline Summary :
- Open Started:
Span #Inst Min Avg Max
100% 3 2s213ms 4s410ms 6s810ms
Example, Case 3 - Verbose timestamps
As instances did not exceed the limit, no aggregation.
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 3
#instances = 3
Node Lifecycle Event Timeline:
- Open Started: 4s383ms, 1s895ms, 2s114ms
- Open Finished: 4s383ms, 1s895ms, 2s114ms
- First Batch Requested: 4s383ms, 1s895ms, 2s114ms
- First Batch Returned: 4s600ms, 4s012ms, 3s576ms
- Last Batch Returned: 8s792ms, 4s012ms, 3s576ms
- Closed: 8s793ms, 8s775ms, 8s769ms
Example, Case 4 - Verbose timestamps, no limit
As there is no limit, no aggregation is performed.
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 0
#instances = 3
Node Lifecycle Event Timeline:
- Open Started: 4s383ms, 1s895ms, 2s114ms
- Open Finished: 4s383ms, 1s895ms, 2s114ms
- First Batch Requested: 4s383ms, 1s895ms, 2s114ms
- First Batch Returned: 4s600ms, 4s012ms, 3s576ms
- Last Batch Returned: 8s792ms, 4s012ms, 3s576ms
- Closed: 8s793ms, 8s775ms, 8s769ms
We achieve this by reusing the generation of aggregated event sequences
within the profile V2, as the extensive considerations of missing events
is already implemented there.
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile.cc
1 file changed, 132 insertions(+), 42 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/2
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 2
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 4:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22245/4/be/src/util/runtime-profile.cc
File be/src/util/runtime-profile.cc:
http://gerrit.cloudera.org:8080/#/c/22245/4/be/src/util/runtime-profile.cc@2858
PS4, Line 2858: stream << prefix << " " << "| " << symbolic_summary_line.str() << "|\n";
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/22245/4/be/src/util/runtime-profile.cc@2859
PS4, Line 2859: stream << prefix << " " << "| " << count_summary_line.str() << "|\n" ;
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 4
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Mar 2025 13:54:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18714/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 4
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Mar 2025 14:16:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 5:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/22245/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22245/5//COMMIT_MSG@19
PS5, Line 19: in the following comprehensive
: format
Spell it out in commit message: "in histogram format".
Please also explain in commit message how the histogram is grouped.
http://gerrit.cloudera.org:8080/#/c/22245/5//COMMIT_MSG@37
PS5, Line 37: - Open Finished (1s325ms):
: Min: 649.730ms, Avg: 1s000ms, Max: 1s325ms
: Histogram: [ 4 0 4 0 4 ], #Inst. Count: 12
The histogram can be more self explanatory if Interval and Width is also printed.
https://en.wikipedia.org/wiki/Histogram
Maybe something like this instead:
- Open Finished (1s325ms):
Min: 649.730ms, Avg: 1s000ms, Max: 1s325ms, Count: 12
HistogramCount: 4, 0, 4, 0, 4
HistogramInterval: 649.730ms, 784.116ms, 919.076ms, 1.054s, 1.188s
HistogramWidth: 135.054ms
http://gerrit.cloudera.org:8080/#/c/22245/5/be/src/util/runtime-profile.cc
File be/src/util/runtime-profile.cc:
http://gerrit.cloudera.org:8080/#/c/22245/5/be/src/util/runtime-profile.cc@2755
PS5, Line 2755: {
Unnecessary scope?
http://gerrit.cloudera.org:8080/#/c/22245/5/be/src/util/runtime-profile.cc@2785
PS5, Line 2785: lock_guard<SpinLock> l(event_sequence_lock_);
Why is it need to hold lock again here if you have copy event_sequence_map_ to agg_event_sequences?
Perhaps, you want to hold this lock for the whole L2755 to L2873?
http://gerrit.cloudera.org:8080/#/c/22245/5/be/src/util/runtime-profile.cc@2818
PS5, Line 2818: agg_event_sequence.ToJson(event_sequence_json, &doc);
I'm concern about the performance here because it converts into rapidjson first.
Please consider creating common struct/data structure that is reusable by this method and AggEventSequence.ToJson()?
http://gerrit.cloudera.org:8080/#/c/22245/5/be/src/util/runtime-profile.cc@2832
PS5, Line 2832: stringstream count_summary_line;
Reuse this stringstream by moving the declaration before L2816 loop.
It will be slightly more memory efficient.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 5
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Mar 2025 19:56:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation of aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation of aggregate event sequences
......................................................................
Patch Set 1:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22245/1/be/src/util/runtime-profile.cc
File be/src/util/runtime-profile.cc:
http://gerrit.cloudera.org:8080/#/c/22245/1/be/src/util/runtime-profile.cc@2787
PS1, Line 2787: [](const std::pair<string, AggEventSequence>& pair) { return pair.second; });
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/22245/1/be/src/util/runtime-profile.cc@2810
PS1, Line 2810: stream << prefix << " Div. Min Max Avg #Inst. Count\n";
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/22245/1/be/src/util/runtime-profile.cc@2852
PS1, Line 2852: stream << PrettyPrinter::Print(ts_list[ts_count].GetInt64(), TUnit::TIME_NS) << "\n";
line too long (97 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 1
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Dec 2024 14:54:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11862/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 12
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 19:09:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13:
(1 comment)
Thank you for the comments.
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json@3024
PS13, Line 3024: 1135987328.0, 1139987328.0, 0.0, 0.0, 1147987200.0
> In the text profile and JSON's visual representations, as 3 digits after mi
Thank you for the comments. Please let me know, if I should switch back to double.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 10:42:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json@3024
PS13, Line 3024: 1135987328.0, 1139987328.0, 0.0, 0.0, 1147987200.0
> The values are consistent with each run, the precision was changed from dou
It is probably better to leave as double and truncate the decimals with formatting. Float can be imprecise for large numbers.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 11:42:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 9:
(1 comment)
Thank you for reminding me, I have updated the generated profiles now.
http://gerrit.cloudera.org:8080/#/c/22245/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22245/8//COMMIT_MSG@20
PS8, Line 20: The summary generated from aggregated instance-level timestamps
: (i.e. IMPALA-13304) is used to achieve this within the profile V2,
: which covers the possbility of missing events.
> Is this impact only JSON format or TEXT as well?
Done.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 9
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 May 2025 09:00:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 15: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11877/
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 15
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 May 2025 04:21:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Refactored small parts of info_strings representation into
PrettyPrintInfoStrings as per TODOs mentioned in IMPALA-9846.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
2 files changed, 151 insertions(+), 50 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/7
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 7
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 8: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 8
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Apr 2025 14:19:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for textual representation of event sequences present
in the aggregated profile(v2).
The patch also tries[WIP] to refactor parts of RuntimeProfileBase
into RuntimeProfile(i.e. info_strings_) as proposed during implementation
of profile V2 in IMPALA-9846.
With the format present in V1 and V2, it becomes difficult to analyze
an event's timestamps across instances.
The event sequences are now displayed in the following comprehensive
format, which would help with an easier view for skew analysis among
other possible use cases.
We achieve this by reusing the generation of aggregated event sequences
within the profile V2, as the extensive considerations of missing events
is already implemented there.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (1s140ms):
Instances - Count: 12, Min: 931.215ms, Avg: 260.020ms, Max: 1s140ms
| O . . . O |
| 8 - - - 4 |
- Open Finished (1s426ms):
Instances - Count: 12, Min: 1s134ms, Avg: 422.490ms, Max: 1s426ms
| O O O . O |
| 2 3 4 - 3 |
- First Batch Requested (1s426ms):
Instances - Count: 12, Min: 1s134ms, Avg: 422.498ms, Max: 1s426ms
| O O O . O |
| 2 3 4 - 3 |
...
- Last Batch Returned (3s408ms):
Instances - Count: 12, Min: 3s106ms, Avg: 1s358ms, Max: 3s408ms
| O x x O x |
| 7 1 1 2 1 |
Here, x - single node, O - group of nodes
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 727.203ms
- Open Finished: 4s881ms
- First Batch Requested: 4s881ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
2 files changed, 140 insertions(+), 50 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/4
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 4
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18847/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 6
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Apr 2025 10:22:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 8:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/22245/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22245/5//COMMIT_MSG@19
PS5, Line 19:
: The su
> Done. I have updated the message now.
Done
http://gerrit.cloudera.org:8080/#/c/22245/5//COMMIT_MSG@37
PS5, Line 37: - Open Started: 5s885ms, 1s708ms, 3s434ms
: - Open Finished: 5s885ms, 1s708ms, 3s435ms
: - First Batch Requested: 5s885ms, 1s708ms, 3s435ms
> The formats present in PS 3 to 5 were to reduce the number of lines in the
Done
http://gerrit.cloudera.org:8080/#/c/22245/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22245/8//COMMIT_MSG@20
PS8, Line 20: The summary generated from aggregated instance-level timestamps
: (i.e. IMPALA-13304) is used to achieve this within the profile V2,
: which covers the possbility of missing events.
Is this impact only JSON format or TEXT as well?
Please check if these test data needs an update.
./impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt
./impala-profiles/impala_profile_log_tpcds_compute_stats_v2
./impala-profiles/impala_profile_log_tpcds_compute_stats_v2.expected.json
./impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.pretty.json
./impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.txt
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 8
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Apr 2025 18:56:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11837/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 9
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 May 2025 10:32:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 10: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11840/
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 10
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 May 2025 04:17:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 3:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22245/3/be/src/util/runtime-profile.cc
File be/src/util/runtime-profile.cc:
http://gerrit.cloudera.org:8080/#/c/22245/3/be/src/util/runtime-profile.cc@2858
PS3, Line 2858: stream << prefix << " " << "| " << symbolic_summary_line.str() << "|\n";
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/22245/3/be/src/util/runtime-profile.cc@2859
PS3, Line 2859: stream << prefix << " " << "| " << count_summary_line.str() << "|\n" ;
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 3
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Mar 2025 13:51:11 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for textual representation of event sequences present
in the aggregated profile(v2).
The patch also tries[WIP] to refactor parts of RuntimeProfileBase
into RuntimeProfile(i.e. info_strings_) as proposed during implementation
of profile V2 in IMPALA-9846.
With the format present in V1 and V2, it becomes difficult to analyze
an event's timestamps across instances.
The event sequences are now displayed in the following comprehensive
format, which would help with an easier view for skew analysis among
other possible use cases.
We achieve this by reusing the generation of aggregated event sequences
within the profile V2, as the extensive considerations of missing events
is already implemented there.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (1s324ms):
Min: 649.156ms, Avg: 999.670ms, Max: 1s324ms
Histogram: [ 4 0 4 0 4 ], #Inst. Count: 12
- Open Finished (1s325ms):
Min: 649.730ms, Avg: 1s000ms, Max: 1s325ms
Histogram: [ 4 0 4 0 4 ], #Inst. Count: 12
- First Batch Requested (1s325ms):
Min: 989.065ms, Avg: 1s113ms, Max: 1s325ms
Histogram: [ 8 0 0 0 4 ], #Inst. Count: 12
- First Batch Returned (1s361ms):
Min: 1s260ms, Avg: 1s321ms, Max: 1s361ms
Histogram: [ 4 0 0 2 6 ], #Inst. Count: 12
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 727.203ms
- Open Finished: 4s881ms
- First Batch Requested: 4s881ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
2 files changed, 130 insertions(+), 50 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/5
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 5
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22245/4//COMMIT_MSG@36
PS4, Line 36: | O . . . O |
: | 8 - - - 4 |
I don't immediately understand what this diagram means.
For text, perhaps just printing percentile is sufficient, like this:
https://github.com/apache/impala/blob/8101b0784ccd1b87b85c1a055e92866614eb36a8/be/src/util/runtime-profile.cc#L2306-L2347
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 4
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Mar 2025 16:04:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 7:
(2 comments)
Thank you for the comments
http://gerrit.cloudera.org:8080/#/c/22245/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22245/6//COMMIT_MSG@33
PS6, Line 33:
> New format looks good but there isn't much value in printing HistogramInter
Done
http://gerrit.cloudera.org:8080/#/c/22245/6/be/src/util/runtime-profile.cc
File be/src/util/runtime-profile.cc:
http://gerrit.cloudera.org:8080/#/c/22245/6/be/src/util/runtime-profile.cc@2870
PS6, Line 2870: // When number of instances == 0
> Remove HistogramInterval and HistogramWidth.
Done
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 7
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Apr 2025 11:25:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 11:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19143/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 11
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 14:24:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/15/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt:
http://gerrit.cloudera.org:8080/#/c/22245/15/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt@741
PS15, Line 741: Count: 8
> Why this is only 8? Where are the other 3?
Sorry, I mean the other 4.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 15
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 May 2025 23:25:13 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19183/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 14
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 May 2025 22:28:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json@3024
PS13, Line 3024: 1135987390.0, 1139987346.0, 0.0, 0.0, 1147987257.0
> Yeah, lets switch back to double. I suspect some counters do need that high
Done.
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 15
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 May 2025 22:02:11 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11877/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 15
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 May 2025 22:00:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
File testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt:
http://gerrit.cloudera.org:8080/#/c/22245/13/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt@5922
PS13, Line 5922: - Prepare Finished: 3.999ms
: - Open Finished: 3s819ms
: - First Batch Produced: 3s819ms
: - First Batch Sent: 3s819ms
: - ExecInternal Finished: 3s819ms
Why the delta, number in parentheses, lost in this case?
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 May 2025 09:25:47 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13624: Implement textual representation for aggregate event sequences
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/22245 )
Change subject: IMPALA-13624: Implement textual representation for aggregate event sequences
......................................................................
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Tests:
For test_profile_tool.py, updated the generated outputs for text
and JSON profiles.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2.expected.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.pretty.json
9 files changed, 689 insertions(+), 1,703 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/22245/13
--
To view, visit http://gerrit.cloudera.org:8080/22245
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Gerrit-Change-Number: 22245
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>