You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "zzzzming95 (via GitHub)" <gi...@apache.org> on 2023/12/25 12:44:28 UTC
[PR] [KYUUBI #5911] Make `fetch_result_rows_rate` metric correct when result format is arrow [kyuubi]
zzzzming95 opened a new pull request, #5920:
URL: https://github.com/apache/kyuubi/pull/5920
# :mag: Description
## Issue References ๐
<!-- Append the issue number after #. If there is no issue for you to link create one or -->
<!-- If there are no issues to link, please provide details here. -->
This pull request fixes #5911 .
Make `fetch_result_rows_rate` metric correct when result format is arrow
## Describe Your Solution ๐ง
fetch_result_rows_rate metric wrong when set kyuubi.operation.result.format be arrow. This pr fix this issue by reading arrow header.
## Types of changes :bookmark:
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
select a table , `fetch_result_rows_rate` metric is wrong .
#### Behavior With This Pull Request :tada:
select a table , `fetch_result_rows_rate` metric will be correct .
#### Related Unit Tests
No, Manual testing.
---
# Checklists
## ๐ Author Self Checklist
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## ๐ Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5911] Make `fetch_result_rows_rate` metric correct when result format is arrow [kyuubi]
Posted by "zzzzming95 (via GitHub)" <gi...@apache.org>.
zzzzming95 commented on code in PR #5920:
URL: https://github.com/apache/kyuubi/pull/5920#discussion_r1436172115
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/BackendServiceMetric.scala:
##########
@@ -197,6 +217,24 @@ trait BackendServiceMetric extends BackendService {
case t: TI16Column => t.getValues.size()
case t: TBoolColumn => t.getValues.size()
case t: TByteColumn => t.getValues.size()
+ case t: TBinaryColumn
+ if resultFormat == "arrow" =>
+ t.getValues.asScala.map(listBuff => {
+ val batchBytes = listBuff.array()
+ val input = new ByteArrayInputStream(batchBytes)
+ val readChannel = new ReadChannel(Channels.newChannel(input))
Review Comment:
Let me see if there is a way to reduce system load.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5911] Make `fetch_result_rows_rate` metric correct when result format is arrow [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5920: [KYUUBI #5911] Make `fetch_result_rows_rate` metric correct when result format is arrow
URL: https://github.com/apache/kyuubi/pull/5920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5911] Make `fetch_result_rows_rate` metric correct when result format is arrow [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5920:
URL: https://github.com/apache/kyuubi/pull/5920#issuecomment-1869004939
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `4 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`86bec4d`)](https://app.codecov.io/gh/apache/kyuubi/commit/86bec4d914cc0de089b5183f87e3a45ade9d60e3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.60% compared to head [(`e2d76cf`)](https://app.codecov.io/gh/apache/kyuubi/pull/5920?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.55%.
| [Files](https://app.codecov.io/gh/apache/kyuubi/pull/5920?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...rg/apache/kyuubi/server/BackendServiceMetric.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5920?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvQmFja2VuZFNlcnZpY2VNZXRyaWMuc2NhbGE=) | 81.81% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/5920?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #5920 +/- ##
============================================
- Coverage 61.60% 61.55% -0.06%
Complexity 23 23
============================================
Files 616 616
Lines 36387 36416 +29
Branches 4978 4982 +4
============================================
- Hits 22418 22417 -1
- Misses 11557 11579 +22
- Partials 2412 2420 +8
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/5920?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5911] Make `fetch_result_rows_rate` metric correct when result format is arrow [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5920:
URL: https://github.com/apache/kyuubi/pull/5920#issuecomment-2799390732
Thanks for the PR! This PR is being closed due to inactivity. This isn't a judgement on the merit of the PR in any way. If this is still an issue with the latest version of Kyuubi, please reopen it and ask a committer to remove the Stale tag!
Thank you for using Kyuubi!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5911] Make `fetch_result_rows_rate` metric correct when result format is arrow [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5920:
URL: https://github.com/apache/kyuubi/pull/5920#discussion_r1436090692
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/BackendServiceMetric.scala:
##########
@@ -197,6 +217,24 @@ trait BackendServiceMetric extends BackendService {
case t: TI16Column => t.getValues.size()
case t: TBoolColumn => t.getValues.size()
case t: TByteColumn => t.getValues.size()
+ case t: TBinaryColumn
+ if resultFormat == "arrow" =>
+ t.getValues.asScala.map(listBuff => {
+ val batchBytes = listBuff.array()
+ val input = new ByteArrayInputStream(batchBytes)
+ val readChannel = new ReadChannel(Channels.newChannel(input))
Review Comment:
The CPU/memory overhead looks high, it's dangerous to do such thing on Kyuubi server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5911] Make `fetch_result_rows_rate` metric correct when result format is arrow [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5920:
URL: https://github.com/apache/kyuubi/pull/5920#discussion_r1436090692
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/BackendServiceMetric.scala:
##########
@@ -197,6 +217,24 @@ trait BackendServiceMetric extends BackendService {
case t: TI16Column => t.getValues.size()
case t: TBoolColumn => t.getValues.size()
case t: TByteColumn => t.getValues.size()
+ case t: TBinaryColumn
+ if resultFormat == "arrow" =>
+ t.getValues.asScala.map(listBuff => {
+ val batchBytes = listBuff.array()
+ val input = new ByteArrayInputStream(batchBytes)
+ val readChannel = new ReadChannel(Channels.newChannel(input))
Review Comment:
The CPU/memory overhead looks high, it's dangerous do such thing on Kyuubi server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org