You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2023/09/19 18:51:33 UTC
[GitHub] [pinot] ArnavBalyan opened a new pull request, #11625: Force FileChannel to commit data to disk
ArnavBalyan opened a new pull request, #11625:
URL: https://github.com/apache/pinot/pull/11625
- We have FileChannel in certain places but don't force which can cause data not to be written to disk.
- Added force to make sure we don't have data remaining in OS I/O buffers.
- `bugfix` for #11368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on pull request #11625: Force FileChannel to commit data to disk
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on PR #11625:
URL: https://github.com/apache/pinot/pull/11625#issuecomment-1728331278
@Jackie-Jiang : can you help run unit-tests on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #11625: Force FileChannel to commit data to disk
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11625:
URL: https://github.com/apache/pinot/pull/11625#issuecomment-1730096165
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11625](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (9a95c3a) into [master](https://app.codecov.io/gh/apache/pinot/commit/fa514773508fd6e188d74ae63c597b4ff01c1401?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (fa51477) will **decrease** coverage by `0.13%`.
> Report is 12 commits behind head on master.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #11625 +/- ##
============================================
- Coverage 63.08% 62.96% -0.13%
+ Complexity 1106 1105 -1
============================================
Files 2326 2326
Lines 124942 124945 +3
Branches 19147 19147
============================================
- Hits 78821 78667 -154
- Misses 40499 40648 +149
- Partials 5622 5630 +8
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.49% <0.00%> (-48.56%)` | :arrow_down: |
| [java-17](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.49% <0.00%> (-48.42%)` | :arrow_down: |
| [java-20](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.92% <100.00%> (+<0.01%)` | :arrow_up: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.96% <100.00%> (-0.13%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.95% <100.00%> (-0.13%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.09% <100.00%> (-0.22%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.53% <0.00%> (+0.03%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...reator/impl/inv/geospatial/BaseH3IndexCreator.java](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9pbnYvZ2Vvc3BhdGlhbC9CYXNlSDNJbmRleENyZWF0b3IuamF2YQ==) | `96.49% <100.00%> (+0.06%)` | :arrow_up: |
| [...nt/creator/impl/inv/json/BaseJsonIndexCreator.java](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9pbnYvanNvbi9CYXNlSnNvbkluZGV4Q3JlYXRvci5qYXZh) | `93.84% <100.00%> (+0.09%)` | :arrow_up: |
| [...al/startree/v2/builder/StarTreeIndexSeparator.java](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS92Mi9idWlsZGVyL1N0YXJUcmVlSW5kZXhTZXBhcmF0b3IuamF2YQ==) | `86.53% <100.00%> (+0.26%)` | :arrow_up: |
... and [28 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11625/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ArnavBalyan commented on a diff in pull request #11625: Force FileChannel to commit data to disk
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #11625:
URL: https://github.com/apache/pinot/pull/11625#discussion_r1334697119
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/startree/v2/builder/StarTreeIndexSeparator.java:
##########
@@ -143,6 +143,7 @@ private void writeIndexToFile(File destFile, StarTreeIndexMapUtils.IndexValue va
throws IOException {
try (FileChannel dest = new RandomAccessFile(destFile, "rw").getChannel()) {
org.apache.pinot.common.utils.FileUtils.transferBytes(_indexFileChannel, value._offset, value._size, dest);
+ _indexFileChannel.force(true);
Review Comment:
Done thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11625: Force FileChannel to commit data to disk
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11625:
URL: https://github.com/apache/pinot/pull/11625#discussion_r1333405351
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/startree/v2/builder/StarTreeIndexSeparator.java:
##########
@@ -143,6 +143,7 @@ private void writeIndexToFile(File destFile, StarTreeIndexMapUtils.IndexValue va
throws IOException {
try (FileChannel dest = new RandomAccessFile(destFile, "rw").getChannel()) {
org.apache.pinot.common.utils.FileUtils.transferBytes(_indexFileChannel, value._offset, value._size, dest);
+ _indexFileChannel.force(true);
Review Comment:
`_indexFileChannel` is read only. Do you intend to force `dest` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ArnavBalyan commented on pull request #11625: Force FileChannel to commit data to disk
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #11625:
URL: https://github.com/apache/pinot/pull/11625#issuecomment-1726331022
cc @Jackie-Jiang and @ankitsultana if you could please review thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #11625: Force FileChannel to commit data to disk
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11625:
URL: https://github.com/apache/pinot/pull/11625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org