You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2023/09/19 18:51:33 UTC

[GitHub] [pinot] ArnavBalyan opened a new pull request, #11625: Force FileChannel to commit data to disk

ArnavBalyan opened a new pull request, #11625:
URL: https://github.com/apache/pinot/pull/11625

    - We have FileChannel in certain places but don't force which can cause data not to be written to disk.
    - Added force to make sure we don't have data remaining in OS I/O buffers.
    - `bugfix` for #11368 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ankitsultana commented on pull request #11625: Force FileChannel to commit data to disk

Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on PR #11625:
URL: https://github.com/apache/pinot/pull/11625#issuecomment-1728331278

   @Jackie-Jiang : can you help run unit-tests on this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #11625: Force FileChannel to commit data to disk

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11625:
URL: https://github.com/apache/pinot/pull/11625#issuecomment-1730096165

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#11625](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (9a95c3a) into [master](https://app.codecov.io/gh/apache/pinot/commit/fa514773508fd6e188d74ae63c597b4ff01c1401?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (fa51477) will **decrease** coverage by `0.13%`.
   > Report is 12 commits behind head on master.
   > The diff coverage is `100.00%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #11625      +/-   ##
   ============================================
   - Coverage     63.08%   62.96%   -0.13%     
   + Complexity     1106     1105       -1     
   ============================================
     Files          2326     2326              
     Lines        124942   124945       +3     
     Branches      19147    19147              
   ============================================
   - Hits          78821    78667     -154     
   - Misses        40499    40648     +149     
   - Partials       5622     5630       +8     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.49% <0.00%> (-48.56%)` | :arrow_down: |
   | [java-17](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.49% <0.00%> (-48.42%)` | :arrow_down: |
   | [java-20](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.92% <100.00%> (+<0.01%)` | :arrow_up: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.96% <100.00%> (-0.13%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.95% <100.00%> (-0.13%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.09% <100.00%> (-0.22%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11625/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.53% <0.00%> (+0.03%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...reator/impl/inv/geospatial/BaseH3IndexCreator.java](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9pbnYvZ2Vvc3BhdGlhbC9CYXNlSDNJbmRleENyZWF0b3IuamF2YQ==) | `96.49% <100.00%> (+0.06%)` | :arrow_up: |
   | [...nt/creator/impl/inv/json/BaseJsonIndexCreator.java](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9pbnYvanNvbi9CYXNlSnNvbkluZGV4Q3JlYXRvci5qYXZh) | `93.84% <100.00%> (+0.09%)` | :arrow_up: |
   | [...al/startree/v2/builder/StarTreeIndexSeparator.java](https://app.codecov.io/gh/apache/pinot/pull/11625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS92Mi9idWlsZGVyL1N0YXJUcmVlSW5kZXhTZXBhcmF0b3IuamF2YQ==) | `86.53% <100.00%> (+0.26%)` | :arrow_up: |
   
   ... and [28 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11625/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ArnavBalyan commented on a diff in pull request #11625: Force FileChannel to commit data to disk

Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #11625:
URL: https://github.com/apache/pinot/pull/11625#discussion_r1334697119


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/startree/v2/builder/StarTreeIndexSeparator.java:
##########
@@ -143,6 +143,7 @@ private void writeIndexToFile(File destFile, StarTreeIndexMapUtils.IndexValue va
       throws IOException {
     try (FileChannel dest = new RandomAccessFile(destFile, "rw").getChannel()) {
       org.apache.pinot.common.utils.FileUtils.transferBytes(_indexFileChannel, value._offset, value._size, dest);
+      _indexFileChannel.force(true);

Review Comment:
   Done thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11625: Force FileChannel to commit data to disk

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11625:
URL: https://github.com/apache/pinot/pull/11625#discussion_r1333405351


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/startree/v2/builder/StarTreeIndexSeparator.java:
##########
@@ -143,6 +143,7 @@ private void writeIndexToFile(File destFile, StarTreeIndexMapUtils.IndexValue va
       throws IOException {
     try (FileChannel dest = new RandomAccessFile(destFile, "rw").getChannel()) {
       org.apache.pinot.common.utils.FileUtils.transferBytes(_indexFileChannel, value._offset, value._size, dest);
+      _indexFileChannel.force(true);

Review Comment:
   `_indexFileChannel` is read only. Do you intend to force `dest` here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ArnavBalyan commented on pull request #11625: Force FileChannel to commit data to disk

Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #11625:
URL: https://github.com/apache/pinot/pull/11625#issuecomment-1726331022

   cc @Jackie-Jiang and @ankitsultana if you could please review thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang merged pull request #11625: Force FileChannel to commit data to disk

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11625:
URL: https://github.com/apache/pinot/pull/11625


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org