You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/12/14 16:03:59 UTC

[GitHub] [hive] szlta opened a new pull request #1776: HIVE-24337: Cache delete delta files in LLAP cache

szlta opened a new pull request #1776:
URL: https://github.com/apache/hive/pull/1776


   Changes:
   - Added new method in LLAP IO API so that a cached LLAP record reader can be created for arbitrary files
   - Removed some unnecessary / inefficient code parts in LlapInputFormat and AcidInputFormat
   
   Functional testing: existing testing already covers this change:
   - TestVectorizedOrcAcidRowBatchReader unit test covers the code paths of this change when the feature is off (no LLAP present)
   - TestMiniLlapCliDriver/acid_direct_update_delete.q qtest covers the code paths where LLAP is present and ACID delete deltas are cached into LLAP IO (since the feature flag is by default turned on)
   
   Performance testing: used S3 to see performance gains on a small (6 rows) table read that had ~20 delete events.
   - query duration decreased from ~17s to ~7s


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] szlta commented on pull request #1776: HIVE-24337: Cache delete delta files in LLAP cache

Posted by GitBox <gi...@apache.org>.
szlta commented on pull request #1776:
URL: https://github.com/apache/hive/pull/1776#issuecomment-754720731


   Tested with hive.llap.io.cache.deletedeltas=metadata.
   All tests passed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] szlta commented on pull request #1776: HIVE-24337: Cache delete delta files in LLAP cache

Posted by GitBox <gi...@apache.org>.
szlta commented on pull request #1776:
URL: https://github.com/apache/hive/pull/1776#issuecomment-754542630


   Tested with hive.llap.io.cache.deletedeltas=all.
   Jenkins found 1 failed test:
   org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
   Re-run manually and passed, thus it is flaky.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] szlta merged pull request #1776: HIVE-24337: Cache delete delta files in LLAP cache

Posted by GitBox <gi...@apache.org>.
szlta merged pull request #1776:
URL: https://github.com/apache/hive/pull/1776


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] szlta commented on pull request #1776: HIVE-24337: Cache delete delta files in LLAP cache

Posted by GitBox <gi...@apache.org>.
szlta commented on pull request #1776:
URL: https://github.com/apache/hive/pull/1776#issuecomment-755194322


   Tested with hive.llap.io.cache.deletedeltas=none.
   All tests passed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org