You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "SHREELEKHYA GAMPA (Jira)" <ji...@apache.org> on 2021/03/03 15:18:00 UTC
[jira] [Updated] (CARBONDATA-4037) Improve the table status and
segment file writing
[ https://issues.apache.org/jira/browse/CARBONDATA-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
SHREELEKHYA GAMPA updated CARBONDATA-4037:
------------------------------------------
Description:
Currently, we update table status and segment files multiple times for a single iud/merge/compact operation and delete the index files immediately after merge. When concurrent queries are run, there may be situations like user query is trying to access the segment index files and they are not present, which is availability issue.
* To solve above issue, we can make mergeindex files generation mandatory and fail load/compaction if mergeindex fails. Then if merge index is success, update table status file and can delete index files immediately. However, in legacy stores when alter segment merge is called, after merge index success, do not delete index files immediately as it may cause issues for parallel queries.
was:
Currently, we update table status and segment files multiple times for a single iud/merge/compact operation and delete the index files immediately after merge. When concurrent queries are run, there may be situations like user query is trying to access the segment index files and they are not present, which is availability issue.
* Instead of deleting carbon index files immediately after merge, delete index files only when clean files command is executed and delete only those that have existed for more than 1 hour.
* Generate segment file after merge index and update table status at beginning and after merge index.
order:
create table status file => index files => merge index => generate segment file => update table status
> Improve the table status and segment file writing
> -------------------------------------------------
>
> Key: CARBONDATA-4037
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4037
> Project: CarbonData
> Issue Type: Improvement
> Reporter: SHREELEKHYA GAMPA
> Priority: Minor
> Time Spent: 15h 40m
> Remaining Estimate: 0h
>
> Currently, we update table status and segment files multiple times for a single iud/merge/compact operation and delete the index files immediately after merge. When concurrent queries are run, there may be situations like user query is trying to access the segment index files and they are not present, which is availability issue.
> * To solve above issue, we can make mergeindex files generation mandatory and fail load/compaction if mergeindex fails. Then if merge index is success, update table status file and can delete index files immediately. However, in legacy stores when alter segment merge is called, after merge index success, do not delete index files immediately as it may cause issues for parallel queries.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)