You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "Akash R Nilugal (JIRA)" <ji...@apache.org> on 2018/07/16 07:00:01 UTC
[jira] [Updated] (CARBONDATA-2021) when delete is success and
update is failed while writing status file then a stale carbon data file
is created.
[ https://issues.apache.org/jira/browse/CARBONDATA-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Akash R Nilugal updated CARBONDATA-2021:
----------------------------------------
Description:
when delete is success and update is failed while writing status file then a stale carbon data file is created.
so removing that file on clean up . and also not considering that one during query.
when the update operation is running and the user stops it abruptly,
then the carbon data file will be remained in the store .
so extra data is coming.
during the next update the clean up of the files need to be handled.
and in query also new data file should be excluded.
create table struct_bigint1(struct1 struct<bigint1:bigint,bigint2:bigint,bigint3:bigint>) stored by 'carbondata' TBLPROPERTIES ('SORT_SCOPE'='GLOBAL_SORT');
2: Load the data:
load data inpath 'hdfs://hacluster/user/adaptivedata/struct_bigint/BigintDeltaData2.txt' into table struct_bigint1 options ('delimiter'='|','fileheader'='struct1','complex_delimiter_level_1'=',','GLOBAL_SORT_PARTITIONS'='1');
32767,2147483647,9223372036854770000
32766,2147450880,9223372036854760000
was:
when delete is success and update is failed while writing status file then a stale carbon data file is created.
so removing that file on clean up . and also not considering that one during query.
when the update operation is running and the user stops it abruptly,
then the carbon data file will be remained in the store .
so extra data is coming.
during the next update the clean up of the files need to be handled.
and in query also new data file should be excluded.
> when delete is success and update is failed while writing status file then a stale carbon data file is created.
> ----------------------------------------------------------------------------------------------------------------
>
> Key: CARBONDATA-2021
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2021
> Project: CarbonData
> Issue Type: Bug
> Reporter: Akash R Nilugal
> Assignee: Akash R Nilugal
> Priority: Minor
> Fix For: 1.3.0
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> when delete is success and update is failed while writing status file then a stale carbon data file is created.
> so removing that file on clean up . and also not considering that one during query.
> when the update operation is running and the user stops it abruptly,
> then the carbon data file will be remained in the store .
> so extra data is coming.
> during the next update the clean up of the files need to be handled.
> and in query also new data file should be excluded.
>
> create table struct_bigint1(struct1 struct<bigint1:bigint,bigint2:bigint,bigint3:bigint>) stored by 'carbondata' TBLPROPERTIES ('SORT_SCOPE'='GLOBAL_SORT');
> 2: Load the data:
> load data inpath 'hdfs://hacluster/user/adaptivedata/struct_bigint/BigintDeltaData2.txt' into table struct_bigint1 options ('delimiter'='|','fileheader'='struct1','complex_delimiter_level_1'=',','GLOBAL_SORT_PARTITIONS'='1');
>
>
> 32767,2147483647,9223372036854770000
> 32766,2147450880,9223372036854760000
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)