You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ch...@apache.org on 2021/12/20 14:55:44 UTC
[carbondata] branch master updated (0ced3c8 -> f266a73)
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git.
from 0ced3c8 [maven-release-plugin] Reverted the pom changes to 2.2.0-SNAPSHOT
new 81c2e29 Supplementary information for add segment syntax .
new 598d1ce Add FAQ How to manage mix file format in carbondata table.
new 885a21c FAQ: carbon rename to carbondata
new 69ab06c remove add segment refer
new 7af81ad remove useless numerical value , revert some typo issues
new 42d59be Revert "remove useless numerical value , revert some typo issues"
new 580f7f6 Revert "remove add segment refer"
new 341f1bf Revert "FAQ: carbon rename to carbondata"
new ce5747d Revert "Add FAQ How to manage mix file format in carbondata table."
new f544e59 Revert "Supplementary information for add segment syntax ."
new c29fee2 Supplementary information for add segment syntax
new 379f5ad Update docs/addsegment-guide.md
new a1b6d99 Update docs/addsegment-guide.md
new f266a73 [CARBONDATA-4315] Supplementary information for add segment syntax This closes #4239
The 14 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/addsegment-guide.md | 134 +++++++++++++++++++++++++++++++++++++++++++++--
docs/faq.md | 5 ++
2 files changed, 136 insertions(+), 3 deletions(-)
[carbondata] 11/14: Supplementary information for add segment syntax
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit c29fee2d488b483a9c08db4fff14beb1f7df781d
Author: bieremayi <li...@sina.com>
AuthorDate: Tue Dec 7 17:18:58 2021 +0800
Supplementary information for add segment syntax
---
docs/addsegment-guide.md | 134 +++++++++++++++++++++++++++++++++++++++++++++--
docs/faq.md | 5 ++
2 files changed, 136 insertions(+), 3 deletions(-)
diff --git a/docs/addsegment-guide.md b/docs/addsegment-guide.md
index 78b15e6..619293f 100644
--- a/docs/addsegment-guide.md
+++ b/docs/addsegment-guide.md
@@ -27,16 +27,144 @@ Heterogeneous format segments aims to solve this problem by avoiding data conver
### Add segment with path and format
Users can add the existing data as a segment to the carbon table provided the schema of the data
and the carbon table should be the same.
+
+ Syntax
+
+ ```
+ ALTER TABLE [db_name.]table_name ADD SEGMENT OPTIONS(property_name=property_value, ...)
+ ```
+
+**Supported properties:**
+
+| Property | Description |
+| ------------------------------------------------------------ | ------------------------------------------------------------ |
+| [PATH](#path) | User external old table path |
+| [FORMAT](#format) | User external old table file format |
+| [PARTITION](#partition) | Partition info for partition table , should be form of "a:int, b:string" |
+
+
+-
+ You can use the following options to add segment:
+
+ - ##### PATH:
+ User old table path.
+
+ ```
+ OPTIONS('PATH'='hdfs://usr/oldtable')
+ ```
+
+ - ##### FORMAT:
+ User old table file format. eg : parquet, orc
+
+ ```
+ OPTIONS('FORMAT'='parquet')
+ ```
+ - ##### PARTITION:
+ Partition info for partition table , should be form of "a:int, b:string"
+
+ ```
+ OPTIONS('PARTITION'='a:int, b:string')
+ ```
+
-```
-alter table table_name add segment options ('path'= 'hdfs://usr/oldtable','format'='parquet')
-```
In the above command user can add the existing data to the carbon table as a new segment and also
can provide the data format.
During add segment, it will infer the schema from data and validates the schema against the carbon table.
If the schema doesn’t match it throws an exception.
+**Example:**
+
+Exist old hive partition table , stored as orc or parquet file format:
+
+
+```sql
+CREATE TABLE default.log_parquet_par (
+ id BIGINT,
+ event_time BIGINT,
+ ip STRING
+)PARTITIONED BY (
+ day INT,
+ hour INT,
+ type INT
+)
+STORED AS parquet
+LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par';
+```
+
+Parquet File Location :
+
+```
+/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0
+/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=1
+```
+
+
+```sql
+CREATE TABLE default.log_orc_par (
+ id BIGINT,
+ event_time BIGINT,
+ ip STRING
+)PARTITIONED BY (
+ day INT,
+ hour INT,
+ type INT
+)
+STORED AS orc
+LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_orc_par';
+```
+
+Orc File Location :
+
+```
+/user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=0
+/user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=1
+```
+
+**Steps:**
+
+step1: Create carbondata format table with the same schema.
+
+```sql
+CREATE TABLE default.log_carbon_par (
+ id BIGINT,
+ event_time BIGINT,
+ ip STRING
+)PARTITIONED BY (
+ day INT,
+ hour INT,
+ type INT
+)
+STORED AS carbondata
+LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_carbon_par';
+```
+
+step2: Execute add segment sql.
+
+|SQL|Is Success|Error Message|
+|:---|:---|:---|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/','format'='parquet','partition'='day:int,hour:int,type:int');|Yes|/|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_orc_par/','format'='orc','partition'='day:int,hour:int,type:int');|Yes|/|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day=20211123 and hour=12 and type=0');|No|Error in query: invalid partition option: Map(path -> hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0, format -> parquet, partition -> day=20211123 and hour=12 and type=0)|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day=20211123,hour=12,type=0');|No|Error in query: invalid partition option: Map(path -> hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0, format -> parquet, partition -> day=20211123,hour=12,type=0)|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet');|No|Error in query: partition option is required when adding segment to partition table|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day:int,hour:int,type:int');|No|ERROR SparkSQLDriver: Failed in [alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day:int,hour:int,type:int')]|
+
+step3: Result check.
+
+```sql
+SHOW SEGMENTS FOR TABLE default.log_carbon_par;
+```
+
+| ID | Status | Load Start Time | Load Time Taken | Partition | Data Size | Index Size | File Format |
+|:---|:---|:---|:---|:---|:---|:---|:---|
+|4 |Success |2021-11-29 17:59:40.819 |7.026S |{day=20211123,hour=12,type=1}, ... |xxx| xxx |columnar_v3|
+|3 |Success |2021-11-29 16:34:28.106 |0.418S |{day=20211123,hour=12,type=0} |xxx |NA | orc|
+|2 |Success |2021-11-29 16:34:27.733 |0.222S |{day=20211123,hour=12,type=1} |xxx |NA | orc|
+|1 |Success |2021-11-29 16:30:17.207 |0.275S |{day=20211123,hour=12,type=0} |xxx |NA | parquet|
+|0 |Success |2021-11-29 16:30:16.48 |0.452S |{day=20211123,hour=12,type=1} |xxx |NA | parquet|
+
+
### Changes to tablestatus file
Carbon adds the new segment by adding segment information to tablestatus file. In order to add the path and format information to tablestatus, we are going to add `segmentPath` and `format` to the tablestatus file.
And any extra `options` will be added to the segment file.
diff --git a/docs/faq.md b/docs/faq.md
index a330e24..977fb7e 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -29,6 +29,7 @@
* [Why different time zone result for select query output when query SDK writer output?](#why-different-time-zone-result-for-select-query-output-when-query-sdk-writer-output)
* [How to check LRU cache memory footprint?](#how-to-check-lru-cache-memory-footprint)
* [How to deal with the trailing task in query?](#How-to-deal-with-the-trailing-task-in-query)
+* [How to manage hybrid file format in carbondata table?](#How-to-manage-hybrid-file-format-in-carbondata-table)
# TroubleShooting
@@ -213,6 +214,10 @@ spark.speculation is a group of configuration, that can monitor trailing tasks a
spark.blacklist.enabled, avoid reduction of available executors due to blacklist mechanism.
+## How to manage hybrid file format in carbondata table
+
+Refer [Heterogeneous format segments in carbondata](./addsegment-guide.md)
+
## Getting tablestatus.lock issues When loading data
**Symptom**
[carbondata] 03/14: FAQ: carbon rename to carbondata
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 885a21c413508a9ef959d329c56872d1cfc48b66
Author: bieremayi <li...@sina.com>
AuthorDate: Mon Nov 29 18:39:07 2021 +0800
FAQ: carbon rename to carbondata
---
docs/faq.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/faq.md b/docs/faq.md
index d31aad0..4daf2f5 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -29,7 +29,7 @@
* [Why different time zone result for select query output when query SDK writer output?](#why-different-time-zone-result-for-select-query-output-when-query-sdk-writer-output)
* [How to check LRU cache memory footprint?](#how-to-check-lru-cache-memory-footprint)
* [How to deal with the trailing task in query?](#How-to-deal-with-the-trailing-task-in-query)
-* [How to manage mix file format in carbon table?](#How-to-manage-mix-file-format-in-carbon-table)
+* [How to manage mix file format in carbondata table?](#How-to-manage-mix-file-format-in-carbondata-table)
# TroubleShooting
[carbondata] 10/14: Revert "Supplementary information for add segment syntax ."
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit f544e59de1b75cf595f0b194928a8fb255928bca
Author: bieremayi <li...@sina.com>
AuthorDate: Tue Dec 7 17:07:39 2021 +0800
Revert "Supplementary information for add segment syntax ."
This reverts commit 81c2e29604d66e35f3757122cbc7cda0ed463cac.
---
docs/addsegment-guide.md | 42 +++-----------------------------
docs/segment-management-on-carbondata.md | 3 +--
2 files changed, 4 insertions(+), 41 deletions(-)
diff --git a/docs/addsegment-guide.md b/docs/addsegment-guide.md
index 648c426..78b15e6 100644
--- a/docs/addsegment-guide.md
+++ b/docs/addsegment-guide.md
@@ -27,46 +27,10 @@ Heterogeneous format segments aims to solve this problem by avoiding data conver
### Add segment with path and format
Users can add the existing data as a segment to the carbon table provided the schema of the data
and the carbon table should be the same.
-
- Syntax
-
- ```
- ALTER TABLE [db_name.]table_name ADD SEGMENT OPTIONS(property_name=property_value, ...)
- ```
-
-**Supported properties:**
-
-| Property | Description |
-| ------------------------------------------------------------ | ------------------------------------------------------------ |
-| [PATH](#path) | User external old table path |
-| [FORMAT](#format) | User external old table file format |
-| [PARTITION](#partition) | Extract partition info for partition table , should be form of "a:int, b:string" |
-
-
--
- You can use the following options to add segment:
-
- - ##### PATH:
- User old table path.
-
- ```
- OPTIONS('PATH'='hdfs://usr/oldtable')
- ```
-
- - ##### FORMAT:
- User old table file format. eg : json, parquet, jdbc, orc, csv, text
-
- ```
- OPTIONS('FORMAT'='parquet')
- ```
- - ##### PARTITION:
- Extract partition info for partition table , should be form of "a:int, b:string"
-
- ```
- OPTIONS('PARTITION'='a:int, b:string')
- ```
-
+```
+alter table table_name add segment options ('path'= 'hdfs://usr/oldtable','format'='parquet')
+```
In the above command user can add the existing data to the carbon table as a new segment and also
can provide the data format.
diff --git a/docs/segment-management-on-carbondata.md b/docs/segment-management-on-carbondata.md
index ef01799..6c144b1 100644
--- a/docs/segment-management-on-carbondata.md
+++ b/docs/segment-management-on-carbondata.md
@@ -25,7 +25,6 @@ concept which helps to maintain consistency of data and easy transaction managem
- [Delete Segment by ID](#delete-segment-by-id)
- [Delete Segment by Date](#delete-segment-by-date)
- [Query Data with Specified Segments](#query-data-with-specified-segments)
-- [Add Segment](./addsegment-guide.md)
### SHOW SEGMENT
@@ -208,4 +207,4 @@ concept which helps to maintain consistency of data and easy transaction managem
spark.sql("select count(empno) from carbon.input.segments.db.carbontable_Multi_Thread").show();
}
}
- ```
\ No newline at end of file
+ ```
[carbondata] 04/14: remove add segment refer
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 69ab06ce2724649af49708ee55ff2e6abacad46e
Author: bieremayi <li...@sina.com>
AuthorDate: Thu Dec 2 19:26:07 2021 +0800
remove add segment refer
---
docs/segment-management-on-carbondata.md | 1 -
1 file changed, 1 deletion(-)
diff --git a/docs/segment-management-on-carbondata.md b/docs/segment-management-on-carbondata.md
index ef01799..7dc84be 100644
--- a/docs/segment-management-on-carbondata.md
+++ b/docs/segment-management-on-carbondata.md
@@ -25,7 +25,6 @@ concept which helps to maintain consistency of data and easy transaction managem
- [Delete Segment by ID](#delete-segment-by-id)
- [Delete Segment by Date](#delete-segment-by-date)
- [Query Data with Specified Segments](#query-data-with-specified-segments)
-- [Add Segment](./addsegment-guide.md)
### SHOW SEGMENT
[carbondata] 07/14: Revert "remove add segment refer"
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 580f7f6651832f6942e64ced5569934155ccd31c
Author: bieremayi <li...@sina.com>
AuthorDate: Tue Dec 7 17:07:01 2021 +0800
Revert "remove add segment refer"
This reverts commit 69ab06ce2724649af49708ee55ff2e6abacad46e.
---
docs/segment-management-on-carbondata.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/docs/segment-management-on-carbondata.md b/docs/segment-management-on-carbondata.md
index 7dc84be..ef01799 100644
--- a/docs/segment-management-on-carbondata.md
+++ b/docs/segment-management-on-carbondata.md
@@ -25,6 +25,7 @@ concept which helps to maintain consistency of data and easy transaction managem
- [Delete Segment by ID](#delete-segment-by-id)
- [Delete Segment by Date](#delete-segment-by-date)
- [Query Data with Specified Segments](#query-data-with-specified-segments)
+- [Add Segment](./addsegment-guide.md)
### SHOW SEGMENT
[carbondata] 14/14: [CARBONDATA-4315] Supplementary information for add segment syntax This closes #4239
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit f266a73e531fda02f0ab2bffffe0a788ee08cea9
Merge: 0ced3c8 a1b6d99
Author: chenliang613 <ch...@apache.org>
AuthorDate: Mon Dec 20 22:55:16 2021 +0800
[CARBONDATA-4315] Supplementary information for add segment syntax This closes #4239
docs/addsegment-guide.md | 134 +++++++++++++++++++++++++++++++++++++++++++++--
docs/faq.md | 5 ++
2 files changed, 136 insertions(+), 3 deletions(-)
[carbondata] 06/14: Revert "remove useless numerical value , revert some typo issues"
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 42d59be0be3eb58b26cb50ed16b4f2b2dd43cfb1
Author: bieremayi <li...@sina.com>
AuthorDate: Tue Dec 7 17:06:29 2021 +0800
Revert "remove useless numerical value , revert some typo issues"
This reverts commit 7af81adbf6800655902754c9b1dac5c1fb263f9d.
---
docs/addsegment-guide.md | 24 ++++++++++++------------
docs/faq.md | 6 +++---
2 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/docs/addsegment-guide.md b/docs/addsegment-guide.md
index 619293f..3985beb 100644
--- a/docs/addsegment-guide.md
+++ b/docs/addsegment-guide.md
@@ -40,7 +40,7 @@ Users can add the existing data as a segment to the carbon table provided the sc
| ------------------------------------------------------------ | ------------------------------------------------------------ |
| [PATH](#path) | User external old table path |
| [FORMAT](#format) | User external old table file format |
-| [PARTITION](#partition) | Partition info for partition table , should be form of "a:int, b:string" |
+| [PARTITION](#partition) | Extract partition info for partition table , should be form of "a:int, b:string" |
-
@@ -60,7 +60,7 @@ Users can add the existing data as a segment to the carbon table provided the sc
OPTIONS('FORMAT'='parquet')
```
- ##### PARTITION:
- Partition info for partition table , should be form of "a:int, b:string"
+ Extract partition info for partition table , should be form of "a:int, b:string"
```
OPTIONS('PARTITION'='a:int, b:string')
@@ -95,8 +95,8 @@ LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par';
Parquet File Location :
```
-/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0
-/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=1
+25.1 K 75.2 K /user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0
+8.7 K 26.2 K /user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=1
```
@@ -117,8 +117,8 @@ LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_orc_par';
Orc File Location :
```
-/user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=0
-/user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=1
+19.4 K 58.2 K /user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=0
+7.1 K 21.4 K /user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=1
```
**Steps:**
@@ -156,13 +156,13 @@ step3: Result check.
SHOW SEGMENTS FOR TABLE default.log_carbon_par;
```
-| ID | Status | Load Start Time | Load Time Taken | Partition | Data Size | Index Size | File Format |
+|SegmentSequenceId|Status|Load Time|Elapsed Time |Partition|Data Size|Index Size|File Format|
|:---|:---|:---|:---|:---|:---|:---|:---|
-|4 |Success |2021-11-29 17:59:40.819 |7.026S |{day=20211123,hour=12,type=1}, ... |xxx| xxx |columnar_v3|
-|3 |Success |2021-11-29 16:34:28.106 |0.418S |{day=20211123,hour=12,type=0} |xxx |NA | orc|
-|2 |Success |2021-11-29 16:34:27.733 |0.222S |{day=20211123,hour=12,type=1} |xxx |NA | orc|
-|1 |Success |2021-11-29 16:30:17.207 |0.275S |{day=20211123,hour=12,type=0} |xxx |NA | parquet|
-|0 |Success |2021-11-29 16:30:16.48 |0.452S |{day=20211123,hour=12,type=1} |xxx |NA | parquet|
+|4 |Success |2021-11-29 17:59:40.819 |7.026S |{day=20211123,hour=12,type=1}, ... |29.86KB| 4.80KB |columnar_v3|
+|3 |Success |2021-11-29 16:34:28.106 |0.418S |{day=20211123,hour=12,type=0} |19.40KB |NA | orc|
+|2 |Success |2021-11-29 16:34:27.733 |0.222S |{day=20211123,hour=12,type=1} |7.12KB |NA | orc|
+|1 |Success |2021-11-29 16:30:17.207 |0.275S |{day=20211123,hour=12,type=0} |25.08KB |NA | parquet|
+|0 |Success |2021-11-29 16:30:16.48 |0.452S |{day=20211123,hour=12,type=1} |8.72KB |NA | parquet|
### Changes to tablestatus file
diff --git a/docs/faq.md b/docs/faq.md
index 304e826..4daf2f5 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -29,7 +29,7 @@
* [Why different time zone result for select query output when query SDK writer output?](#why-different-time-zone-result-for-select-query-output-when-query-sdk-writer-output)
* [How to check LRU cache memory footprint?](#how-to-check-lru-cache-memory-footprint)
* [How to deal with the trailing task in query?](#How-to-deal-with-the-trailing-task-in-query)
-* [How to manage hybrid file format in carbondata table?](#How-to-manage-hybrid-file-format-in-carbondata-table)
+* [How to manage mix file format in carbondata table?](#How-to-manage-mix-file-format-in-carbondata-table)
# TroubleShooting
@@ -214,9 +214,9 @@ spark.speculation is a group of configuration, that can monitor trailing tasks a
spark.blacklist.enabled, avoid reduction of available executors due to blacklist mechanism.
-## How to manage hybrid file format in carbondata table
+## How to manage mix file format in carbondata table
-Refer [Heterogeneous format segments in carbondata](./addsegment-guide.md)
+[Heterogeneous format segments in carbondata](./addsegment-guide.md)
## Getting tablestatus.lock issues When loading data
[carbondata] 12/14: Update docs/addsegment-guide.md
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 379f5ade13511838cca62a867ebe58ccf116f878
Author: bieremayi <li...@sina.com>
AuthorDate: Sat Dec 18 21:00:33 2021 +0800
Update docs/addsegment-guide.md
ths!
Co-authored-by: Indhumathi27 <34...@users.noreply.github.com>
---
docs/addsegment-guide.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/addsegment-guide.md b/docs/addsegment-guide.md
index 619293f..f9b7695 100644
--- a/docs/addsegment-guide.md
+++ b/docs/addsegment-guide.md
@@ -60,7 +60,7 @@ Users can add the existing data as a segment to the carbon table provided the sc
OPTIONS('FORMAT'='parquet')
```
- ##### PARTITION:
- Partition info for partition table , should be form of "a:int, b:string"
+ Partition info for partition table , should be form of "a:int, b:string"
```
OPTIONS('PARTITION'='a:int, b:string')
[carbondata] 01/14: Supplementary information for add segment syntax .
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 81c2e29604d66e35f3757122cbc7cda0ed463cac
Author: bieremayi <li...@sina.com>
AuthorDate: Thu Nov 25 17:46:57 2021 +0800
Supplementary information for add segment syntax .
1. add segment option (partition)
2. segment-management-on-carbondata.md link addsegment-guide.md
---
docs/addsegment-guide.md | 42 +++++++++++++++++++++++++++++---
docs/segment-management-on-carbondata.md | 3 ++-
2 files changed, 41 insertions(+), 4 deletions(-)
diff --git a/docs/addsegment-guide.md b/docs/addsegment-guide.md
index 78b15e6..648c426 100644
--- a/docs/addsegment-guide.md
+++ b/docs/addsegment-guide.md
@@ -27,10 +27,46 @@ Heterogeneous format segments aims to solve this problem by avoiding data conver
### Add segment with path and format
Users can add the existing data as a segment to the carbon table provided the schema of the data
and the carbon table should be the same.
+
+ Syntax
+
+ ```
+ ALTER TABLE [db_name.]table_name ADD SEGMENT OPTIONS(property_name=property_value, ...)
+ ```
+
+**Supported properties:**
+
+| Property | Description |
+| ------------------------------------------------------------ | ------------------------------------------------------------ |
+| [PATH](#path) | User external old table path |
+| [FORMAT](#format) | User external old table file format |
+| [PARTITION](#partition) | Extract partition info for partition table , should be form of "a:int, b:string" |
+
+
+-
+ You can use the following options to add segment:
+
+ - ##### PATH:
+ User old table path.
+
+ ```
+ OPTIONS('PATH'='hdfs://usr/oldtable')
+ ```
+
+ - ##### FORMAT:
+ User old table file format. eg : json, parquet, jdbc, orc, csv, text
+
+ ```
+ OPTIONS('FORMAT'='parquet')
+ ```
+ - ##### PARTITION:
+ Extract partition info for partition table , should be form of "a:int, b:string"
+
+ ```
+ OPTIONS('PARTITION'='a:int, b:string')
+ ```
+
-```
-alter table table_name add segment options ('path'= 'hdfs://usr/oldtable','format'='parquet')
-```
In the above command user can add the existing data to the carbon table as a new segment and also
can provide the data format.
diff --git a/docs/segment-management-on-carbondata.md b/docs/segment-management-on-carbondata.md
index 6c144b1..ef01799 100644
--- a/docs/segment-management-on-carbondata.md
+++ b/docs/segment-management-on-carbondata.md
@@ -25,6 +25,7 @@ concept which helps to maintain consistency of data and easy transaction managem
- [Delete Segment by ID](#delete-segment-by-id)
- [Delete Segment by Date](#delete-segment-by-date)
- [Query Data with Specified Segments](#query-data-with-specified-segments)
+- [Add Segment](./addsegment-guide.md)
### SHOW SEGMENT
@@ -207,4 +208,4 @@ concept which helps to maintain consistency of data and easy transaction managem
spark.sql("select count(empno) from carbon.input.segments.db.carbontable_Multi_Thread").show();
}
}
- ```
+ ```
\ No newline at end of file
[carbondata] 08/14: Revert "FAQ: carbon rename to carbondata"
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 341f1bff1d6286b68a3d8d2250f837aa06eb8790
Author: bieremayi <li...@sina.com>
AuthorDate: Tue Dec 7 17:07:15 2021 +0800
Revert "FAQ: carbon rename to carbondata"
This reverts commit 885a21c413508a9ef959d329c56872d1cfc48b66.
---
docs/faq.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/faq.md b/docs/faq.md
index 4daf2f5..d31aad0 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -29,7 +29,7 @@
* [Why different time zone result for select query output when query SDK writer output?](#why-different-time-zone-result-for-select-query-output-when-query-sdk-writer-output)
* [How to check LRU cache memory footprint?](#how-to-check-lru-cache-memory-footprint)
* [How to deal with the trailing task in query?](#How-to-deal-with-the-trailing-task-in-query)
-* [How to manage mix file format in carbondata table?](#How-to-manage-mix-file-format-in-carbondata-table)
+* [How to manage mix file format in carbon table?](#How-to-manage-mix-file-format-in-carbon-table)
# TroubleShooting
[carbondata] 09/14: Revert "Add FAQ How to manage mix file format in carbondata table."
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit ce5747db73c9066cc7609e7534cfdb89e8281e9a
Author: bieremayi <li...@sina.com>
AuthorDate: Tue Dec 7 17:07:28 2021 +0800
Revert "Add FAQ How to manage mix file format in carbondata table."
This reverts commit 598d1cee348887538a0d5a20dce6c2106a2b23f6.
---
docs/addsegment-guide.md | 94 +-----------------------------------------------
docs/faq.md | 6 ----
2 files changed, 1 insertion(+), 99 deletions(-)
diff --git a/docs/addsegment-guide.md b/docs/addsegment-guide.md
index 3985beb..648c426 100644
--- a/docs/addsegment-guide.md
+++ b/docs/addsegment-guide.md
@@ -54,7 +54,7 @@ Users can add the existing data as a segment to the carbon table provided the sc
```
- ##### FORMAT:
- User old table file format. eg : parquet, orc
+ User old table file format. eg : json, parquet, jdbc, orc, csv, text
```
OPTIONS('FORMAT'='parquet')
@@ -73,98 +73,6 @@ In the above command user can add the existing data to the carbon table as a new
During add segment, it will infer the schema from data and validates the schema against the carbon table.
If the schema doesn’t match it throws an exception.
-**Example:**
-
-Exist old hive partition table , stored as orc or parquet file format:
-
-
-```sql
-CREATE TABLE default.log_parquet_par (
- id BIGINT,
- event_time BIGINT,
- ip STRING
-)PARTITIONED BY (
- day INT,
- hour INT,
- type INT
-)
-STORED AS parquet
-LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par';
-```
-
-Parquet File Location :
-
-```
-25.1 K 75.2 K /user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0
-8.7 K 26.2 K /user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=1
-```
-
-
-```sql
-CREATE TABLE default.log_orc_par (
- id BIGINT,
- event_time BIGINT,
- ip STRING
-)PARTITIONED BY (
- day INT,
- hour INT,
- type INT
-)
-STORED AS orc
-LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_orc_par';
-```
-
-Orc File Location :
-
-```
-19.4 K 58.2 K /user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=0
-7.1 K 21.4 K /user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=1
-```
-
-**Steps:**
-
-step1: Create carbondata format table with the same schema.
-
-```sql
-CREATE TABLE default.log_carbon_par (
- id BIGINT,
- event_time BIGINT,
- ip STRING
-)PARTITIONED BY (
- day INT,
- hour INT,
- type INT
-)
-STORED AS carbondata
-LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_carbon_par';
-```
-
-step2: Execute add segment sql.
-
-|SQL|Is Success|Error Message|
-|:---|:---|:---|
-|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/','format'='parquet','partition'='day:int,hour:int,type:int');|Yes|/|
-|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_orc_par/','format'='orc','partition'='day:int,hour:int,type:int');|Yes|/|
-|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day=20211123 and hour=12 and type=0');|No|Error in query: invalid partition option: Map(path -> hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0, format -> parquet, partition -> day=20211123 and hour=12 and type=0)|
-|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day=20211123,hour=12,type=0');|No|Error in query: invalid partition option: Map(path -> hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0, format -> parquet, partition -> day=20211123,hour=12,type=0)|
-|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet');|No|Error in query: partition option is required when adding segment to partition table|
-|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day:int,hour:int,type:int');|No|ERROR SparkSQLDriver: Failed in [alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day:int,hour:int,type:int')]|
-
-step3: Result check.
-
-```sql
-SHOW SEGMENTS FOR TABLE default.log_carbon_par;
-```
-
-|SegmentSequenceId|Status|Load Time|Elapsed Time |Partition|Data Size|Index Size|File Format|
-|:---|:---|:---|:---|:---|:---|:---|:---|
-|4 |Success |2021-11-29 17:59:40.819 |7.026S |{day=20211123,hour=12,type=1}, ... |29.86KB| 4.80KB |columnar_v3|
-|3 |Success |2021-11-29 16:34:28.106 |0.418S |{day=20211123,hour=12,type=0} |19.40KB |NA | orc|
-|2 |Success |2021-11-29 16:34:27.733 |0.222S |{day=20211123,hour=12,type=1} |7.12KB |NA | orc|
-|1 |Success |2021-11-29 16:30:17.207 |0.275S |{day=20211123,hour=12,type=0} |25.08KB |NA | parquet|
-|0 |Success |2021-11-29 16:30:16.48 |0.452S |{day=20211123,hour=12,type=1} |8.72KB |NA | parquet|
-
-
### Changes to tablestatus file
Carbon adds the new segment by adding segment information to tablestatus file. In order to add the path and format information to tablestatus, we are going to add `segmentPath` and `format` to the tablestatus file.
And any extra `options` will be added to the segment file.
diff --git a/docs/faq.md b/docs/faq.md
index d31aad0..a330e24 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -29,7 +29,6 @@
* [Why different time zone result for select query output when query SDK writer output?](#why-different-time-zone-result-for-select-query-output-when-query-sdk-writer-output)
* [How to check LRU cache memory footprint?](#how-to-check-lru-cache-memory-footprint)
* [How to deal with the trailing task in query?](#How-to-deal-with-the-trailing-task-in-query)
-* [How to manage mix file format in carbon table?](#How-to-manage-mix-file-format-in-carbon-table)
# TroubleShooting
@@ -214,11 +213,6 @@ spark.speculation is a group of configuration, that can monitor trailing tasks a
spark.blacklist.enabled, avoid reduction of available executors due to blacklist mechanism.
-## How to manage mix file format in carbondata table
-
-[Heterogeneous format segments in carbondata](./addsegment-guide.md)
-
-
## Getting tablestatus.lock issues When loading data
**Symptom**
[carbondata] 02/14: Add FAQ How to manage mix file format in carbondata table.
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 598d1cee348887538a0d5a20dce6c2106a2b23f6
Author: bieremayi <li...@sina.com>
AuthorDate: Mon Nov 29 18:18:05 2021 +0800
Add FAQ How to manage mix file format in carbondata table.
1. add segment example.
2. faq.md link to addsegment-guide.md
---
docs/addsegment-guide.md | 94 +++++++++++++++++++++++++++++++++++++++++++++++-
docs/faq.md | 6 ++++
2 files changed, 99 insertions(+), 1 deletion(-)
diff --git a/docs/addsegment-guide.md b/docs/addsegment-guide.md
index 648c426..3985beb 100644
--- a/docs/addsegment-guide.md
+++ b/docs/addsegment-guide.md
@@ -54,7 +54,7 @@ Users can add the existing data as a segment to the carbon table provided the sc
```
- ##### FORMAT:
- User old table file format. eg : json, parquet, jdbc, orc, csv, text
+ User old table file format. eg : parquet, orc
```
OPTIONS('FORMAT'='parquet')
@@ -73,6 +73,98 @@ In the above command user can add the existing data to the carbon table as a new
During add segment, it will infer the schema from data and validates the schema against the carbon table.
If the schema doesn’t match it throws an exception.
+**Example:**
+
+Exist old hive partition table , stored as orc or parquet file format:
+
+
+```sql
+CREATE TABLE default.log_parquet_par (
+ id BIGINT,
+ event_time BIGINT,
+ ip STRING
+)PARTITIONED BY (
+ day INT,
+ hour INT,
+ type INT
+)
+STORED AS parquet
+LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par';
+```
+
+Parquet File Location :
+
+```
+25.1 K 75.2 K /user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0
+8.7 K 26.2 K /user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=1
+```
+
+
+```sql
+CREATE TABLE default.log_orc_par (
+ id BIGINT,
+ event_time BIGINT,
+ ip STRING
+)PARTITIONED BY (
+ day INT,
+ hour INT,
+ type INT
+)
+STORED AS orc
+LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_orc_par';
+```
+
+Orc File Location :
+
+```
+19.4 K 58.2 K /user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=0
+7.1 K 21.4 K /user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=1
+```
+
+**Steps:**
+
+step1: Create carbondata format table with the same schema.
+
+```sql
+CREATE TABLE default.log_carbon_par (
+ id BIGINT,
+ event_time BIGINT,
+ ip STRING
+)PARTITIONED BY (
+ day INT,
+ hour INT,
+ type INT
+)
+STORED AS carbondata
+LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_carbon_par';
+```
+
+step2: Execute add segment sql.
+
+|SQL|Is Success|Error Message|
+|:---|:---|:---|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/','format'='parquet','partition'='day:int,hour:int,type:int');|Yes|/|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_orc_par/','format'='orc','partition'='day:int,hour:int,type:int');|Yes|/|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day=20211123 and hour=12 and type=0');|No|Error in query: invalid partition option: Map(path -> hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0, format -> parquet, partition -> day=20211123 and hour=12 and type=0)|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day=20211123,hour=12,type=0');|No|Error in query: invalid partition option: Map(path -> hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0, format -> parquet, partition -> day=20211123,hour=12,type=0)|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet');|No|Error in query: partition option is required when adding segment to partition table|
+|alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day:int,hour:int,type:int');|No|ERROR SparkSQLDriver: Failed in [alter table default.log_carbon_par add segment options ('path'= 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0','format'='parquet','partition'='day:int,hour:int,type:int')]|
+
+step3: Result check.
+
+```sql
+SHOW SEGMENTS FOR TABLE default.log_carbon_par;
+```
+
+|SegmentSequenceId|Status|Load Time|Elapsed Time |Partition|Data Size|Index Size|File Format|
+|:---|:---|:---|:---|:---|:---|:---|:---|
+|4 |Success |2021-11-29 17:59:40.819 |7.026S |{day=20211123,hour=12,type=1}, ... |29.86KB| 4.80KB |columnar_v3|
+|3 |Success |2021-11-29 16:34:28.106 |0.418S |{day=20211123,hour=12,type=0} |19.40KB |NA | orc|
+|2 |Success |2021-11-29 16:34:27.733 |0.222S |{day=20211123,hour=12,type=1} |7.12KB |NA | orc|
+|1 |Success |2021-11-29 16:30:17.207 |0.275S |{day=20211123,hour=12,type=0} |25.08KB |NA | parquet|
+|0 |Success |2021-11-29 16:30:16.48 |0.452S |{day=20211123,hour=12,type=1} |8.72KB |NA | parquet|
+
+
### Changes to tablestatus file
Carbon adds the new segment by adding segment information to tablestatus file. In order to add the path and format information to tablestatus, we are going to add `segmentPath` and `format` to the tablestatus file.
And any extra `options` will be added to the segment file.
diff --git a/docs/faq.md b/docs/faq.md
index a330e24..d31aad0 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -29,6 +29,7 @@
* [Why different time zone result for select query output when query SDK writer output?](#why-different-time-zone-result-for-select-query-output-when-query-sdk-writer-output)
* [How to check LRU cache memory footprint?](#how-to-check-lru-cache-memory-footprint)
* [How to deal with the trailing task in query?](#How-to-deal-with-the-trailing-task-in-query)
+* [How to manage mix file format in carbon table?](#How-to-manage-mix-file-format-in-carbon-table)
# TroubleShooting
@@ -213,6 +214,11 @@ spark.speculation is a group of configuration, that can monitor trailing tasks a
spark.blacklist.enabled, avoid reduction of available executors due to blacklist mechanism.
+## How to manage mix file format in carbondata table
+
+[Heterogeneous format segments in carbondata](./addsegment-guide.md)
+
+
## Getting tablestatus.lock issues When loading data
**Symptom**
[carbondata] 05/14: remove useless numerical value , revert some typo issues
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 7af81adbf6800655902754c9b1dac5c1fb263f9d
Author: bieremayi <li...@sina.com>
AuthorDate: Sat Dec 4 16:42:07 2021 +0800
remove useless numerical value , revert some typo issues
---
docs/addsegment-guide.md | 24 ++++++++++++------------
docs/faq.md | 6 +++---
2 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/docs/addsegment-guide.md b/docs/addsegment-guide.md
index 3985beb..619293f 100644
--- a/docs/addsegment-guide.md
+++ b/docs/addsegment-guide.md
@@ -40,7 +40,7 @@ Users can add the existing data as a segment to the carbon table provided the sc
| ------------------------------------------------------------ | ------------------------------------------------------------ |
| [PATH](#path) | User external old table path |
| [FORMAT](#format) | User external old table file format |
-| [PARTITION](#partition) | Extract partition info for partition table , should be form of "a:int, b:string" |
+| [PARTITION](#partition) | Partition info for partition table , should be form of "a:int, b:string" |
-
@@ -60,7 +60,7 @@ Users can add the existing data as a segment to the carbon table provided the sc
OPTIONS('FORMAT'='parquet')
```
- ##### PARTITION:
- Extract partition info for partition table , should be form of "a:int, b:string"
+ Partition info for partition table , should be form of "a:int, b:string"
```
OPTIONS('PARTITION'='a:int, b:string')
@@ -95,8 +95,8 @@ LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_parquet_par';
Parquet File Location :
```
-25.1 K 75.2 K /user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0
-8.7 K 26.2 K /user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=1
+/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=0
+/user/hive/warehouse/log_parquet_par/day=20211123/hour=12/type=1
```
@@ -117,8 +117,8 @@ LOCATION 'hdfs://bieremayi/user/hive/warehouse/log_orc_par';
Orc File Location :
```
-19.4 K 58.2 K /user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=0
-7.1 K 21.4 K /user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=1
+/user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=0
+/user/hive/warehouse/log_orc_par/day=20211123/hour=12/type=1
```
**Steps:**
@@ -156,13 +156,13 @@ step3: Result check.
SHOW SEGMENTS FOR TABLE default.log_carbon_par;
```
-|SegmentSequenceId|Status|Load Time|Elapsed Time |Partition|Data Size|Index Size|File Format|
+| ID | Status | Load Start Time | Load Time Taken | Partition | Data Size | Index Size | File Format |
|:---|:---|:---|:---|:---|:---|:---|:---|
-|4 |Success |2021-11-29 17:59:40.819 |7.026S |{day=20211123,hour=12,type=1}, ... |29.86KB| 4.80KB |columnar_v3|
-|3 |Success |2021-11-29 16:34:28.106 |0.418S |{day=20211123,hour=12,type=0} |19.40KB |NA | orc|
-|2 |Success |2021-11-29 16:34:27.733 |0.222S |{day=20211123,hour=12,type=1} |7.12KB |NA | orc|
-|1 |Success |2021-11-29 16:30:17.207 |0.275S |{day=20211123,hour=12,type=0} |25.08KB |NA | parquet|
-|0 |Success |2021-11-29 16:30:16.48 |0.452S |{day=20211123,hour=12,type=1} |8.72KB |NA | parquet|
+|4 |Success |2021-11-29 17:59:40.819 |7.026S |{day=20211123,hour=12,type=1}, ... |xxx| xxx |columnar_v3|
+|3 |Success |2021-11-29 16:34:28.106 |0.418S |{day=20211123,hour=12,type=0} |xxx |NA | orc|
+|2 |Success |2021-11-29 16:34:27.733 |0.222S |{day=20211123,hour=12,type=1} |xxx |NA | orc|
+|1 |Success |2021-11-29 16:30:17.207 |0.275S |{day=20211123,hour=12,type=0} |xxx |NA | parquet|
+|0 |Success |2021-11-29 16:30:16.48 |0.452S |{day=20211123,hour=12,type=1} |xxx |NA | parquet|
### Changes to tablestatus file
diff --git a/docs/faq.md b/docs/faq.md
index 4daf2f5..304e826 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -29,7 +29,7 @@
* [Why different time zone result for select query output when query SDK writer output?](#why-different-time-zone-result-for-select-query-output-when-query-sdk-writer-output)
* [How to check LRU cache memory footprint?](#how-to-check-lru-cache-memory-footprint)
* [How to deal with the trailing task in query?](#How-to-deal-with-the-trailing-task-in-query)
-* [How to manage mix file format in carbondata table?](#How-to-manage-mix-file-format-in-carbondata-table)
+* [How to manage hybrid file format in carbondata table?](#How-to-manage-hybrid-file-format-in-carbondata-table)
# TroubleShooting
@@ -214,9 +214,9 @@ spark.speculation is a group of configuration, that can monitor trailing tasks a
spark.blacklist.enabled, avoid reduction of available executors due to blacklist mechanism.
-## How to manage mix file format in carbondata table
+## How to manage hybrid file format in carbondata table
-[Heterogeneous format segments in carbondata](./addsegment-guide.md)
+Refer [Heterogeneous format segments in carbondata](./addsegment-guide.md)
## Getting tablestatus.lock issues When loading data
[carbondata] 13/14: Update docs/addsegment-guide.md
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit a1b6d99c21dd4a8cfa8e5f6b4a38f45edea40539
Author: bieremayi <li...@sina.com>
AuthorDate: Sat Dec 18 21:01:36 2021 +0800
Update docs/addsegment-guide.md
ths!
Co-authored-by: Indhumathi27 <34...@users.noreply.github.com>
---
docs/addsegment-guide.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/addsegment-guide.md b/docs/addsegment-guide.md
index f9b7695..0726a36 100644
--- a/docs/addsegment-guide.md
+++ b/docs/addsegment-guide.md
@@ -54,7 +54,7 @@ Users can add the existing data as a segment to the carbon table provided the sc
```
- ##### FORMAT:
- User old table file format. eg : parquet, orc
+ User old table file format. eg : parquet, orc
```
OPTIONS('FORMAT'='parquet')