You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2023/02/21 07:47:12 UTC
[GitHub] [doris] morningman opened a new pull request, #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
morningman opened a new pull request, #16983:
URL: https://github.com/apache/doris/pull/16983
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on code in PR #16983:
URL: https://github.com/apache/doris/pull/16983#discussion_r1113751175
##########
docs/en/docs/lakehouse/file.md:
##########
@@ -85,6 +85,57 @@ As can be seen, Doris is able to automatically infer column types based on the m
Besides Parquet, Doris supports analysis and auto column type inference of ORC, CSV, and Json files.
+**CSV Schema**
+
+<version since="dev"></version>
+
+在默认情况下，对 CSV 格式文件，所有列类型均为 String。可以通过 `csv_schema` 属性单独指定列名和列类型。Doris 会使用指定的列类型进行文件读取。格式如下：
+
+`name1:type1,name2:type2,...`
+
+对于格式不匹配的列（比如文件中为字符串，用户定义为 int），或缺失列（比如文件中有4列，用户定义了5列），则这些列将返回null。
+
+当前支持的列类型为：
+
Review Comment:
delete these chinese
##########
fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java:
##########
@@ -180,6 +193,74 @@ protected void parseProperties(Map<String, String> validParams) throws AnalysisE
stripOuterArray = Boolean.valueOf(validParams.get(STRIP_OUTER_ARRAY)).booleanValue();
numAsString = Boolean.valueOf(validParams.get(NUM_AS_STRING)).booleanValue();
fuzzyParse = Boolean.valueOf(validParams.get(FUZZY_PARSE)).booleanValue();
+
+ if (formatString.equals("csv") || formatString.equals("csv_with_names")
+ || formatString.equals("csv_with_names_and_types")) {
Review Comment:
For `csv_with_names_and_types` type, may be we should parse csv schema from csv file?
##########
fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java:
##########
@@ -180,6 +193,74 @@ protected void parseProperties(Map<String, String> validParams) throws AnalysisE
stripOuterArray = Boolean.valueOf(validParams.get(STRIP_OUTER_ARRAY)).booleanValue();
numAsString = Boolean.valueOf(validParams.get(NUM_AS_STRING)).booleanValue();
fuzzyParse = Boolean.valueOf(validParams.get(FUZZY_PARSE)).booleanValue();
+
+ if (formatString.equals("csv") || formatString.equals("csv_with_names")
+ || formatString.equals("csv_with_names_and_types")) {
+ parseCsvSchema(csvSchema, validParams);
+ }
+ }
+
+ // public for unit test
+ public static void parseCsvSchema(List<Column> csvSchema, Map<String, String> validParams)
+ throws AnalysisException {
+ String csvSchemaStr = validParams.get(CSV_SCHEMA);
+ if (Strings.isNullOrEmpty(csvSchemaStr)) {
+ return;
+ }
+ // the schema str is like: "k1:int;k2:bigint;k3:varchar(20)"
+ String[] schemaStrs = csvSchemaStr.split(";");
+ try {
+ for (String schemaStr : schemaStrs) {
+ String[] kv = schemaStr.replace(" ", "").split(":");
+ if (kv.length != 2) {
+ throw new AnalysisException("invalid csv schema: " + csvSchemaStr);
+ }
+ Column column = null;
+ String name = kv[0].toLowerCase();
+ FeNameFormat.checkColumnName(name);
+ String type = kv[1].toLowerCase();
+ if (type.equals("tinyint")) {
+ column = new Column(name, PrimitiveType.TINYINT, true);
+ } else if (type.equals("smallint")) {
+ column = new Column(name, PrimitiveType.SMALLINT, true);
+ } else if (type.equals("int")) {
+ column = new Column(name, PrimitiveType.INT, true);
+ } else if (type.equals("bigint")) {
+ column = new Column(name, PrimitiveType.BIGINT, true);
+ } else if (type.equals("largeint")) {
+ column = new Column(name, PrimitiveType.LARGEINT, true);
+ } else if (type.equals("float")) {
+ column = new Column(name, PrimitiveType.FLOAT, true);
+ } else if (type.equals("double")) {
+ column = new Column(name, PrimitiveType.DOUBLE, true);
+ } else if (type.startsWith("decimal")) {
+ // regex decimal(p, s)
+ Pattern pattern = Pattern.compile("decimal\\((\\d+),(\\d+)\\)");
+ Matcher matcher = pattern.matcher(type);
+ if (!matcher.find()) {
+ throw new AnalysisException("invalid decimal type: " + type);
+ }
+ int precision = Integer.parseInt(matcher.group(1));
+ int scale = Integer.parseInt(matcher.group(2));
+ column = new Column(name, ScalarType.createDecimalV3Type(precision, scale), false, null, true, null,
+ "");
+ } else if (type.equals("date")) {
+ column = new Column(name, ScalarType.createDateType(), false, null, true, null, "");
+ } else if (type.equals("datetime")) {
+ column = new Column(name, ScalarType.createDatetimeType(), false, null, true, null, "");
Review Comment:
datetimeV2 supports up to 6 digits (microseconds), like datetimev2(6).Should here support this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16983:
URL: https://github.com/apache/doris/pull/16983#issuecomment-1437993309
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #16983:
URL: https://github.com/apache/doris/pull/16983#issuecomment-1449733193
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #16983:
URL: https://github.com/apache/doris/pull/16983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #16983:
URL: https://github.com/apache/doris/pull/16983#issuecomment-1450315007
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #16983:
URL: https://github.com/apache/doris/pull/16983#issuecomment-1451219242
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16983:
URL: https://github.com/apache/doris/pull/16983#issuecomment-1449736976
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16983:
URL: https://github.com/apache/doris/pull/16983#issuecomment-1451145100
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #16983:
URL: https://github.com/apache/doris/pull/16983#discussion_r1115283629
##########
fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java:
##########
@@ -180,6 +193,74 @@ protected void parseProperties(Map<String, String> validParams) throws AnalysisE
stripOuterArray = Boolean.valueOf(validParams.get(STRIP_OUTER_ARRAY)).booleanValue();
numAsString = Boolean.valueOf(validParams.get(NUM_AS_STRING)).booleanValue();
fuzzyParse = Boolean.valueOf(validParams.get(FUZZY_PARSE)).booleanValue();
+
+ if (formatString.equals("csv") || formatString.equals("csv_with_names")
+ || formatString.equals("csv_with_names_and_types")) {
Review Comment:
Yes, may be later, because I didn't find a standard format of csv names and types yet.
And I think if user specified the schema, it should be override the names and type in file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #16983:
URL: https://github.com/apache/doris/pull/16983#discussion_r1121431421
##########
docs/en/docs/lakehouse/file.md:
##########
@@ -85,6 +85,57 @@ As can be seen, Doris is able to automatically infer column types based on the m
Besides Parquet, Doris supports analysis and auto column type inference of ORC, CSV, and Json files.
+**CSV Schema**
+
+<version since="dev"></version>
+
+在默认情况下，对 CSV 格式文件，所有列类型均为 String。可以通过 `csv_schema` 属性单独指定列名和列类型。Doris 会使用指定的列类型进行文件读取。格式如下：
+
+`name1:type1,name2:type2,...`
+
+对于格式不匹配的列（比如文件中为字符串，用户定义为 int），或缺失列（比如文件中有4列，用户定义了5列），则这些列将返回null。
+
+当前支持的列类型为：
+
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #16983:
URL: https://github.com/apache/doris/pull/16983#discussion_r1121432177
##########
fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java:
##########
@@ -180,6 +193,74 @@ protected void parseProperties(Map<String, String> validParams) throws AnalysisE
stripOuterArray = Boolean.valueOf(validParams.get(STRIP_OUTER_ARRAY)).booleanValue();
numAsString = Boolean.valueOf(validParams.get(NUM_AS_STRING)).booleanValue();
fuzzyParse = Boolean.valueOf(validParams.get(FUZZY_PARSE)).booleanValue();
+
+ if (formatString.equals("csv") || formatString.equals("csv_with_names")
+ || formatString.equals("csv_with_names_and_types")) {
+ parseCsvSchema(csvSchema, validParams);
+ }
+ }
+
+ // public for unit test
+ public static void parseCsvSchema(List<Column> csvSchema, Map<String, String> validParams)
+ throws AnalysisException {
+ String csvSchemaStr = validParams.get(CSV_SCHEMA);
+ if (Strings.isNullOrEmpty(csvSchemaStr)) {
+ return;
+ }
+ // the schema str is like: "k1:int;k2:bigint;k3:varchar(20)"
+ String[] schemaStrs = csvSchemaStr.split(";");
+ try {
+ for (String schemaStr : schemaStrs) {
+ String[] kv = schemaStr.replace(" ", "").split(":");
+ if (kv.length != 2) {
+ throw new AnalysisException("invalid csv schema: " + csvSchemaStr);
+ }
+ Column column = null;
+ String name = kv[0].toLowerCase();
+ FeNameFormat.checkColumnName(name);
+ String type = kv[1].toLowerCase();
+ if (type.equals("tinyint")) {
+ column = new Column(name, PrimitiveType.TINYINT, true);
+ } else if (type.equals("smallint")) {
+ column = new Column(name, PrimitiveType.SMALLINT, true);
+ } else if (type.equals("int")) {
+ column = new Column(name, PrimitiveType.INT, true);
+ } else if (type.equals("bigint")) {
+ column = new Column(name, PrimitiveType.BIGINT, true);
+ } else if (type.equals("largeint")) {
+ column = new Column(name, PrimitiveType.LARGEINT, true);
+ } else if (type.equals("float")) {
+ column = new Column(name, PrimitiveType.FLOAT, true);
+ } else if (type.equals("double")) {
+ column = new Column(name, PrimitiveType.DOUBLE, true);
+ } else if (type.startsWith("decimal")) {
+ // regex decimal(p, s)
+ Pattern pattern = Pattern.compile("decimal\\((\\d+),(\\d+)\\)");
+ Matcher matcher = pattern.matcher(type);
+ if (!matcher.find()) {
+ throw new AnalysisException("invalid decimal type: " + type);
+ }
+ int precision = Integer.parseInt(matcher.group(1));
+ int scale = Integer.parseInt(matcher.group(2));
+ column = new Column(name, ScalarType.createDecimalV3Type(precision, scale), false, null, true, null,
+ "");
+ } else if (type.equals("date")) {
+ column = new Column(name, ScalarType.createDateType(), false, null, true, null, "");
+ } else if (type.equals("datetime")) {
+ column = new Column(name, ScalarType.createDatetimeType(), false, null, true, null, "");
Review Comment:
done, all will use datetimev2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #16983:
URL: https://github.com/apache/doris/pull/16983#issuecomment-1449892029
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.04 seconds
stream load tsv: 474 seconds loaded 74807831229 Bytes, about 150 MB/s
stream load json: 40 seconds loaded 2358488459 Bytes, about 56 MB/s
stream load orc: 70 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230301110333_clickbench_pr_106387.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16983: [improvement](dry-run)(tvf) support csv schema in tvf and add "dry_run_query" variable
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16983:
URL: https://github.com/apache/doris/pull/16983#issuecomment-1450321840
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org