You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2023/01/04 03:49:02 UTC
[GitHub] [doris] yuxuan-luo opened a new issue, #15592: [Bug] 1.2.1 Stream Load hidden_columns fail
yuxuan-luo opened a new issue, #15592:
URL: https://github.com/apache/doris/issues/15592
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
1.2.1
### What's Wrong?
CREATE TABLE user_log_1 (
user_id VARCHAR(20),
item_id VARCHAR(30),
category_id VARCHAR(30),
behavior VARCHAR(30),
ts datetime
) ENGINE=OLAP
UNIQUE KEY(`user_id`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`user_id`) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"function_column.sequence_type" = 'Date',
"in_memory" = "false",
"storage_format" = "V2"
);
-- 数据
1,112321,10023,pv,2021-09-27 10:40:34
1,112326,10023,pv,2021-09-27 10:41:34
1,112325,10023,pv,2021-09-27 10:42:34
1,112324,10023,pv,2021-09-27 10:43:34
1,112323,10023,pv,2021-09-27 10:44:34
1,112327,10023,pv,2021-09-27 10:47:34
-- stream load
curl --location-trusted -u admin -H "function_column.sequence_col: ts" -H "column_separator:," -H "hidden_columns: __DORIS_SEQUENCE_COL__" -T test2.csv http://10.0.0.102:8040/api/hugo/user_log_1/_stream_load
-- return
"Message": "too many filtered rows",
Reason: actual column number in csv file is more than schema column number.actual number: 6, column separator: [,], line delimiter: [
], schema column number: 5; . src line [1,112321,10023,pv,2021-09-27 10:40:34,2021-09-27];
### What You Expected?
可以正确通过 hidden_columns: __DORIS_SEQUENCE_COL__ 指定导入数据中包含的隐藏列，在上面例子中导入数据中最后一列数据为__DORIS_SEQUENCE_COL__
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] liaoxin01 commented on issue #15592: [Bug] 1.2.1 Stream Load hidden_columns fail
Posted by GitBox <gi...@apache.org>.
liaoxin01 commented on issue #15592:
URL: https://github.com/apache/doris/issues/15592#issuecomment-1370650389
隐藏列处理应该是有些问题，一般不建议直接使用隐藏列。
有两种使用方式：
1. 推荐使用建表直接指定表中某一列为sequence列，导入时不用制定参数了
2. 建表时指定function_column.sequence_type
详见：https://doris.incubator.apache.org/zh-CN/docs/dev/data-operate/update-delete/sequence-column-manual?_highlight=sequence#%E8%AE%BE%E7%BD%AEsequence_col%E6%8E%A8%E8%8D%90
上面的case可以直接这么用，数据中不用多加一列sequence列，直接使用
1,112321,10023,pv,2021-09-27 10:40:34
1,112326,10023,pv,2021-09-27 10:41:34
1,112325,10023,pv,2021-09-27 10:42:34
1,112324,10023,pv,2021-09-27 10:43:34
1,112323,10023,pv,2021-09-27 10:44:34
1,112327,10023,pv,2021-09-27 10:47:34
curl --location-trusted -u admin -H "function_column.sequence_col: ts" -H "column_separator:," -T test2.csv http://10.0.0.102:8040/api/hugo/user_log_1/_stream_load
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org