[GitHub] [incubator-doris] JNSimba opened a new issue #8133: [Bug] The max_filter_ratio parameter of streamload is invalid

[GitBox <gi...@apache.org>]

JNSimba opened a new issue #8133:
URL: https://github.com/apache/incubator-doris/issues/8133


   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Version
   
   0.15.1-rc09
   
   ### What's Wrong?
   
   I want to ignore data with mismatched lengths during streamload, but I found that if there is only one data in the batch, `max_filter_ratio` will be invalid.
   Eg:
   1.create table sql
   ```sql
   CREATE TABLE `test_jieru1` (
     `id` int(11) NULL COMMENT "",
     `name` varchar(10) NULL COMMENT ""
   ) ENGINE=OLAP
   UNIQUE KEY(`id`)
   COMMENT "OLAP"
   DISTRIBUTED BY HASH(`id`) BUCKETS 1
   PROPERTIES (
   "replication_allocation" = "tag.location.default: 3",
   "in_memory" = "false",
   "storage_format" = "V2"
   );
   ```
   
   2.data in example.json
   `[{"id":"1","name":"zhangsanzhangsanzhangsan"}]`
   
   3.streamload option
   `curl -v --location-trusted -u root:123456 -H "format: json" -H "strip_outer_array: true" -H "jsonpaths: [\"$.id\", \"$.name\"]" -H "max_filter_ratio:1" -H "columns: id,name" -T example.json http://127.0.0.1:8030/api/test/test/_stream_load`
   
   respones:
   ```
   {
       "TxnId": 67452740,
       "Label": "711803f3-d44f-46c8-bc58-68f4ebb394d9",
       "Status": "Fail",
       "Message": "all partitions have no load data",
       "NumberTotalRows": 1,
       "NumberLoadedRows": 0,
       "NumberFilteredRows": 1,
       "NumberUnselectedRows": 0,
       "LoadBytes": 40,
       "LoadTimeMs": 6,
       "BeginTxnTimeMs": 0,
       "StreamLoadPutTimeMs": 0,
       "ReadDataTimeMs": 0,
       "WriteDataTimeMs": 4,
       "CommitAndPublishTimeMs": 0,
       "ErrorURL": "http://127.0.0.1:8040/api/_load_error_log?file=__shard_1005/error_log_insert_stmt_26496fe992d7e55c-409680f9eba491b4_26496fe992d7e55c_409680f9eba491b4"
   }
   
   errorURL detail is :
   Reason: the length of input is too long than schema. column_name: name; input_str: [zhangsanzhangsanzhangsan] schema length: 10; actual length: 24; . src line: []; 
   ```
   
   **When I turn the data into multiple pieces，it success**
   eg:
   `[{"id":"1","name":"zhangsanzhangsanzhangsan"},{"id":1,"name":"wangwu"}]`
   
   response is 
   ```
   {
       "TxnId": 67452049,
       "Label": "2fea62bd-eb35-4a98-ae64-c861820e135c",
       "Status": "Success",
       "Message": "OK",
       "NumberTotalRows": 2,
       "NumberLoadedRows": 1,
       "NumberFilteredRows": 1,
       "NumberUnselectedRows": 0,
       "LoadBytes": 65,
       "LoadTimeMs": 23,
       "BeginTxnTimeMs": 0,
       "StreamLoadPutTimeMs": 0,
       "ReadDataTimeMs": 0,
       "WriteDataTimeMs": 5,
       "CommitAndPublishTimeMs": 15,
       "ErrorURL": "http://127.0.0.1:8040/api/_load_error_log?file=__shard_945/error_log_insert_stmt_7440e508eae8659f-ce310b4ed5edf196_7440e508eae8659f_ce310b4ed5edf196"
   }
   
   ```
   
   ### What You Expected?
   
   `max_filter_ratio` also takes effect when there is a single data
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org