You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sjyango (via GitHub)" <gi...@apache.org> on 2023/12/26 03:46:53 UTC
[PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
sjyango opened a new pull request, #29044:
URL: https://github.com/apache/doris/pull/29044
## Proposed changes
<!--Describe your changes.-->
Transforming from parsing std:: string to parsing char * to accelerate the parsing of ipv4/v6 data types.
**Comparison of import speed improvement for stream load:**
<ne-clipboard data="%7B%22type%22%3A%22fragment%22%2C%22name%22%3A%22%23fragment%22%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ubc98b83e%22%2C%22name%22%3A%22tableHole%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22Ngv3Z%22%2C%22name%22%3A%22table%22%2C%22attrs%22%3A%7B%22rowCount%22%3A5%2C%22colCount%22%3A3%2C%22colWidths%22%3A%5B250%2C250%2C250%5D%2C%22widthMode%22%3A%22contain%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u49005797%22%2C%22name%22%3A%22tr%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ue8692015%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A0%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u582553f0%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u3a85b
a11%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%22bold%22%3Atrue%7D%2C%22data%22%3A%22Data%20level%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u2453767c%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A1%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u6387549d%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22uc7b13f52%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%22bold%22%3Atrue%7D%2C%22data%22%3A%22Previous(s)%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uf2a3d69f%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A2%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u7c9bfc0c%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%2
2%2C%22id%22%3A%22ua555fbf7%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%22bold%22%3Atrue%7D%2C%22data%22%3A%22Current(s)%22%7D%5D%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u7520c86e%22%2C%22name%22%3A%22tr%22%2C%22attrs%22%3A%7B%22height%22%3A37%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uc4b8afda%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A0%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ubf98ed12%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22uc07cad3b%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%221w%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ufd18c9dc%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A1%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22
id%22%3A%22u7f2bbfc0%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ue9387dc2%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.078%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ub406bfe9%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A2%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ubfb995c0%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u6f388c7a%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.058%22%7D%5D%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u975dccad%22%2C%22name%22%3A%22tr%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u1d478515%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B
%22col%22%3A0%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u7a5223d0%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u6ffc3328%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%2210w%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u6e44f6f7%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A1%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u51e11f83%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ude860054%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.183%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ud9f840c3%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A2%2C%22vertic
alAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ufe338239%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ua43014ee%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.153%22%7D%5D%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u4d88029c%22%2C%22name%22%3A%22tr%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u7b6658b2%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A0%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ueaca1fe2%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ua2c715d4%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22100w%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%
22element%22%2C%22id%22%3A%22u557ba959%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A1%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u065b07b7%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ubb53f527%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%221.260%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u2403b259%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A2%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ue7282dbb%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u12f2c8dd%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.594%22%7D%5D%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%2
2id%22%3A%22uedfe65a7%22%2C%22name%22%3A%22tr%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u5b8c980a%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A0%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uce512180%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u2f7454a1%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%221000w%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ubfb3f415%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A1%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u347ea403%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ue1e2b651%22%2C%22name%22%3A%22%23text
%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%2212.146%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ucc7c2fa6%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A2%2C%22verticalAlign%22%3A%22middle%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u4194e77d%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%22alignment%22%3A%22center%22%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ue9a467bf%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%223.866%22%7D%5D%7D%5D%7D%5D%7D%5D%7D%5D%7D%5D%2C%22attrs%22%3A%7B%7D%7D" source="https%3A%2F%2Fwww.yuque.com%2Fbigboss-owq80%2Fqdfn7g%2Fpg5dzb44e4oq0agn%23xdeCu"></ne-clipboard><div class="lake-content" typography="classic">
Data level | Previous(s) | Current(s)
-- | -- | --
1w | 0.078 | 0.058
10w | 0.183 | 0.153
100w | 1.260 | 0.594
1000w | 12.146 | 3.866
</div>
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869287541
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit e92d02b8b7260f79e6cdf8fe20dbce50c52ed00f, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4796 4501 4468 4468
q2 373 178 161 161
q3 1474 1330 1222 1222
q4 1157 1010 930 930
q5 3178 3167 3155 3155
q6 255 137 132 132
q7 970 492 487 487
q8 2262 2269 2255 2255
q9 6703 6676 6713 6676
q10 3236 3273 3312 3273
q11 336 222 213 213
q12 357 205 211 205
q13 4567 3812 3797 3797
q14 242 214 216 214
q15 570 534 528 528
q16 447 380 388 380
q17 1044 765 648 648
q18 7156 6781 6807 6781
q19 1570 1593 1602 1593
q20 577 330 297 297
q21 3276 2800 2723 2723
q22 367 306 307 306
Total cold run time: 44913 ms
Total hot run time: 40444 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4415 4416 4382 4382
q2 274 166 170 166
q3 3512 3501 3489 3489
q4 2424 2406 2412 2406
q5 5725 5739 5726 5726
q6 249 126 127 126
q7 2368 1852 1896 1852
q8 3602 3612 3614 3612
q9 8987 8953 8955 8953
q10 3940 3993 4022 3993
q11 481 366 387 366
q12 769 606 595 595
q13 4269 3557 3580 3557
q14 282 251 252 251
q15 579 519 514 514
q16 493 455 472 455
q17 1963 1934 1945 1934
q18 8682 8159 8311 8159
q19 1826 1810 1818 1810
q20 2257 1934 1937 1934
q21 6614 6247 6229 6229
q22 559 457 468 457
Total cold run time: 64270 ms
Total hot run time: 60966 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #29044:
URL: https://github.com/apache/doris/pull/29044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1870145797
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "sjyango (via GitHub)" <gi...@apache.org>.
sjyango commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869937491
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "sjyango (via GitHub)" <gi...@apache.org>.
sjyango commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869408915
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1870774011
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869457498
TeamCity be ut coverage result:
Function Coverage: 36.60% (8563/23396)
Line Coverage: 28.67% (69656/242986)
Region Coverage: 27.69% (36042/130181)
Branch Coverage: 24.42% (18429/75474)
Coverage Report: http://coverage.selectdb-in.cc/coverage/941d67dcd87def4d6ef08bf502444482b244499a_941d67dcd87def4d6ef08bf502444482b244499a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "sjyango (via GitHub)" <gi...@apache.org>.
sjyango commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869365760
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869924610
TeamCity be ut coverage result:
Function Coverage: 36.59% (8562/23397)
Line Coverage: 28.66% (69655/242998)
Region Coverage: 27.68% (36041/130188)
Branch Coverage: 24.41% (18426/75476)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0ae016e23ca8807c1363f2a2380cece3dca701b8_0ae016e23ca8807c1363f2a2380cece3dca701b8/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869472028
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.74 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 20.7 seconds inserted 10000000 Rows, about 483K ops/s
storage size: 17187898988 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869932477
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.61 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17184197061 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869342192
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.06 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 37.2 seconds inserted 10000000 Rows, about 268K ops/s
storage size: 17183893842 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "sjyango (via GitHub)" <gi...@apache.org>.
sjyango commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869305282
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "sjyango (via GitHub)" <gi...@apache.org>.
sjyango commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869320563
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29044:
URL: https://github.com/apache/doris/pull/29044#discussion_r1436222600
##########
be/src/vec/runtime/ipv4_value.h:
##########
@@ -34,87 +33,41 @@ class IPv4Value {
explicit IPv4Value(vectorized::IPv4 ipv4) { _value = ipv4; }
- explicit IPv4Value(std::string ipv4) {}
-
- [[nodiscard]] const vectorized::IPv4& value() const { return _value; }
+ const vectorized::IPv4& value() const { return _value; }
vectorized::IPv4& value() { return _value; }
void set_value(vectorized::IPv4 ipv4) { _value = ipv4; }
- bool from_string(std::string ipv4) { return from_string(_value, ipv4); }
-
- [[nodiscard]] std::string to_string() const { return to_string(_value); }
+ bool from_string(const std::string& ipv4_str) { return from_string(_value, ipv4_str); }
- static bool from_string(vectorized::IPv4& value, std::string ipv4) {
- remove_ipv4_space(ipv4);
+ std::string to_string() const { return to_string(_value); }
- // shortest ipv4 string is `0.0.0.0` whose length is 7
- if (ipv4.size() < 7 || !is_valid_string(ipv4)) {
+ static bool from_string(vectorized::IPv4& value, const std::string& ipv4_str) {
+ if (ipv4_str.empty()) {
return false;
}
-
- vectorized::IPv4 octets[4] = {0};
- std::istringstream iss(ipv4);
- std::string octet;
- uint8_t octet_index = 0;
-
- while (getline(iss, octet, '.')) {
- if (octet_index >= 4) {
- return false;
- }
-
- StringParser::ParseResult result;
- vectorized::IPv4 val = StringParser::string_to_unsigned_int<vectorized::IPv4>(
- octet.c_str(), octet.length(), &result);
- if (result != StringParser::PARSE_SUCCESS || val > 255) {
- return false;
- }
-
- octets[octet_index++] = val;
- }
-
- if (octet_index != 4) {
+ int64_t parse_value;
+ const char* src = ipv4_str.c_str();
+ const char* end = ipv4_str.c_str() + ipv4_str.size() - 1;
+ while (std::isspace(*src)) ++src;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
while (std::isspace(*src)) { ++src;
}
```
##########
be/src/vec/runtime/ipv4_value.h:
##########
@@ -34,87 +33,41 @@
explicit IPv4Value(vectorized::IPv4 ipv4) { _value = ipv4; }
- explicit IPv4Value(std::string ipv4) {}
-
- [[nodiscard]] const vectorized::IPv4& value() const { return _value; }
+ const vectorized::IPv4& value() const { return _value; }
vectorized::IPv4& value() { return _value; }
void set_value(vectorized::IPv4 ipv4) { _value = ipv4; }
- bool from_string(std::string ipv4) { return from_string(_value, ipv4); }
-
- [[nodiscard]] std::string to_string() const { return to_string(_value); }
+ bool from_string(const std::string& ipv4_str) { return from_string(_value, ipv4_str); }
- static bool from_string(vectorized::IPv4& value, std::string ipv4) {
- remove_ipv4_space(ipv4);
+ std::string to_string() const { return to_string(_value); }
- // shortest ipv4 string is `0.0.0.0` whose length is 7
- if (ipv4.size() < 7 || !is_valid_string(ipv4)) {
+ static bool from_string(vectorized::IPv4& value, const std::string& ipv4_str) {
+ if (ipv4_str.empty()) {
return false;
}
-
- vectorized::IPv4 octets[4] = {0};
- std::istringstream iss(ipv4);
- std::string octet;
- uint8_t octet_index = 0;
-
- while (getline(iss, octet, '.')) {
- if (octet_index >= 4) {
- return false;
- }
-
- StringParser::ParseResult result;
- vectorized::IPv4 val = StringParser::string_to_unsigned_int<vectorized::IPv4>(
- octet.c_str(), octet.length(), &result);
- if (result != StringParser::PARSE_SUCCESS || val > 255) {
- return false;
- }
-
- octets[octet_index++] = val;
- }
-
- if (octet_index != 4) {
+ int64_t parse_value;
+ const char* src = ipv4_str.c_str();
+ const char* end = ipv4_str.c_str() + ipv4_str.size() - 1;
+ while (std::isspace(*src)) ++src;
+ while (std::isspace(*end)) --end;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
while (std::isspace(*end)) { --end;
}
```
##########
be/src/vec/runtime/ipv6_value.h:
##########
@@ -35,224 +34,35 @@ class IPv6Value {
explicit IPv6Value(vectorized::IPv6 ipv6) { _value = ipv6; }
- [[nodiscard]] const vectorized::IPv6& value() const { return _value; }
+ const vectorized::IPv6& value() const { return _value; }
vectorized::IPv6& value() { return _value; }
void set_value(vectorized::IPv6 ipv6) { _value = ipv6; }
- bool from_string(std::string ipv6) { return from_string(_value, ipv6); }
-
- bool from_binary_string(std::string ipv6_binary) {
- return from_binary_string(_value, ipv6_binary);
- }
-
- static bool from_string(vectorized::IPv6& x, std::string ipv6) {
- remove_ipv6_space(ipv6);
-
- if (ipv6.empty() || !is_valid_string(ipv6)) {
- return false;
- }
-
- std::transform(ipv6.begin(), ipv6.end(), ipv6.begin(),
- [](unsigned char ch) { return std::tolower(ch); });
- std::istringstream iss(ipv6);
- std::string field;
- uint16_t fields[8] = {0};
- uint8_t zero_index = 0;
- uint8_t num_field = 0;
- uint8_t right_field_num = 0;
-
- while (num_field < 8) {
- if (!getline(iss, field, ':')) {
- break;
- }
-
- if (field.empty()) {
- zero_index = num_field;
- fields[num_field++] = 0;
- } else {
- try {
- if (field.size() > 4 || field > "ffff") {
- return false;
- }
-
- fields[num_field++] = std::stoi(field, nullptr, 16);
- } catch (const std::exception& /*e*/) {
- return false;
- }
- }
- }
-
- if (zero_index != 0) {
- right_field_num = num_field - zero_index - 1;
-
- for (uint8_t i = 7; i > 7 - right_field_num; --i) {
- fields[i] = fields[zero_index + right_field_num + i - 7];
- fields[zero_index + right_field_num + i - 7] = 0;
- }
- }
-
- uint64_t high = (static_cast<uint64_t>(fields[0]) << 48) |
- (static_cast<uint64_t>(fields[1]) << 32) |
- (static_cast<uint64_t>(fields[2]) << 16) | static_cast<uint64_t>(fields[3]);
- uint64_t low = (static_cast<uint64_t>(fields[4]) << 48) |
- (static_cast<uint64_t>(fields[5]) << 32) |
- (static_cast<uint64_t>(fields[6]) << 16) | static_cast<uint64_t>(fields[7]);
-
- x = static_cast<vectorized::IPv6>(high) << 64 | low;
- return true;
- }
+ bool from_string(const std::string& ipv6_str) { return from_string(_value, ipv6_str); }
- static bool from_binary_string(vectorized::IPv6& x, std::string ipv6_binary_str) {
- // Accepts a FixedString(16) value containing the IPv6 address in binary format
- if (ipv6_binary_str.size() != 16) {
+ static bool from_string(vectorized::IPv6& value, const std::string& ipv6_str) {
+ if (ipv6_str.empty()) {
return false;
}
-
- uint64_t high = 0;
- uint64_t low = 0;
-
- const uint8_t* ipv6_binary = reinterpret_cast<const uint8_t*>(ipv6_binary_str.c_str());
-
- for (int i = 0; i < 8; ++i) {
- high |= (static_cast<uint64_t>(ipv6_binary[i]) << (56 - i * 8));
- }
-
- for (int i = 8; i < 16; ++i) {
- low |= (static_cast<uint64_t>(ipv6_binary[i]) << (56 - (i - 8) * 8));
- }
-
- x = static_cast<vectorized::IPv6>(high) << 64 | low;
- return true;
- }
-
- [[nodiscard]] std::string to_string() const { return to_string(_value); }
-
- static std::string to_string(vectorized::IPv6 x) {
- // "0000:0000:0000:0000:0000:0000:0000:0000"
- if (x == 0) {
- return "::";
- }
-
- uint64_t low = static_cast<uint64_t>(x);
- uint64_t high = static_cast<uint64_t>(x >> 64);
-
- uint16_t fields[8] = {static_cast<uint16_t>((high >> 48) & 0xFFFF),
- static_cast<uint16_t>((high >> 32) & 0xFFFF),
- static_cast<uint16_t>((high >> 16) & 0xFFFF),
- static_cast<uint16_t>(high & 0xFFFF),
- static_cast<uint16_t>((low >> 48) & 0xFFFF),
- static_cast<uint16_t>((low >> 32) & 0xFFFF),
- static_cast<uint16_t>((low >> 16) & 0xFFFF),
- static_cast<uint16_t>(low & 0xFFFF)};
-
- uint8_t zero_start = 0, zero_end = 0;
-
- while (zero_start < 8 && zero_end < 8) {
- if (fields[zero_start] != 0) {
- zero_start++;
- zero_end = zero_start;
- continue;
- }
-
- while (zero_end < 7 && fields[zero_end + 1] == 0) {
- zero_end++;
- }
-
- if (zero_end > zero_start) {
- break;
- }
-
- zero_start++;
- zero_end = zero_start;
- }
-
- std::stringstream ss;
-
- if (zero_start == zero_end) {
- for (uint8_t i = 0; i < 7; ++i) {
- ss << std::hex << fields[i] << ":";
- }
- ss << std::hex << fields[7];
- } else {
- for (uint8_t i = 0; i < zero_start; ++i) {
- ss << std::hex << fields[i] << ":";
- }
-
- if (zero_end == 7) {
- ss << ":";
- } else {
- for (uint8_t j = zero_end + 1; j < 8; ++j) {
- ss << std::hex << ":" << fields[j];
- }
- }
- }
-
- return ss.str();
- }
-
- [[nodiscard]] std::string to_binary_string() const { return to_binary_string(_value); }
-
- static std::string to_binary_string(vectorized::IPv6 x) {
- uint64_t low = static_cast<uint64_t>(x);
- uint64_t high = static_cast<uint64_t>(x >> 64);
-
- uint8_t fields[16] = {static_cast<uint8_t>((high >> 56) & 0xFF),
- static_cast<uint8_t>((high >> 48) & 0xFF),
- static_cast<uint8_t>((high >> 40) & 0xFF),
- static_cast<uint8_t>((high >> 32) & 0xFF),
- static_cast<uint8_t>((high >> 24) & 0xFF),
- static_cast<uint8_t>((high >> 16) & 0xFF),
- static_cast<uint8_t>((high >> 8) & 0xFF),
- static_cast<uint8_t>(high & 0xFF),
- static_cast<uint8_t>((low >> 56) & 0xFF),
- static_cast<uint8_t>((low >> 48) & 0xFF),
- static_cast<uint8_t>((low >> 40) & 0xFF),
- static_cast<uint8_t>((low >> 32) & 0xFF),
- static_cast<uint8_t>((low >> 24) & 0xFF),
- static_cast<uint8_t>((low >> 16) & 0xFF),
- static_cast<uint8_t>((low >> 8) & 0xFF),
- static_cast<uint8_t>(low & 0xFF)};
-
- std::stringstream ss;
-
- for (int i = 0; i < 16; ++i) {
- ss << (char)fields[i];
- }
-
- return ss.str();
+ const char* src = ipv6_str.c_str();
+ const char* end = ipv6_str.c_str() + ipv6_str.size() - 1;
+ while (std::isspace(*src)) ++src;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
while (std::isspace(*src)) { ++src;
}
```
##########
be/src/vec/runtime/ipv6_value.h:
##########
@@ -35,224 +34,35 @@
explicit IPv6Value(vectorized::IPv6 ipv6) { _value = ipv6; }
- [[nodiscard]] const vectorized::IPv6& value() const { return _value; }
+ const vectorized::IPv6& value() const { return _value; }
vectorized::IPv6& value() { return _value; }
void set_value(vectorized::IPv6 ipv6) { _value = ipv6; }
- bool from_string(std::string ipv6) { return from_string(_value, ipv6); }
-
- bool from_binary_string(std::string ipv6_binary) {
- return from_binary_string(_value, ipv6_binary);
- }
-
- static bool from_string(vectorized::IPv6& x, std::string ipv6) {
- remove_ipv6_space(ipv6);
-
- if (ipv6.empty() || !is_valid_string(ipv6)) {
- return false;
- }
-
- std::transform(ipv6.begin(), ipv6.end(), ipv6.begin(),
- [](unsigned char ch) { return std::tolower(ch); });
- std::istringstream iss(ipv6);
- std::string field;
- uint16_t fields[8] = {0};
- uint8_t zero_index = 0;
- uint8_t num_field = 0;
- uint8_t right_field_num = 0;
-
- while (num_field < 8) {
- if (!getline(iss, field, ':')) {
- break;
- }
-
- if (field.empty()) {
- zero_index = num_field;
- fields[num_field++] = 0;
- } else {
- try {
- if (field.size() > 4 || field > "ffff") {
- return false;
- }
-
- fields[num_field++] = std::stoi(field, nullptr, 16);
- } catch (const std::exception& /*e*/) {
- return false;
- }
- }
- }
-
- if (zero_index != 0) {
- right_field_num = num_field - zero_index - 1;
-
- for (uint8_t i = 7; i > 7 - right_field_num; --i) {
- fields[i] = fields[zero_index + right_field_num + i - 7];
- fields[zero_index + right_field_num + i - 7] = 0;
- }
- }
-
- uint64_t high = (static_cast<uint64_t>(fields[0]) << 48) |
- (static_cast<uint64_t>(fields[1]) << 32) |
- (static_cast<uint64_t>(fields[2]) << 16) | static_cast<uint64_t>(fields[3]);
- uint64_t low = (static_cast<uint64_t>(fields[4]) << 48) |
- (static_cast<uint64_t>(fields[5]) << 32) |
- (static_cast<uint64_t>(fields[6]) << 16) | static_cast<uint64_t>(fields[7]);
-
- x = static_cast<vectorized::IPv6>(high) << 64 | low;
- return true;
- }
+ bool from_string(const std::string& ipv6_str) { return from_string(_value, ipv6_str); }
- static bool from_binary_string(vectorized::IPv6& x, std::string ipv6_binary_str) {
- // Accepts a FixedString(16) value containing the IPv6 address in binary format
- if (ipv6_binary_str.size() != 16) {
+ static bool from_string(vectorized::IPv6& value, const std::string& ipv6_str) {
+ if (ipv6_str.empty()) {
return false;
}
-
- uint64_t high = 0;
- uint64_t low = 0;
-
- const uint8_t* ipv6_binary = reinterpret_cast<const uint8_t*>(ipv6_binary_str.c_str());
-
- for (int i = 0; i < 8; ++i) {
- high |= (static_cast<uint64_t>(ipv6_binary[i]) << (56 - i * 8));
- }
-
- for (int i = 8; i < 16; ++i) {
- low |= (static_cast<uint64_t>(ipv6_binary[i]) << (56 - (i - 8) * 8));
- }
-
- x = static_cast<vectorized::IPv6>(high) << 64 | low;
- return true;
- }
-
- [[nodiscard]] std::string to_string() const { return to_string(_value); }
-
- static std::string to_string(vectorized::IPv6 x) {
- // "0000:0000:0000:0000:0000:0000:0000:0000"
- if (x == 0) {
- return "::";
- }
-
- uint64_t low = static_cast<uint64_t>(x);
- uint64_t high = static_cast<uint64_t>(x >> 64);
-
- uint16_t fields[8] = {static_cast<uint16_t>((high >> 48) & 0xFFFF),
- static_cast<uint16_t>((high >> 32) & 0xFFFF),
- static_cast<uint16_t>((high >> 16) & 0xFFFF),
- static_cast<uint16_t>(high & 0xFFFF),
- static_cast<uint16_t>((low >> 48) & 0xFFFF),
- static_cast<uint16_t>((low >> 32) & 0xFFFF),
- static_cast<uint16_t>((low >> 16) & 0xFFFF),
- static_cast<uint16_t>(low & 0xFFFF)};
-
- uint8_t zero_start = 0, zero_end = 0;
-
- while (zero_start < 8 && zero_end < 8) {
- if (fields[zero_start] != 0) {
- zero_start++;
- zero_end = zero_start;
- continue;
- }
-
- while (zero_end < 7 && fields[zero_end + 1] == 0) {
- zero_end++;
- }
-
- if (zero_end > zero_start) {
- break;
- }
-
- zero_start++;
- zero_end = zero_start;
- }
-
- std::stringstream ss;
-
- if (zero_start == zero_end) {
- for (uint8_t i = 0; i < 7; ++i) {
- ss << std::hex << fields[i] << ":";
- }
- ss << std::hex << fields[7];
- } else {
- for (uint8_t i = 0; i < zero_start; ++i) {
- ss << std::hex << fields[i] << ":";
- }
-
- if (zero_end == 7) {
- ss << ":";
- } else {
- for (uint8_t j = zero_end + 1; j < 8; ++j) {
- ss << std::hex << ":" << fields[j];
- }
- }
- }
-
- return ss.str();
- }
-
- [[nodiscard]] std::string to_binary_string() const { return to_binary_string(_value); }
-
- static std::string to_binary_string(vectorized::IPv6 x) {
- uint64_t low = static_cast<uint64_t>(x);
- uint64_t high = static_cast<uint64_t>(x >> 64);
-
- uint8_t fields[16] = {static_cast<uint8_t>((high >> 56) & 0xFF),
- static_cast<uint8_t>((high >> 48) & 0xFF),
- static_cast<uint8_t>((high >> 40) & 0xFF),
- static_cast<uint8_t>((high >> 32) & 0xFF),
- static_cast<uint8_t>((high >> 24) & 0xFF),
- static_cast<uint8_t>((high >> 16) & 0xFF),
- static_cast<uint8_t>((high >> 8) & 0xFF),
- static_cast<uint8_t>(high & 0xFF),
- static_cast<uint8_t>((low >> 56) & 0xFF),
- static_cast<uint8_t>((low >> 48) & 0xFF),
- static_cast<uint8_t>((low >> 40) & 0xFF),
- static_cast<uint8_t>((low >> 32) & 0xFF),
- static_cast<uint8_t>((low >> 24) & 0xFF),
- static_cast<uint8_t>((low >> 16) & 0xFF),
- static_cast<uint8_t>((low >> 8) & 0xFF),
- static_cast<uint8_t>(low & 0xFF)};
-
- std::stringstream ss;
-
- for (int i = 0; i < 16; ++i) {
- ss << (char)fields[i];
- }
-
- return ss.str();
+ const char* src = ipv6_str.c_str();
+ const char* end = ipv6_str.c_str() + ipv6_str.size() - 1;
+ while (std::isspace(*src)) ++src;
+ while (std::isspace(*end)) --end;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
while (std::isspace(*end)) { --end;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "sjyango (via GitHub)" <gi...@apache.org>.
sjyango commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869389515
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "sjyango (via GitHub)" <gi...@apache.org>.
sjyango commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869233738
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869278075
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.02 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 38.4 seconds inserted 10000000 Rows, about 260K ops/s
storage size: 17183762016 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869335098
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 3c79573f737f57eb1e429247dce2e70dbd3e0b2f, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4704 4465 4446 4446
q2 395 160 161 160
q3 1472 1322 1233 1233
q4 1151 989 899 899
q5 3155 3159 3149 3149
q6 265 135 135 135
q7 1054 484 490 484
q8 2251 2276 2254 2254
q9 6719 6668 6699 6668
q10 3236 3291 3282 3282
q11 324 207 207 207
q12 362 205 211 205
q13 4569 3844 3800 3800
q14 241 214 214 214
q15 568 520 530 520
q16 434 385 384 384
q17 1042 787 554 554
q18 7030 6884 6874 6874
q19 1572 1582 1617 1582
q20 532 304 297 297
q21 3250 2794 2754 2754
q22 375 302 297 297
Total cold run time: 44701 ms
Total hot run time: 40398 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4405 4389 4350 4350
q2 268 166 161 161
q3 3502 3500 3472 3472
q4 2428 2403 2410 2403
q5 5700 5736 5703 5703
q6 251 127 127 127
q7 2402 1885 1862 1862
q8 3600 3605 3590 3590
q9 9032 8980 8942 8942
q10 3900 3985 3994 3985
q11 486 374 366 366
q12 768 664 603 603
q13 4271 3542 3560 3542
q14 293 257 252 252
q15 572 522 520 520
q16 496 472 449 449
q17 1954 1956 1937 1937
q18 8668 8091 8186 8091
q19 1810 1821 1797 1797
q20 2253 1957 1926 1926
q21 6588 6213 6224 6213
q22 543 468 466 466
Total cold run time: 64190 ms
Total hot run time: 60757 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869332128
TeamCity be ut coverage result:
Function Coverage: 36.62% (8564/23386)
Line Coverage: 28.69% (69645/242773)
Region Coverage: 27.71% (36040/130083)
Branch Coverage: 24.44% (18427/75406)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3c79573f737f57eb1e429247dce2e70dbd3e0b2f_3c79573f737f57eb1e429247dce2e70dbd3e0b2f/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "sjyango (via GitHub)" <gi...@apache.org>.
sjyango commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869364937
run PipelineX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #29044:
URL: https://github.com/apache/doris/pull/29044#discussion_r1436350417
##########
be/src/vec/common/format_ip.h:
##########
@@ -20,7 +20,9 @@
#pragma once
+#include <vec/common/hex.h>
#include <vec/common/string_utils/string_utils.h>
+#include <vec/core/types.h>
Review Comment:
why include this header?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "sjyango (via GitHub)" <gi...@apache.org>.
sjyango commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869238483
run arm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869927363
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 0ae016e23ca8807c1363f2a2380cece3dca701b8, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4772 4442 4450 4442
q2 374 141 158 141
q3 1476 1329 1175 1175
q4 1154 953 976 953
q5 3190 3170 3163 3163
q6 260 137 136 136
q7 1027 502 486 486
q8 2239 2277 2242 2242
q9 6737 6685 6720 6685
q10 3222 3283 3310 3283
q11 323 205 210 205
q12 361 208 211 208
q13 4564 3811 3758 3758
q14 247 212 210 210
q15 587 529 530 529
q16 446 387 381 381
q17 1035 800 615 615
q18 6991 6731 6849 6731
q19 1563 1596 1593 1593
q20 561 321 316 316
q21 3211 2767 2744 2744
q22 368 307 312 307
Total cold run time: 44708 ms
Total hot run time: 40303 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4406 4417 4367 4367
q2 269 167 166 166
q3 3516 3511 3505 3505
q4 2442 2431 2425 2425
q5 5731 5704 5728 5704
q6 250 126 124 124
q7 2401 1830 1866 1830
q8 3608 3606 3601 3601
q9 8982 8984 8974 8974
q10 3945 4011 4013 4011
q11 478 370 378 370
q12 772 601 626 601
q13 4281 3564 3531 3531
q14 288 263 269 263
q15 567 520 519 519
q16 499 471 457 457
q17 1986 1948 1954 1948
q18 8591 8341 8320 8320
q19 1822 1846 1831 1831
q20 2248 1953 1947 1947
q21 6624 6272 6208 6208
q22 553 480 467 467
Total cold run time: 64259 ms
Total hot run time: 61169 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869467690
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 941d67dcd87def4d6ef08bf502444482b244499a, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4764 4466 4459 4459
q2 390 169 153 153
q3 1485 1309 1228 1228
q4 1156 1010 962 962
q5 3173 3148 3176 3148
q6 257 134 131 131
q7 1032 482 495 482
q8 2298 2285 2250 2250
q9 6727 6685 6717 6685
q10 3227 3291 3274 3274
q11 331 217 211 211
q12 361 207 214 207
q13 4532 3820 3791 3791
q14 236 213 217 213
q15 580 511 518 511
q16 447 389 377 377
q17 1034 772 627 627
q18 7113 6764 6846 6764
q19 1586 1599 1610 1599
q20 531 313 288 288
q21 3294 2792 2785 2785
q22 368 299 312 299
Total cold run time: 44922 ms
Total hot run time: 40444 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4408 4408 4392 4392
q2 274 168 171 168
q3 3535 3500 3486 3486
q4 2449 2430 2434 2430
q5 5738 5737 5716 5716
q6 251 126 123 123
q7 2380 1874 1878 1874
q8 3616 3617 3616 3616
q9 8992 8952 8951 8951
q10 3918 4004 4044 4004
q11 497 380 372 372
q12 772 596 607 596
q13 4265 3539 3532 3532
q14 291 262 252 252
q15 572 523 530 523
q16 504 435 450 435
q17 1982 1963 1950 1950
q18 8640 8205 8244 8205
q19 1837 1830 1835 1830
q20 2250 1954 1951 1951
q21 6661 6290 6268 6268
q22 552 453 464 453
Total cold run time: 64384 ms
Total hot run time: 61127 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869259494
TeamCity be ut coverage result:
Function Coverage: 36.61% (8562/23385)
Line Coverage: 28.68% (69636/242765)
Region Coverage: 27.70% (36029/130080)
Branch Coverage: 24.43% (18422/75406)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e92d02b8b7260f79e6cdf8fe20dbce50c52ed00f_e92d02b8b7260f79e6cdf8fe20dbce50c52ed00f/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](datatype) Update Parser for IPv4/v6 data types [doris]
Posted by "sjyango (via GitHub)" <gi...@apache.org>.
sjyango commented on PR #29044:
URL: https://github.com/apache/doris/pull/29044#issuecomment-1869912757
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org