You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaka11chen (via GitHub)" <gi...@apache.org> on 2023/11/24 12:14:54 UTC
[PR] [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. [doris]
kaka11chen opened a new pull request, #27563:
URL: https://github.com/apache/doris/pull/27563
## Proposed changes
### Issue:
```
2023-11-24 17:59:16,171 WARN (mysql-nio-pool-0|236) [HiveScanNode.getSplits():211] get file split failed for table: lineitem
org.apache.doris.datasource.CacheException: failed to get input splits for write ids acid.lineitem:9223372036854775807:9223372036854775807:: in catalog regression1
at org.apache.doris.datasource.hive.HiveMetaStoreCache.getFilesByTransaction(HiveMetaStoreCache.java:863) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.HiveScanNode.getFileSplitByTransaction(HiveScanNode.java:308) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.HiveScanNode.getFileSplitByPartitions(HiveScanNode.java:223) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.HiveScanNode.getSplits(HiveScanNode.java:206) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.FileQueryScanNode.createScanRangeLocations(FileQueryScanNode.java:270) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.FileQueryScanNode.doFinalize(FileQueryScanNode.java:229) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:221) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.util.Utils.execWithUncheckedException(Utils.java:67) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalFileScan(PhysicalPlanTranslator.java:522) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalFileScan(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalFileScan.accept(PhysicalFileScan.java:98) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:1568) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalProject.accept(PhysicalProject.java:105) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:784) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalHashAggregate.accept(PhysicalHashAggregate.java:185) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:265) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalDistribute.accept(PhysicalDistribute.java:96) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:784) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalHashAggregate.accept(PhysicalHashAggregate.java:185) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:355) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalResultSink.accept(PhysicalResultSink.java:70) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:239) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:134) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:580) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:451) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:441) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:358) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:511) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:762) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]
Caused by: java.lang.NullPointerException
at org.apache.hadoop.fs.Path.<init>(Path.java:150) ~[hadoop-common-3.3.6.jar:?]
at org.apache.hadoop.fs.Path.<init>(Path.java:129) ~[hadoop-common-3.3.6.jar:?]
at org.apache.doris.datasource.hive.HiveMetaStoreCache.getFilesByTransaction(HiveMetaStoreCache.java:805) ~[doris-fe.jar:1.2-SNAPSHOT]
... 37 more
```
### Solution
Fix NPE when query empty hive transactional table.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27563:
URL: https://github.com/apache/doris/pull/27563#issuecomment-1825793653
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27563:
URL: https://github.com/apache/doris/pull/27563#issuecomment-1825793588
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27563:
URL: https://github.com/apache/doris/pull/27563#issuecomment-1825746137
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 44eb2032557155872f57c793cf431f80b2288dca, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4905 4636 4630 4630
q2 365 141 140 140
q3 2011 1898 1896 1896
q4 1383 1258 1236 1236
q5 3993 3922 3973 3922
q6 255 128 131 128
q7 1397 885 869 869
q8 2784 2789 2757 2757
q9 9644 9544 9403 9403
q10 3473 3524 3544 3524
q11 375 240 251 240
q12 435 293 292 292
q13 4577 3809 3805 3805
q14 321 290 297 290
q15 589 525 511 511
q16 660 581 582 581
q17 1134 939 907 907
q18 7845 7513 7447 7447
q19 1667 1689 1658 1658
q20 537 318 292 292
q21 4409 3933 4017 3933
q22 484 380 379 379
Total cold run time: 53243 ms
Total hot run time: 48840 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4617 4604 4572 4572
q2 351 244 260 244
q3 4040 4027 4025 4025
q4 2718 2700 2708 2700
q5 9860 9904 9914 9904
q6 248 119 124 119
q7 3023 2510 2459 2459
q8 4466 4438 4457 4438
q9 12990 12839 12834 12834
q10 4093 4187 4148 4148
q11 769 661 664 661
q12 971 807 830 807
q13 4313 3615 3574 3574
q14 381 369 342 342
q15 573 518 526 518
q16 738 655 700 655
q17 3847 3892 3827 3827
q18 9690 9144 9004 9004
q19 1816 1779 1800 1779
q20 2394 2046 2025 2025
q21 8812 8519 8521 8519
q22 920 852 778 778
Total cold run time: 81630 ms
Total hot run time: 77932 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on PR #27563:
URL: https://github.com/apache/doris/pull/27563#issuecomment-1825598882
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #27563:
URL: https://github.com/apache/doris/pull/27563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on PR #27563:
URL: https://github.com/apache/doris/pull/27563#issuecomment-1825592126
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27563:
URL: https://github.com/apache/doris/pull/27563#issuecomment-1825743055
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.83 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17100689076 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org