You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/11/29 04:02:40 UTC
[GitHub] [doris] xiaoDjun opened a new pull request, #14661: [fix](spark load)partition column is not duplicate key, spark load IndexOutOfBoundsException error
xiaoDjun opened a new pull request, #14661:
URL: https://github.com/apache/doris/pull/14661
# Proposed changes
Issue Number: close #14600
## Problem summary
In spark load, the job failed when the partition column is not in the duplicate key column list.
We traced the sparkDpp.java and found that, the partition columns ware built with the key index in base metadata. But when partition column was not key, an IndexOutOfBoundsException occurs when obtaining it through DppColumns.
The exception is:
```
java.lang.IndexOutOfBoundsException: Index: 3, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at org.apache.doris.load.loadv2.dpp.DppColumns.<init>(DppColumns.java:39)
at org.apache.doris.load.loadv2.dpp.DorisRangePartitioner.getPartition(DorisRangePartitioner.java:56)
at org.apache.doris.load.loadv2.dpp.SparkDpp$2.call(SparkDpp.java:487)
at org.apache.doris.load.loadv2.dpp.SparkDpp$2.call(SparkDpp.java:460)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
```
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14661: [fix](spark load)partition column is not duplicate key, spark load IndexOutOfBounds error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14661:
URL: https://github.com/apache/doris/pull/14661#issuecomment-1330046709
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14661: [fix](spark load)partition column is not duplicate key, spark load IndexOutOfBounds error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14661:
URL: https://github.com/apache/doris/pull/14661#issuecomment-1330046739
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #14661: [fix](spark load)partition column is not duplicate key, spark load IndexOutOfBounds error
Posted by GitBox <gi...@apache.org>.
yiguolei merged PR #14661:
URL: https://github.com/apache/doris/pull/14661
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14661: [fix](spark load)partition column is not duplicate key, spark load IndexOutOfBounds error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14661:
URL: https://github.com/apache/doris/pull/14661#issuecomment-1330067855
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14661: [fix](spark load)partition column is not duplicate key, spark load IndexOutOfBounds error
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #14661:
URL: https://github.com/apache/doris/pull/14661#issuecomment-1330090451
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.61 seconds
load time: 486 seconds
storage size: 17123333158 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221129051721_clickbench_pr_54806.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org