You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/09/03 12:05:07 UTC
[GitHub] [incubator-doris] xy720 opened a new pull request #4532: [Spark Load][Bug] Keep the column splitting in spark load consistent with broker load / mini load
xy720 opened a new pull request #4532:
URL: https://github.com/apache/incubator-doris/pull/4532
## Proposed changes
There is a 4 columns source data:
```
1|1|jim|2|
2|1|grace|2|
3|2|tom|2|
4|3|bush|3|
5|3|helen|3|
5|3|helen|6|
6|3|helen|3|
6|3|helen|3|
...
```
Given the same column terminator '|', broker load determines that it is 5 columns, and spark load determines that it is 4 columns.
And there is another 4 columns source
```
1|1|jim|2
2|1|grace|2
3|2|tom|2
4|3|bush|3
5|3|helen|3
5|3|helen|6
6|3|helen|3
6|3|helen|3
...
```
Given the same column terminator '|', both the broker load and spark load determines that it is 4 columns.
**To Reproduce**
Steps to reproduce the behavior:
1. Submit a broker load.
```
load label ssb_db.broker_load_label
(
data infile ("hdfs://ymy-host:port/user/palo/table1")
into table test_tbl
COLUMNS TERMINATED BY "|"
(k1,k2,name,clicks )
)
with broker "doris" ("username" = "test", "password" = "test");
```
2. Submit a spark load.
```
load label ssb_db.spark_load_label
( data infile ("hdfs://ymy-host:port/user/palo/table1")
into table test_tbl
COLUMNS TERMINATED BY "|"
(k1,k2,name,clicks )
) with resource "spark0"
("spark.executor.memory" = "24g", "spark.executor.cores" = "2", "spark.executor.instances" = "8");
```
3. Broker Load will report an error "quality not good enough to cancel"
**The reson of this bug**
This is because the first character and the last character of a line are not considered to be delimeter in spark load.
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [x] Bugfix (non-breaking change which fixes an issue)
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [x] I have create an issue on (Fix #ISSUE), and have described the bug/feature there in detail
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #4532: [Spark Load][Bug] Keep the column splitting in spark load consistent with broker load / mini load
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #4532:
URL: https://github.com/apache/incubator-doris/pull/4532
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #4532: [Spark Load][Bug] Keep the column splitting in spark load consistent with broker load / mini load
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #4532:
URL: https://github.com/apache/incubator-doris/pull/4532#discussion_r483352298
##########
File path: fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
##########
@@ -640,6 +642,22 @@ private StructType createScrSchema(List<String> srcColumns) {
return srcSchema;
}
+ // This method is to keep the splitting consistent with broker load / mini load
+ private String[] splitLine(String line, char sep) {
Review comment:
fixed it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #4532: [Spark Load][Bug] Keep the column splitting in spark load consistent with broker load / mini load
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #4532:
URL: https://github.com/apache/incubator-doris/pull/4532#discussion_r483064082
##########
File path: fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
##########
@@ -640,6 +642,22 @@ private StructType createScrSchema(List<String> srcColumns) {
return srcSchema;
}
+ // This method is to keep the splitting consistent with broker load / mini load
+ private String[] splitLine(String line, char sep) {
Review comment:
if `line` is an empty string, this method should return an empty string array.
But here you will return a string array with one empty string in it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org