You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/08/30 10:18:02 UTC
[GitHub] [doris] hf200012 opened a new pull request, #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2
hf200012 opened a new pull request, #12188:
URL: https://github.com/apache/doris/pull/12188
# Proposed changes
Spark load supports version 3.x, default supports 3.2.2
Spark load currently supports version 2.x, spark 3.x has been released to 3.3.0, we need to abandon support for version 2.x, because version 3.x in the ExpressionEncoder toRow method is no longer supported, so the two versions can not be compatible, can only choose one of them
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] spaces-X commented on a diff in pull request #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2
Posted by GitBox <gi...@apache.org>.
spaces-X commented on code in PR #12188:
URL: https://github.com/apache/doris/pull/12188#discussion_r958318305
##########
fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java:
##########
@@ -266,7 +266,7 @@ private void writeRepartitionAndSortedRDDToParquet(JavaPairRDD<List<Object>, Obj
}
lastBucketKey = curBucketKey;
}
- InternalRow internalRow = encoder.toRow(rowWithoutBucketKey);
+ InternalRow internalRow = encoder.createDeserializer().apply(rowWithoutBucketKey);
Review Comment:
```suggestion
InternalRow internalRow = encoder.createSerializer().apply(rowWithoutBucketKey);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dujl commented on pull request #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2
Posted by GitBox <gi...@apache.org>.
dujl commented on PR #12188:
URL: https://github.com/apache/doris/pull/12188#issuecomment-1231483528
update spark version directly will result in some user can not use spark-load because they only have spark2 runtime env.
how about create a new module spark3-dpp to support spark3 env?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] spaces-X commented on pull request #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2
Posted by GitBox <gi...@apache.org>.
spaces-X commented on PR #12188:
URL: https://github.com/apache/doris/pull/12188#issuecomment-1238909424
Some ideas to supoort the multi-version spark engine in spark load:
1. The spark load module is relatively independent and has the characteristics of low update frequency. So maybe we can just update and maintain the fat jar package.
2. For each future version, e.g. 1.x.x, we only maintain and update the latest jar package.
3. For the elder version(such as spark 2.x), we just package it into a jar and upload it to the official website.
4. Based on 1 and 2, the spark-dpp build phase can skipped by default when building fe. Instead, the corresponding spark-dpp jar package can be download by enviroment variable url in `build.sh` and users can choose not to upgrade the spark-dpp jar package when upgrading fe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hf200012 commented on a diff in pull request #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2
Posted by GitBox <gi...@apache.org>.
hf200012 commented on code in PR #12188:
URL: https://github.com/apache/doris/pull/12188#discussion_r958321699
##########
fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java:
##########
@@ -266,7 +266,7 @@ private void writeRepartitionAndSortedRDDToParquet(JavaPairRDD<List<Object>, Obj
}
lastBucketKey = curBucketKey;
}
- InternalRow internalRow = encoder.toRow(rowWithoutBucketKey);
+ InternalRow internalRow = encoder.createDeserializer().apply(rowWithoutBucketKey);
Review Comment:
Just wrote a mistake has been corrected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dujl commented on pull request #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2
Posted by GitBox <gi...@apache.org>.
dujl commented on PR #12188:
URL: https://github.com/apache/doris/pull/12188#issuecomment-1239010649
> Some ideas to supoort the multi-version spark engine in spark load:
>
> 1. The spark load module is relatively independent and has the characteristics of low update frequency. So maybe we can just update and maintain the fat jar package.
> 2. For each future version, e.g. 1.x.x, we only maintain and update the latest jar package.
> 3. For the elder version(such as spark 2.x), we just package it into a jar and upload it to the official website.
> 4. Based on 1 and 2, the spark-dpp build phase can skipped by default when building fe. Instead, the corresponding spark-dpp jar package can be download by enviroment variable url in `build.sh` and users can choose not to upgrade the spark-dpp jar package when upgrading fe.
The spark runtime environment is very complex. Different user maybe has different version. So It would be great if we could provide flexible version dependencie.
For case 1&2, There may be multiple spark versions in for a foreseeable period of timeļ¼Such as current mainstream version of spark includes spark2.x and spark3.x. So it is very great to provide spark 2.x and spark3.x dependencie with different compile profile. Like hudi/iceberg.
For case 3, some user maybe need compile the spark-dpp with their own version which major version same as our the elder version(such as spark 2.x). It is very inconvenient if we can only provide binary packages on website.
For case 4, agree with u to move spark-dpp out of the doris' core. But some compile env can not down package from the extranet
My idea is:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12188:
URL: https://github.com/apache/doris/pull/12188#issuecomment-1520976891
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] closed pull request #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2
URL: https://github.com/apache/doris/pull/12188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #12188: [improvement](spark-load)Spark load supports version 3.x, default supports 3.2.2
Posted by GitBox <gi...@apache.org>.
morningman commented on PR #12188:
URL: https://github.com/apache/doris/pull/12188#issuecomment-1231638970
I think its time move the Spark Load out of the Doris' core.
The compatibility issue of Spark and other Hadoop related components is terrible and it is not suitable to be in Doris' core code any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org