You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/07/18 10:22:31 UTC
[GitHub] [doris] SaintBacchus opened a new pull request, #10977: [feature-wip][multi-catalog] Support s3 storage for file scan node
SaintBacchus opened a new pull request, #10977:
URL: https://github.com/apache/doris/pull/10977
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #10977: [feature-wip][multi-catalog] Support s3 storage for file scan node
Posted by GitBox <gi...@apache.org>.
morningman commented on PR #10977:
URL: https://github.com/apache/doris/pull/10977#issuecomment-1189226471
hi @SaintBacchus , we had just fix the p0 test, please rebase to pass it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10977: [feature-wip][multi-catalog] Support s3 storage for file scan node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10977:
URL: https://github.com/apache/doris/pull/10977#issuecomment-1190192244
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10977: [feature-wip][multi-catalog] Support s3 storage for file scan node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10977:
URL: https://github.com/apache/doris/pull/10977#issuecomment-1190192296
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SaintBacchus commented on pull request #10977: [feature-wip][multi-catalog] Support s3 storage for file scan node
Posted by GitBox <gi...@apache.org>.
SaintBacchus commented on PR #10977:
URL: https://github.com/apache/doris/pull/10977#issuecomment-1189728618
OK
> 2022年7月19日 23:53，Mingyu Chen ***@***.***> 写道：
>
>
> hi @SaintBacchus <https://github.com/SaintBacchus> , we had just fix the p0 test, please rebase to pass it.
>
> —
> Reply to this email directly, view it on GitHub <https://github.com/apache/doris/pull/10977#issuecomment-1189226471>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYP2GBDNFIDNBKEX7FAKNLVU3FOLANCNFSM533XH66Q>.
> You are receiving this because you were mentioned.
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #10977: [feature-wip][multi-catalog] Support s3 storage for file scan node
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #10977:
URL: https://github.com/apache/doris/pull/10977#discussion_r923281229
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/ExternalHiveScanProvider.java:
##########
@@ -69,7 +69,13 @@ public TFileFormatType getTableFormatType() throws DdlException, MetaNotFoundExc
}
@Override
- public TFileType getTableFileType() {
+ public TFileType getTableFileType() throws MetaNotFoundException {
+ String location = hmsTable.getRemoteTable().getSd().getLocation();
+ if (location != null && !location.isEmpty()) {
+ if (location.startsWith("s3a") || location.startsWith("s3n")) {
+ return TFileType.FILE_S3;
+ }
+ }
Review Comment:
better to use `if else` to check the location for s3 and hdfs strictly, and throw exception if meeting unknown file type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SaintBacchus commented on a diff in pull request #10977: [feature-wip][multi-catalog] Support s3 storage for file scan node
Posted by GitBox <gi...@apache.org>.
SaintBacchus commented on code in PR #10977:
URL: https://github.com/apache/doris/pull/10977#discussion_r923336943
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/ExternalHiveScanProvider.java:
##########
@@ -69,7 +69,13 @@ public TFileFormatType getTableFormatType() throws DdlException, MetaNotFoundExc
}
@Override
- public TFileType getTableFileType() {
+ public TFileType getTableFileType() throws MetaNotFoundException {
+ String location = hmsTable.getRemoteTable().getSd().getLocation();
+ if (location != null && !location.isEmpty()) {
+ if (location.startsWith("s3a") || location.startsWith("s3n")) {
+ return TFileType.FILE_S3;
+ }
+ }
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #10977: [feature-wip][multi-catalog] Support s3 storage for file scan node
Posted by GitBox <gi...@apache.org>.
morningman merged PR #10977:
URL: https://github.com/apache/doris/pull/10977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org