You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wsjz (via GitHub)" <gi...@apache.org> on 2023/05/18 10:05:07 UTC
[GitHub] [doris] wsjz opened a new pull request, #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
wsjz opened a new pull request, #19819:
URL: https://github.com/apache/doris/pull/19819
# Proposed changes
Issue Number: #19679
## Problem summary
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1574693710
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1552833367
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1562828449
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1566836793
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1558930137
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1560802216
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1563720987
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1569612313
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] AshinGau commented on a diff in pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "AshinGau (via GitHub)" <gi...@apache.org>.
AshinGau commented on code in PR #19819:
URL: https://github.com/apache/doris/pull/19819#discussion_r1208992556
##########
be/src/vec/exec/scan/vfile_scanner.cpp:
##########
@@ -577,6 +578,15 @@ Status VFileScanner::_get_next_reader() {
Status init_status;
// TODO: use data lake type
switch (_params.format_type) {
+ case TFileFormatType::FORMAT_JNI: {
Review Comment:
Two layer nested logic:
```
if (TFileFormatType::FORMAT_JNI) {
// do jni scanner
if (range.table_format_params.table_format_type == "maxcompute") {
// max compute
} else if (range.table_format_params.table_format_type == "hudi") {
// hudi
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1558859783
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1568297533
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #19819:
URL: https://github.com/apache/doris/pull/19819#discussion_r1214048572
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/MaxComputeScanNode.java:
##########
@@ -74,7 +78,37 @@ protected Map<String, String> getLocationProperties() throws UserException {
@Override
protected List<Split> getSplits() throws UserException {
List<Split> result = new ArrayList<>();
- result.add(new FileSplit(new Path("/"), 0, -1, -1, 0L, new String[0], Collections.emptyList()));
+ // String splitPath = catalog.getTunnelUrl();
+ // TODO: use single max compute scan node rather than file scan node
+ com.aliyun.odps.Table odpsTable = table.getOdpsTable();
+ if (desc.getSlots().isEmpty() || odpsTable.getFileNum() <= 0) {
+ return result;
+ }
+ try {
+ List<Pair<Long, Long>> sliceRange = new ArrayList<>();
+ long totalRows = catalog.getTotalRows(table.getDbName(), table.getName());
Review Comment:
access metaservice， just call when catalog init
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #19819:
URL: https://github.com/apache/doris/pull/19819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1569630786
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1563722869
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1560804102
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] AshinGau commented on a diff in pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "AshinGau (via GitHub)" <gi...@apache.org>.
AshinGau commented on code in PR #19819:
URL: https://github.com/apache/doris/pull/19819#discussion_r1208983499
##########
fe/java-udf/src/main/java/org/apache/doris/jni/vec/ScanPredicate.java:
##########
@@ -119,6 +119,11 @@ public String toString() {
return inspectObject().toString();
}
+ @Override
+ public boolean isNull() {
+ return (boolean) inspectObject();
Review Comment:
`isNull` is not `getBoolean`.
##########
fe/java-udf/src/main/java/org/apache/doris/jni/vec/VectorColumn.java:
##########
@@ -556,7 +559,8 @@ public void appendValue(ColumnValue o) {
case DECIMAL32:
case DECIMAL64:
case DECIMAL128:
- appendDecimal(o.getDecimal());
Review Comment:
Why add an intermediate result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] AshinGau commented on a diff in pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "AshinGau (via GitHub)" <gi...@apache.org>.
AshinGau commented on code in PR #19819:
URL: https://github.com/apache/doris/pull/19819#discussion_r1209938312
##########
fe/java-udf/src/main/java/org/apache/doris/jni/vec/ScanPredicate.java:
##########
@@ -119,6 +119,11 @@ public String toString() {
return inspectObject().toString();
}
+ @Override
+ public boolean isNull() {
+ return valueBytes == null || valueBytes.length == 0;
Review Comment:
return false always
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1574605992
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1563771198
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #19819:
URL: https://github.com/apache/doris/pull/19819#discussion_r1214047256
##########
be/src/vec/exec/jni_connector.cpp:
##########
@@ -63,14 +63,15 @@ JniConnector::~JniConnector() {
}
Status JniConnector::open(RuntimeState* state, RuntimeProfile* profile) {
- RETURN_IF_ERROR(JniUtil::GetJNIEnv(&_env));
- if (_env == nullptr) {
+ JNIEnv* env = nullptr;
+ RETURN_IF_ERROR(JniUtil::GetJNIEnv(&env));
+ if (env == nullptr) {
return Status::InternalError("Failed to get/create JVM");
}
- RETURN_IF_ERROR(_init_jni_scanner(_env, state->batch_size()));
+ RETURN_IF_ERROR(_init_jni_scanner(env, state->batch_size()));
// Call org.apache.doris.jni.JniScanner#open
- _env->CallVoidMethod(_jni_scanner_obj, _jni_scanner_open);
- RETURN_ERROR_IF_EXC(_env);
+ env->CallVoidMethod(_jni_scanner_obj, _jni_scanner_open);
+ RETURN_ERROR_IF_EXC(env);
Review Comment:
env is thread local in jvm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1568240814
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1568208799
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1567624520
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wsjz commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1569616634
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #19819:
URL: https://github.com/apache/doris/pull/19819#discussion_r1214018756
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/MaxComputeScanNode.java:
##########
@@ -74,7 +78,37 @@ protected Map<String, String> getLocationProperties() throws UserException {
@Override
protected List<Split> getSplits() throws UserException {
List<Split> result = new ArrayList<>();
- result.add(new FileSplit(new Path("/"), 0, -1, -1, 0L, new String[0], Collections.emptyList()));
+ // String splitPath = catalog.getTunnelUrl();
+ // TODO: use single max compute scan node rather than file scan node
+ com.aliyun.odps.Table odpsTable = table.getOdpsTable();
+ if (desc.getSlots().isEmpty() || odpsTable.getFileNum() <= 0) {
+ return result;
+ }
+ try {
+ List<Pair<Long, Long>> sliceRange = new ArrayList<>();
+ long totalRows = catalog.getTotalRows(table.getDbName(), table.getName());
+ long fileNum = odpsTable.getFileNum();
+ long splitSize = (long) Math.ceil((double) totalRows / fileNum);
Review Comment:
If the `totalRows` is less than `fileNum`, the `splitSize` will be 0.
##########
fe/java-udf/src/main/java/org/apache/doris/jni/vec/VectorColumn.java:
##########
@@ -21,6 +21,8 @@
import org.apache.doris.jni.utils.TypeNativeBytes;
import org.apache.doris.jni.vec.ColumnType.Type;
+import org.apache.log4j.Logger;
Review Comment:
Wrong logger
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/MaxComputeScanNode.java:
##########
@@ -74,7 +78,37 @@ protected Map<String, String> getLocationProperties() throws UserException {
@Override
protected List<Split> getSplits() throws UserException {
List<Split> result = new ArrayList<>();
- result.add(new FileSplit(new Path("/"), 0, -1, -1, 0L, new String[0], Collections.emptyList()));
+ // String splitPath = catalog.getTunnelUrl();
+ // TODO: use single max compute scan node rather than file scan node
+ com.aliyun.odps.Table odpsTable = table.getOdpsTable();
+ if (desc.getSlots().isEmpty() || odpsTable.getFileNum() <= 0) {
+ return result;
+ }
+ try {
+ List<Pair<Long, Long>> sliceRange = new ArrayList<>();
+ long totalRows = catalog.getTotalRows(table.getDbName(), table.getName());
Review Comment:
Is this `getTotalRows()` method expensive?
##########
be/src/vec/exec/jni_connector.cpp:
##########
@@ -63,14 +63,15 @@ JniConnector::~JniConnector() {
}
Status JniConnector::open(RuntimeState* state, RuntimeProfile* profile) {
- RETURN_IF_ERROR(JniUtil::GetJNIEnv(&_env));
- if (_env == nullptr) {
+ JNIEnv* env = nullptr;
+ RETURN_IF_ERROR(JniUtil::GetJNIEnv(&env));
+ if (env == nullptr) {
return Status::InternalError("Failed to get/create JVM");
}
- RETURN_IF_ERROR(_init_jni_scanner(_env, state->batch_size()));
+ RETURN_IF_ERROR(_init_jni_scanner(env, state->batch_size()));
// Call org.apache.doris.jni.JniScanner#open
- _env->CallVoidMethod(_jni_scanner_obj, _jni_scanner_open);
- RETURN_ERROR_IF_EXC(_env);
+ env->CallVoidMethod(_jni_scanner_obj, _jni_scanner_open);
+ RETURN_ERROR_IF_EXC(env);
Review Comment:
Why using a local env here? And do we need to release this env?
Add some comment in code.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/MaxComputeScanNode.java:
##########
@@ -74,7 +78,37 @@ protected Map<String, String> getLocationProperties() throws UserException {
@Override
protected List<Split> getSplits() throws UserException {
List<Split> result = new ArrayList<>();
- result.add(new FileSplit(new Path("/"), 0, -1, -1, 0L, new String[0], Collections.emptyList()));
+ // String splitPath = catalog.getTunnelUrl();
+ // TODO: use single max compute scan node rather than file scan node
+ com.aliyun.odps.Table odpsTable = table.getOdpsTable();
+ if (desc.getSlots().isEmpty() || odpsTable.getFileNum() <= 0) {
+ return result;
+ }
+ try {
+ List<Pair<Long, Long>> sliceRange = new ArrayList<>();
+ long totalRows = catalog.getTotalRows(table.getDbName(), table.getName());
+ long fileNum = odpsTable.getFileNum();
+ long splitSize = (long) Math.ceil((double) totalRows / fileNum);
Review Comment:
And what if there is only 100 rows, and 10 files. Do we need to split into 10 ranges? I think this is too small.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/FileQueryScanNode.java:
##########
@@ -193,7 +193,10 @@ private void setColumnPositionMapping()
throws UserException {
TableIf tbl = getTargetTable();
List<Integer> columnIdxs = Lists.newArrayList();
-
+ if (params.getRequiredSlots() == null) {
Review Comment:
What is this for？ Add some comment to explain it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] AshinGau commented on pull request #19819: [feature-wip](multi-catalog)(step2)support read max compute data by JNI
Posted by "AshinGau (via GitHub)" <gi...@apache.org>.
AshinGau commented on PR #19819:
URL: https://github.com/apache/doris/pull/19819#issuecomment-1569434277
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org