You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "MYiYang (via GitHub)" <gi...@apache.org> on 2023/12/14 08:43:38 UTC
[PR] [refactor](stream-load) Adjust the content of the review Added direct load via BeNodes [doris-spark-connector]
MYiYang opened a new pull request, #171:
URL: https://github.com/apache/doris-spark-connector/pull/171
[fateure](stream-load) Added direct load via BeNodes
[refactor](stream-load) Adjust the content of the review
# changes
Added the mode of displaying BENODES configuration for stream-load operation.
Configure doris.benodes. The doris.fenodes configuration needs to be deleted.
Such as 0.0.0.0:8040, X.X.X.X: 8040.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](stream-load) Adjust the content of the review Added direct load via BeNodes [doris-spark-connector]
Posted by "MYiYang (via GitHub)" <gi...@apache.org>.
MYiYang commented on code in PR #171:
URL: https://github.com/apache/doris-spark-connector/pull/171#discussion_r1430860293
##########
spark-doris-connector/src/main/java/org/apache/doris/spark/rest/RestService.java:
##########
@@ -560,6 +582,42 @@ static List<BackendRow> parseBackend(String response, Logger logger) throws Dori
return backendRows;
}
+ /**
+ * get Doris BE node.
+ * @param logger slf4j logger
+ * @return the Doris BE node
+ * @throws IllegalArgumentException BE nodes is illegal
+ */
+ public static List<BackendV2.BackendRowV2> getBeBackendRows(SparkSettings sparkSettings, Logger logger) throws DorisException {
Review Comment:
Ok, let me think about what I should call it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](stream-load) Adjust the content of the review Added direct load via BeNodes [doris-spark-connector]
Posted by "MYiYang (via GitHub)" <gi...@apache.org>.
MYiYang commented on code in PR #171:
URL: https://github.com/apache/doris-spark-connector/pull/171#discussion_r1430859881
##########
spark-doris-connector/src/main/java/org/apache/doris/spark/rest/RestService.java:
##########
@@ -592,17 +650,20 @@ public static List<BackendV2.BackendRowV2> getBackendRows(SparkSettings sparkSet
}
/**
- * choice a Doris BE node to request.
+ * benode does not exist, execute the original FE
+ * @param sparkSettings settings
* @param logger slf4j logger
- * @return the chosen one Doris BE node
- * @throws IllegalArgumentException BE nodes is illegal
+ * @return Doris Be nodes
+ * @throws DorisException
*/
- @VisibleForTesting
- public static String randomBackendV2(SparkSettings sparkSettings, Logger logger) throws DorisException {
- List<BackendV2.BackendRowV2> backends = getBackendRows(sparkSettings, logger);
- Collections.shuffle(backends);
- BackendV2.BackendRowV2 backend = backends.get(0);
- return backend.getIp() + ":" + backend.getHttpPort();
+ private static List<BackendV2.BackendRowV2> getBackendRowV2s(SparkSettings sparkSettings, Logger logger) throws DorisException {
+ List<BackendV2.BackendRowV2> backends = null;
+ if(notBeNode(sparkSettings, logger)){
+ backends = getBackendRows(sparkSettings, logger);
+ }else{
+ backends = getBeBackendRows(sparkSettings, logger);
Review Comment:
Yeah, I should be able to take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](stream-load) Adjust the content of the review Added direct load via BeNodes [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #171:
URL: https://github.com/apache/doris-spark-connector/pull/171#discussion_r1427502167
##########
spark-doris-connector/src/main/java/org/apache/doris/spark/rest/RestService.java:
##########
@@ -560,6 +582,42 @@ static List<BackendRow> parseBackend(String response, Logger logger) throws Dori
return backendRows;
}
+ /**
+ * get Doris BE node.
+ * @param logger slf4j logger
+ * @return the Doris BE node
+ * @throws IllegalArgumentException BE nodes is illegal
+ */
+ public static List<BackendV2.BackendRowV2> getBeBackendRows(SparkSettings sparkSettings, Logger logger) throws DorisException {
Review Comment:
Backend means be. Maybe we can change the function name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](stream-load) Adjust the content of the review Added direct load via BeNodes [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #171:
URL: https://github.com/apache/doris-spark-connector/pull/171#discussion_r1427501099
##########
spark-doris-connector/src/main/java/org/apache/doris/spark/rest/RestService.java:
##########
@@ -592,17 +650,20 @@ public static List<BackendV2.BackendRowV2> getBackendRows(SparkSettings sparkSet
}
/**
- * choice a Doris BE node to request.
+ * benode does not exist, execute the original FE
+ * @param sparkSettings settings
* @param logger slf4j logger
- * @return the chosen one Doris BE node
- * @throws IllegalArgumentException BE nodes is illegal
+ * @return Doris Be nodes
+ * @throws DorisException
*/
- @VisibleForTesting
- public static String randomBackendV2(SparkSettings sparkSettings, Logger logger) throws DorisException {
- List<BackendV2.BackendRowV2> backends = getBackendRows(sparkSettings, logger);
- Collections.shuffle(backends);
- BackendV2.BackendRowV2 backend = backends.get(0);
- return backend.getIp() + ":" + backend.getHttpPort();
+ private static List<BackendV2.BackendRowV2> getBackendRowV2s(SparkSettings sparkSettings, Logger logger) throws DorisException {
+ List<BackendV2.BackendRowV2> backends = null;
+ if(notBeNode(sparkSettings, logger)){
+ backends = getBackendRows(sparkSettings, logger);
+ }else{
+ backends = getBeBackendRows(sparkSettings, logger);
Review Comment:
Is it enough to call the `getBeBackendRows` method directly in the `getBackendRows` method? Other methods should not need to be modified?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](stream-load) Adjust the content of the review Added direct load via BeNodes [doris-spark-connector]
Posted by "MYiYang (via GitHub)" <gi...@apache.org>.
MYiYang closed pull request #171: [refactor](stream-load) Adjust the content of the review Added direct load via BeNodes
URL: https://github.com/apache/doris-spark-connector/pull/171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org