You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wary (via GitHub)" <gi...@apache.org> on 2025/04/14 08:51:10 UTC
[PR] add code check for fe request [doris-spark-connector]
wary opened a new pull request, #309:
URL: https://github.com/apache/doris-spark-connector/pull/309
# Proposed changes
Add code checks in the fe init method to make error messages more readable.
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2048618422
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -114,7 +115,13 @@ private List<Frontend> initFrontends(DorisConfig config) throws Exception {
+ ", reason: " + response.getStatusLine().getReasonPhrase());
}
String entity = EntityUtils.toString(response.getEntity());
- dataNode = extractEntity(entity, "data");
+ LOG.info("get response: {} with fe request url: {}", entity, feReqURL);
+ JsonNode respNode = MAPPER.readTree(entity);
+ String code = respNode.get("code").asText();
+ if(!"0".equalsIgnoreCase(code)) {
+ throw new RuntimeException("fetch fe request with invalid code, response: " + entity);
+ }
+ dataNode = respNode.get("data");
Review Comment:
Yes, you are right , it is indeed necessary to throw here, especially when the executor log cannot be viewed.
Maybe we also need to consider other http calls? Maybe there is a general way to handle it, such as throwing entityString directly when an error occurs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2043440082
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -114,7 +115,13 @@ private List<Frontend> initFrontends(DorisConfig config) throws Exception {
+ ", reason: " + response.getStatusLine().getReasonPhrase());
}
String entity = EntityUtils.toString(response.getEntity());
- dataNode = extractEntity(entity, "data");
+ LOG.info("get response: {} with fe request url: {}", entity, feReqURL);
+ JsonNode respNode = MAPPER.readTree(entity);
+ String code = respNode.get("code").asText();
+ if(!"0".equalsIgnoreCase(code)) {
+ throw new RuntimeException("fetch fe request with invalid code, response: " + entity);
+ }
+ dataNode = respNode.get("data");
Review Comment:
Maybe I didn't express it clearly.
1. Since the "extractEntity" method is called in many places, the calls in other places may also have this problem, and other places may also need to be dealt with?
2. Does this exception necessarily need to be thrown? If it doesn't need to be thrown, perhaps just printing a log entry will be enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "wary (via GitHub)" <gi...@apache.org>.
wary commented on PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#issuecomment-2817530829
> Hi, there is a conflict in import. You can use doris's checkstyle. You can refer to https://doris.apache.org/zh-CN/community/developer-guide/java-format-code#checkstyle-%E6%8F%92%E4%BB%B6
ok, resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "wary (via GitHub)" <gi...@apache.org>.
wary commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2041783466
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -114,7 +114,12 @@ private List<Frontend> initFrontends(DorisConfig config) throws Exception {
+ ", reason: " + response.getStatusLine().getReasonPhrase());
}
String entity = EntityUtils.toString(response.getEntity());
- dataNode = extractEntity(entity, "data");
Review Comment:
Does throwing exceptions make it easier to locate issues?
If the user doesn't have permission to call this interface, the program will throw a null point exception, which can be somewhat misleading. Without prior experience with such problems, simply logging a message might not make it easily noticeable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "wary (via GitHub)" <gi...@apache.org>.
wary commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2048727438
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -114,7 +115,13 @@ private List<Frontend> initFrontends(DorisConfig config) throws Exception {
+ ", reason: " + response.getStatusLine().getReasonPhrase());
}
String entity = EntityUtils.toString(response.getEntity());
- dataNode = extractEntity(entity, "data");
+ LOG.info("get response: {} with fe request url: {}", entity, feReqURL);
+ JsonNode respNode = MAPPER.readTree(entity);
+ String code = respNode.get("code").asText();
+ if(!"0".equalsIgnoreCase(code)) {
+ throw new RuntimeException("fetch fe request with invalid code, response: " + entity);
+ }
+ dataNode = respNode.get("data");
Review Comment:
Add a common method extractDataFromResponse and handle all http response with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "wary (via GitHub)" <gi...@apache.org>.
wary commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2041816877
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -114,7 +114,12 @@ private List<Frontend> initFrontends(DorisConfig config) throws Exception {
+ ", reason: " + response.getStatusLine().getReasonPhrase());
}
String entity = EntityUtils.toString(response.getEntity());
- dataNode = extractEntity(entity, "data");
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "wary (via GitHub)" <gi...@apache.org>.
wary commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2048183473
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -114,7 +115,13 @@ private List<Frontend> initFrontends(DorisConfig config) throws Exception {
+ ", reason: " + response.getStatusLine().getReasonPhrase());
}
String entity = EntityUtils.toString(response.getEntity());
- dataNode = extractEntity(entity, "data");
+ LOG.info("get response: {} with fe request url: {}", entity, feReqURL);
+ JsonNode respNode = MAPPER.readTree(entity);
+ String code = respNode.get("code").asText();
+ if(!"0".equalsIgnoreCase(code)) {
+ throw new RuntimeException("fetch fe request with invalid code, response: " + entity);
+ }
+ dataNode = respNode.get("data");
Review Comment:
> Maybe I didn't express it clearly.
>
> 1. Since the "extractEntity" method is called in many places, the calls in other places may also have this problem, and other places may also need to be dealt with?
>
Yes, Doris Spark Connector has similar issues in many other cases, where it only checks the HTTP code but doesn't verify the status or code in the response, just like this commit operation in previous version.
> 2. Does this exception necessarily need to be thrown? If it doesn't need to be thrown, perhaps just printing a log entry will be enough?
No matter whether the exception is thrown or not, the task will fail due to this error. If we don't throw this exception, the error message won't be accurate enough, causing many misunderstandings during troubleshooting. I think this follows the same logic as the HTTP status code check above—it should be thrown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2041772359
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -114,7 +114,12 @@ private List<Frontend> initFrontends(DorisConfig config) throws Exception {
+ ", reason: " + response.getStatusLine().getReasonPhrase());
}
String entity = EntityUtils.toString(response.getEntity());
- dataNode = extractEntity(entity, "data");
Review Comment:
Can we add a log output here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "wary (via GitHub)" <gi...@apache.org>.
wary commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2051702813
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -49,11 +48,7 @@
import java.io.IOException;
import java.io.Serializable;
-import java.sql.Connection;
-import java.sql.DriverManager;
-import java.sql.PreparedStatement;
-import java.sql.ResultSet;
-import java.sql.SQLException;
+import java.sql.*;
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "wary (via GitHub)" <gi...@apache.org>.
wary commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2048183473
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -114,7 +115,13 @@ private List<Frontend> initFrontends(DorisConfig config) throws Exception {
+ ", reason: " + response.getStatusLine().getReasonPhrase());
}
String entity = EntityUtils.toString(response.getEntity());
- dataNode = extractEntity(entity, "data");
+ LOG.info("get response: {} with fe request url: {}", entity, feReqURL);
+ JsonNode respNode = MAPPER.readTree(entity);
+ String code = respNode.get("code").asText();
+ if(!"0".equalsIgnoreCase(code)) {
+ throw new RuntimeException("fetch fe request with invalid code, response: " + entity);
+ }
+ dataNode = respNode.get("data");
Review Comment:
> Maybe I didn't express it clearly.
>
> 1. Since the "extractEntity" method is called in many places, the calls in other places may also have this problem, and other places may also need to be dealt with?
>
Yes, Doris Spark Connector has similar issues in many other cases, where it only checks the HTTP code but doesn't verify the status or code in the response, just like the commit operation in previous version.
> 2. Does this exception necessarily need to be thrown? If it doesn't need to be thrown, perhaps just printing a log entry will be enough?
No matter whether the exception is thrown or not, the task will fail due to this error. If we don't throw this exception, the error message won't be accurate enough, causing many misunderstandings during troubleshooting. I think this follows the same logic as the HTTP status code check above—it should be thrown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2050517015
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -49,11 +48,7 @@
import java.io.IOException;
import java.io.Serializable;
-import java.sql.Connection;
-import java.sql.DriverManager;
-import java.sql.PreparedStatement;
-import java.sql.ResultSet;
-import java.sql.SQLException;
+import java.sql.*;
Review Comment:
not import *
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "wary (via GitHub)" <gi...@apache.org>.
wary commented on code in PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#discussion_r2041783466
##########
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/DorisFrontendClient.java:
##########
@@ -114,7 +114,12 @@ private List<Frontend> initFrontends(DorisConfig config) throws Exception {
+ ", reason: " + response.getStatusLine().getReasonPhrase());
}
String entity = EntityUtils.toString(response.getEntity());
- dataNode = extractEntity(entity, "data");
Review Comment:
Does throwing exceptions make it easier to locate issues?
If the user doesn't have permission to call this interface, the program will throw a null point exception, which can be somewhat misleading. Without prior experience with such problems, simply logging a message might not make it easily noticeable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] add code check for fe request [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #309:
URL: https://github.com/apache/doris-spark-connector/pull/309#issuecomment-2817497652
Hi, there is a conflict in import. You can use doris's checkstyle. You can refer to https://doris.apache.org/zh-CN/community/developer-guide/java-format-code#checkstyle-%E6%8F%92%E4%BB%B6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org