You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "SaintBacchus (via GitHub)" <gi...@apache.org> on 2023/03/23 09:11:04 UTC
[GitHub] [doris] SaintBacchus opened a new pull request, #18059: [Feature-Wip](MySQL Load)Show load warnings for mysql load
SaintBacchus opened a new pull request, #18059:
URL: https://github.com/apache/doris/pull/18059
# Proposed changes
Issue Number: close #xxx
## Problem summary
1. Support the `show load warnings` for mysql load to get the detail error message.
2. Fix `fillByteBufferAsync` not mark the load as finished in same data load
3. Fix `drain data` only in client mode.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #18059: [Feature-Wip](MySQL Load)Show load warnings for mysql load
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #18059:
URL: https://github.com/apache/doris/pull/18059#discussion_r1147805826
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/MysqlLoadManager.java:
##########
@@ -124,20 +157,22 @@ public LoadJobRowResult executeMySqlLoadJobFromStmt(ConnectContext context, Load
context.setExecTimeout(newTimeOut);
}
String token = tokenManager.acquireToken();
+ boolean clientLocal = dataDesc.isClientLocal();
+ MySqlLoadContext loadContext = new MySqlLoadContext();
+ loadContextMap.put(loadId, loadContext);
LOG.info("execute MySqlLoadJob for id: {}.", loadId);
try (final CloseableHttpClient httpclient = HttpClients.createDefault()) {
for (String file : filePaths) {
- InputStreamEntity entity = getInputStreamEntity(context, dataDesc.isClientLocal(), file, loadId);
+ InputStreamEntity entity = getInputStreamEntity(context, clientLocal, file, loadId);
HttpPut request = generateRequestForMySqlLoad(entity, dataDesc, database, table, token);
- MySqlLoadContext loadContext = new MySqlLoadContext();
loadContext.setRequest(request);
- loadContextMap.put(loadId, loadContext);
try (final CloseableHttpResponse response = httpclient.execute(request)) {
String body = EntityUtils.toString(response.getEntity());
JsonObject result = JsonParser.parseString(body).getAsJsonObject();
if (!result.get("Status").getAsString().equalsIgnoreCase("Success")) {
+ failedRecords.offer(new MySqlLoadFailRecord(loadId, result.get("ErrorURL").getAsString()));
LOG.warn("Execute mysql data load failed with request: {} and response: {}", request, body);
- throw new LoadException(result.get("Message").getAsString());
+ throw new LoadException(result.get("Message").getAsString() + " with load id " + loadId);
Review Comment:
Why not just return the error url to user?
so that you don't need to save the `MySqlLoadFailRecord` in memory at all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SaintBacchus commented on a diff in pull request #18059: [Feature-Wip](MySQL Load)Show load warnings for mysql load
Posted by "SaintBacchus (via GitHub)" <gi...@apache.org>.
SaintBacchus commented on code in PR #18059:
URL: https://github.com/apache/doris/pull/18059#discussion_r1148281026
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/MysqlLoadManager.java:
##########
@@ -124,20 +157,22 @@ public LoadJobRowResult executeMySqlLoadJobFromStmt(ConnectContext context, Load
context.setExecTimeout(newTimeOut);
}
String token = tokenManager.acquireToken();
+ boolean clientLocal = dataDesc.isClientLocal();
+ MySqlLoadContext loadContext = new MySqlLoadContext();
+ loadContextMap.put(loadId, loadContext);
LOG.info("execute MySqlLoadJob for id: {}.", loadId);
try (final CloseableHttpClient httpclient = HttpClients.createDefault()) {
for (String file : filePaths) {
- InputStreamEntity entity = getInputStreamEntity(context, dataDesc.isClientLocal(), file, loadId);
+ InputStreamEntity entity = getInputStreamEntity(context, clientLocal, file, loadId);
HttpPut request = generateRequestForMySqlLoad(entity, dataDesc, database, table, token);
- MySqlLoadContext loadContext = new MySqlLoadContext();
loadContext.setRequest(request);
- loadContextMap.put(loadId, loadContext);
try (final CloseableHttpResponse response = httpclient.execute(request)) {
String body = EntityUtils.toString(response.getEntity());
JsonObject result = JsonParser.parseString(body).getAsJsonObject();
if (!result.get("Status").getAsString().equalsIgnoreCase("Success")) {
+ failedRecords.offer(new MySqlLoadFailRecord(loadId, result.get("ErrorURL").getAsString()));
LOG.warn("Execute mysql data load failed with request: {} and response: {}", request, body);
- throw new LoadException(result.get("Message").getAsString());
+ throw new LoadException(result.get("Message").getAsString() + " with load id " + loadId);
Review Comment:
It's too long for user. A typical error url will have about 150 letters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SaintBacchus closed pull request #18059: [Feature-Wip](MySQL Load)Show load warnings for mysql load
Posted by "SaintBacchus (via GitHub)" <gi...@apache.org>.
SaintBacchus closed pull request #18059: [Feature-Wip](MySQL Load)Show load warnings for mysql load
URL: https://github.com/apache/doris/pull/18059
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org