You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/06/22 12:50:17 UTC
[GitHub] [pulsar] fantapsody opened a new pull request, #16177: [improve][es-sink] Add error log for failed bulk records in ES sink
fantapsody opened a new pull request, #16177:
URL: https://github.com/apache/pulsar/pull/16177
### Motivation
Currently, the ES sink doesn't show any error when any record in a bulk write fails, and the failure may persist until the developer fixed the issue.
### Modifications
Add a warning log when any record in bulk write fails.
### Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [ ] `doc-not-needed`
(Please explain why)
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] fantapsody commented on pull request #16177: [improve][es-sink] Add error log for failed bulk records in ES sink
Posted by GitBox <gi...@apache.org>.
fantapsody commented on PR #16177:
URL: https://github.com/apache/pulsar/pull/16177#issuecomment-1182692830
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi commented on a diff in pull request #16177: [improve][es-sink] Add error log for failed bulk records in ES sink
Posted by GitBox <gi...@apache.org>.
nicoloboschi commented on code in PR #16177:
URL: https://github.com/apache/pulsar/pull/16177#discussion_r903723764
##########
pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchClient.java:
##########
@@ -73,9 +73,16 @@ public void afterBulk(long executionId, List<BulkProcessor.BulkOperationRequest>
}
int index = 0;
for (BulkProcessor.BulkOperationResult result: results) {
- final Record record = bulkOperationList.get(index++).getPulsarRecord();
+ final Record<?> record = bulkOperationList.get(index++).getPulsarRecord();
if (result.isError()) {
record.fail();
+
+ log.warn("Bulk request id={} failed, message id=[{}] index={} error={}", executionId,
Review Comment:
method `checkForIrrecoverableError` will log, ignore or throw an error for a malformed message error depending on `malformedDocAction`.
It'd be better to move this new log inside that method and ensure to log only if it's not a `MALFORMED_ERRORS`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #16177: [improve][es-sink] Add error log for failed bulk records in ES sink
Posted by GitBox <gi...@apache.org>.
codelipenghui merged PR #16177:
URL: https://github.com/apache/pulsar/pull/16177
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] fantapsody commented on pull request #16177: [improve][es-sink] Add error log for failed bulk records in ES sink
Posted by GitBox <gi...@apache.org>.
fantapsody commented on PR #16177:
URL: https://github.com/apache/pulsar/pull/16177#issuecomment-1182637467
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] fantapsody commented on a diff in pull request #16177: [improve][es-sink] Add error log for failed bulk records in ES sink
Posted by GitBox <gi...@apache.org>.
fantapsody commented on code in PR #16177:
URL: https://github.com/apache/pulsar/pull/16177#discussion_r903780892
##########
pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchClient.java:
##########
@@ -73,9 +73,16 @@ public void afterBulk(long executionId, List<BulkProcessor.BulkOperationRequest>
}
int index = 0;
for (BulkProcessor.BulkOperationResult result: results) {
- final Record record = bulkOperationList.get(index++).getPulsarRecord();
+ final Record<?> record = bulkOperationList.get(index++).getPulsarRecord();
if (result.isError()) {
record.fail();
+
+ log.warn("Bulk request id={} failed, message id=[{}] index={} error={}", executionId,
Review Comment:
That makes sense. Please take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] fantapsody commented on pull request #16177: [improve][es-sink] Add error log for failed bulk records in ES sink
Posted by GitBox <gi...@apache.org>.
fantapsody commented on PR #16177:
URL: https://github.com/apache/pulsar/pull/16177#issuecomment-1182697321
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org