You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/03/08 06:49:13 UTC
[GitHub] [druid] zhangyue19921010 opened a new pull request #10956: [Minor]Add metadata-related logs for kill tasks and add UT related.
zhangyue19921010 opened a new pull request #10956:
URL: https://github.com/apache/druid/pull/10956
<!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
<!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
Fixes https://github.com/apache/druid/issues/10423.
<!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
<!-- If you are a committer, follow the PR action item checklist for committers:
https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
### Description
Logs more info for metadata deletion in overlord:
1. This would help users to more clearly understand what delete actions have been done by the kill task including metadata and data.
2. This would help in debugging metadata related issues.
P.S.
Kill task is divided into two stages:
1. Overlord deletes meta infos.
2. Peon deletes DeepStorage files.
So that we can only add meta-related logs in overlord.
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10956:
URL: https://github.com/apache/druid/pull/10956#discussion_r591971224
##########
File path: server/src/test/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinatorTest.java
##########
@@ -848,6 +848,28 @@ public void testDeleteDataSourceMetadata() throws IOException
Assert.assertNull("getDataSourceMetadataNullAfterDelete", coordinator.retrieveDataSourceMetadata("fooDataSource"));
}
+ @Test
+ public void testDeleteSegments()
+ {
+ Exception e = null;
+ try {
+ coordinator.deleteSegments(SEGMENTS);
+ }
+ catch (Exception ex) {
+ e = ex;
+ }
+ Assert.assertNull(e);
Review comment:
If an exception is thrown, the test will fail. We don't need to catch the exception and validate that it's null
```suggestion
coordinator.deleteSegments(SEGMENTS);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #10956:
URL: https://github.com/apache/druid/pull/10956#discussion_r592084345
##########
File path: server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java
##########
@@ -1290,9 +1294,12 @@ public Void inTransaction(Handle handle, TransactionStatus transactionStatus)
private void deleteSegment(final Handle handle, final DataSegment segment)
{
- handle.createStatement(StringUtils.format("DELETE from %s WHERE id = :id", dbTables.getSegmentsTable()))
- .bind("id", segment.getId().toString())
- .execute();
+ String segmentsTable = dbTables.getSegmentsTable();
+ String segmentId = segment.getId().toString();
+ log.debug("Removing segment [%s] [%s] from Metadata Storage [%s]!", segmentId, segment.getDataSource(), segmentsTable);
Review comment:
nit: I guess this could be a lot of segments, it might be nicer to pull this outside of the loop and use `log.debugSegments`
##########
File path: server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java
##########
@@ -1278,9 +1278,13 @@ public void deleteSegments(final Set<DataSegment> segments)
@Override
public Void inTransaction(Handle handle, TransactionStatus transactionStatus)
{
+ int segmentSize = segments.size();
+ String dataSource = "";
for (final DataSegment segment : segments) {
+ dataSource = segment.getDataSource();
deleteSegment(handle, segment);
}
+ log.info("Removed [%d] segments'metadata for dataSource [%s]!", segmentSize, dataSource);
Review comment:
```suggestion
log.info("Removed [%d] segments from metadata for dataSource [%s]!", segmentSize, dataSource);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on a change in pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #10956:
URL: https://github.com/apache/druid/pull/10956#discussion_r592124492
##########
File path: server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java
##########
@@ -1290,9 +1294,12 @@ public Void inTransaction(Handle handle, TransactionStatus transactionStatus)
private void deleteSegment(final Handle handle, final DataSegment segment)
{
- handle.createStatement(StringUtils.format("DELETE from %s WHERE id = :id", dbTables.getSegmentsTable()))
- .bind("id", segment.getId().toString())
- .execute();
+ String segmentsTable = dbTables.getSegmentsTable();
+ String segmentId = segment.getId().toString();
+ log.debug("Removing segment [%s] [%s] from Metadata Storage [%s]!", segmentId, segment.getDataSource(), segmentsTable);
Review comment:
Nice idea, I didn't notice there is `log.debugSegments` in druid before. Changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on a change in pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #10956:
URL: https://github.com/apache/druid/pull/10956#discussion_r592080378
##########
File path: server/src/test/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinatorTest.java
##########
@@ -848,6 +848,28 @@ public void testDeleteDataSourceMetadata() throws IOException
Assert.assertNull("getDataSourceMetadataNullAfterDelete", coordinator.retrieveDataSourceMetadata("fooDataSource"));
}
+ @Test
+ public void testDeleteSegments()
+ {
+ Exception e = null;
+ try {
+ coordinator.deleteSegments(SEGMENTS);
+ }
+ catch (Exception ex) {
+ e = ex;
+ }
+ Assert.assertNull(e);
+ Assert.assertEquals(
+ 0,
+ ImmutableSet.copyOf(
+ coordinator.retrieveUnusedSegmentsForInterval(
+ defaultSegment.getDataSource(),
+ defaultSegment.getInterval()
+ )
+ ).size()
+ );
+ }
Review comment:
Sure, I did a little change. What I want to test is the function of deleting segment metadata in MetaDataStorage.
Here is the workflow of this UT: `Published segments to MetaDataStorage` -> `check segments Published` -> `remove segments in MetaDataStorage` -> `check segments removed`.
Also name have been changed as `testDeleteSegmentsInMetaDataStorage`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zhangyue19921010 edited a comment on pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #10956:
URL: https://github.com/apache/druid/pull/10956#issuecomment-793429296
Failed CI job is not this pr related. Retry will pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #10956:
URL: https://github.com/apache/druid/pull/10956#issuecomment-797187950
Hi @suneet-s and @clintropolis Thanks a lot for your review and merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on a change in pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #10956:
URL: https://github.com/apache/druid/pull/10956#discussion_r592124613
##########
File path: server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java
##########
@@ -1278,9 +1278,13 @@ public void deleteSegments(final Set<DataSegment> segments)
@Override
public Void inTransaction(Handle handle, TransactionStatus transactionStatus)
{
+ int segmentSize = segments.size();
+ String dataSource = "";
for (final DataSegment segment : segments) {
+ dataSource = segment.getDataSource();
deleteSegment(handle, segment);
}
+ log.info("Removed [%d] segments'metadata for dataSource [%s]!", segmentSize, dataSource);
Review comment:
Changed. Thanks for your review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zhangyue19921010 edited a comment on pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #10956:
URL: https://github.com/apache/druid/pull/10956#issuecomment-796574992
Failed CI jobs are not this pr related. Maybe retry will be passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10956:
URL: https://github.com/apache/druid/pull/10956#discussion_r591971612
##########
File path: server/src/test/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinatorTest.java
##########
@@ -848,6 +848,28 @@ public void testDeleteDataSourceMetadata() throws IOException
Assert.assertNull("getDataSourceMetadataNullAfterDelete", coordinator.retrieveDataSourceMetadata("fooDataSource"));
}
+ @Test
+ public void testDeleteSegments()
+ {
+ Exception e = null;
+ try {
+ coordinator.deleteSegments(SEGMENTS);
+ }
+ catch (Exception ex) {
+ e = ex;
+ }
+ Assert.assertNull(e);
+ Assert.assertEquals(
+ 0,
+ ImmutableSet.copyOf(
+ coordinator.retrieveUnusedSegmentsForInterval(
+ defaultSegment.getDataSource(),
+ defaultSegment.getInterval()
+ )
+ ).size()
+ );
+ }
Review comment:
I don't understand what this is testing. Could you explain what you're trying to test here. You could use the test method name format like `test<MethodUnderTest>When<Condition>Should<ExpectedResult>`
See `KafkaRecordSupplierTest` for some examples like `getLatestSequenceNumberWhenPartitionIsEmptyAndUseEarliestOffsetShouldReturnsValidNonNull`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on a change in pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #10956:
URL: https://github.com/apache/druid/pull/10956#discussion_r592079028
##########
File path: server/src/test/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinatorTest.java
##########
@@ -848,6 +848,28 @@ public void testDeleteDataSourceMetadata() throws IOException
Assert.assertNull("getDataSourceMetadataNullAfterDelete", coordinator.retrieveDataSourceMetadata("fooDataSource"));
}
+ @Test
+ public void testDeleteSegments()
+ {
+ Exception e = null;
+ try {
+ coordinator.deleteSegments(SEGMENTS);
+ }
+ catch (Exception ex) {
+ e = ex;
+ }
+ Assert.assertNull(e);
Review comment:
Thanks for your review! Changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #10956:
URL: https://github.com/apache/druid/pull/10956#issuecomment-796574992
Failed CI jobs are not this pr related. Re-try will be passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s merged pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
suneet-s merged pull request #10956:
URL: https://github.com/apache/druid/pull/10956
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on pull request #10956: [Minor]Add metadata-related logs and missing UT for kill tasks.
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #10956:
URL: https://github.com/apache/druid/pull/10956#issuecomment-793429296
Failed CI job is not this pr related. retry will pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org