You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/03/15 06:41:59 UTC
[GitHub] [druid] zachjsh opened a new pull request #9519: Ability to Delete
task logs and segments from Google Storage
zachjsh opened a new pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519
### Description
* implement ability to delete all tasks logs or all task logs
written before a particular date when written to Google storage
* implement ability to delete all segments from Google deep storage
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
zachjsh commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394683573
##########
File path: extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3TaskLogs.java
##########
@@ -165,7 +165,7 @@ String getTaskLogKey(String taskid, String filename)
@Override
public void killAll() throws IOException
{
- log.info("Deleting all task logs from s3 location [bucket: %s prefix: %s].",
+ log.info("Deleting all task logs from s3 location [bucket: '%s' prefix: '%s'].",
config.getS3Bucket(), config.getS3Prefix()
);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394188865
##########
File path: extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentKiller.java
##########
@@ -37,11 +38,18 @@
private static final Logger LOG = new Logger(GoogleDataSegmentKiller.class);
private final GoogleStorage storage;
+ private final GoogleAccountConfig accountConfig;
+ private final GoogleInputDataConfig inputDataConfig;
@Inject
- public GoogleDataSegmentKiller(final GoogleStorage storage)
+ public GoogleDataSegmentKiller(
+ final GoogleStorage storage,
+ GoogleAccountConfig accountConfig,
+ GoogleInputDataConfig inputDataConfig)
Review comment:
formatting
```suggestion
GoogleInputDataConfig inputDataConfig
)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
zachjsh commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394683601
##########
File path: extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleTaskLogs.java
##########
@@ -159,14 +169,34 @@ private String getTaskReportKey(String taskid)
}
@Override
- public void killAll()
+ public void killAll() throws IOException
{
- throw new UnsupportedOperationException("not implemented");
+ LOG.info("Deleting all task logs from gs location [bucket: '%s' prefix: '%s'].",
+ config.getBucket(), config.getPrefix()
+ );
+
+ long now = timeSupplier.getAsLong();
+ killOlderThan(now);
}
@Override
- public void killOlderThan(long timestamp)
+ public void killOlderThan(long timestamp) throws IOException
{
- throw new UnsupportedOperationException("not implemented");
+ LOG.info("Deleting all task logs from gs location [bucket: '%s' prefix: '%s'] older than %s.",
+ config.getBucket(), config.getPrefix(), new Date(timestamp)
+ );
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
zachjsh commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394683678
##########
File path: extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentKiller.java
##########
@@ -93,8 +101,23 @@ private void deleteIfPresent(String bucket, String path) throws IOException
}
@Override
- public void killAll()
+ public void killAll() throws IOException
{
- throw new UnsupportedOperationException("not implemented");
+ LOG.info("Deleting all segment files from gs location [bucket: '%s' prefix: '%s']",
+ accountConfig.getBucket(), accountConfig.getPrefix()
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394189424
##########
File path: extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleTaskLogs.java
##########
@@ -33,19 +34,28 @@
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
+import java.util.Date;
public class GoogleTaskLogs implements TaskLogs
{
private static final Logger LOG = new Logger(GoogleTaskLogs.class);
private final GoogleTaskLogsConfig config;
private final GoogleStorage storage;
+ private final GoogleInputDataConfig inputDataConfig;
+ private final CurrentTimeMillisSupplier timeSupplier;
@Inject
- public GoogleTaskLogs(GoogleTaskLogsConfig config, GoogleStorage storage)
+ public GoogleTaskLogs(
+ GoogleTaskLogsConfig config,
+ GoogleStorage storage,
+ GoogleInputDataConfig inputDataConfig,
+ CurrentTimeMillisSupplier timeSupplier)
Review comment:
```suggestion
CurrentTimeMillisSupplier timeSupplier
)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394189685
##########
File path: extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleTaskLogs.java
##########
@@ -159,14 +169,34 @@ private String getTaskReportKey(String taskid)
}
@Override
- public void killAll()
+ public void killAll() throws IOException
{
- throw new UnsupportedOperationException("not implemented");
+ LOG.info("Deleting all task logs from gs location [bucket: '%s' prefix: '%s'].",
+ config.getBucket(), config.getPrefix()
+ );
Review comment:
```suggestion
LOG.info(
"Deleting all task logs from gs location [bucket: '%s' prefix: '%s'].",
config.getBucket(),
config.getPrefix()
);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
zachjsh commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394683621
##########
File path: extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleTaskLogs.java
##########
@@ -159,14 +169,34 @@ private String getTaskReportKey(String taskid)
}
@Override
- public void killAll()
+ public void killAll() throws IOException
{
- throw new UnsupportedOperationException("not implemented");
+ LOG.info("Deleting all task logs from gs location [bucket: '%s' prefix: '%s'].",
+ config.getBucket(), config.getPrefix()
+ );
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394190486
##########
File path: extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3TaskLogs.java
##########
@@ -165,7 +165,7 @@ String getTaskLogKey(String taskid, String filename)
@Override
public void killAll() throws IOException
{
- log.info("Deleting all task logs from s3 location [bucket: %s prefix: %s].",
+ log.info("Deleting all task logs from s3 location [bucket: '%s' prefix: '%s'].",
config.getS3Bucket(), config.getS3Prefix()
);
Review comment:
```suggestion
log.info(
"Deleting all task logs from s3 location [bucket: '%s' prefix: '%s'].",
config.getS3Bucket(),
config.getS3Prefix()
);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis merged pull request #9519: Ability to Delete
task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
clintropolis merged pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394190082
##########
File path: extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleTaskLogs.java
##########
@@ -159,14 +169,34 @@ private String getTaskReportKey(String taskid)
}
@Override
- public void killAll()
+ public void killAll() throws IOException
{
- throw new UnsupportedOperationException("not implemented");
+ LOG.info("Deleting all task logs from gs location [bucket: '%s' prefix: '%s'].",
+ config.getBucket(), config.getPrefix()
+ );
+
+ long now = timeSupplier.getAsLong();
+ killOlderThan(now);
}
@Override
- public void killOlderThan(long timestamp)
+ public void killOlderThan(long timestamp) throws IOException
{
- throw new UnsupportedOperationException("not implemented");
+ LOG.info("Deleting all task logs from gs location [bucket: '%s' prefix: '%s'] older than %s.",
+ config.getBucket(), config.getPrefix(), new Date(timestamp)
+ );
Review comment:
```suggestion
LOG.info(
"Deleting all task logs from gs location [bucket: '%s' prefix: '%s'] older than %s.",
config.getBucket(),
config.getPrefix(),
new Date(timestamp)
);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
zachjsh commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394683656
##########
File path: extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleTaskLogs.java
##########
@@ -33,19 +34,28 @@
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
+import java.util.Date;
public class GoogleTaskLogs implements TaskLogs
{
private static final Logger LOG = new Logger(GoogleTaskLogs.class);
private final GoogleTaskLogsConfig config;
private final GoogleStorage storage;
+ private final GoogleInputDataConfig inputDataConfig;
+ private final CurrentTimeMillisSupplier timeSupplier;
@Inject
- public GoogleTaskLogs(GoogleTaskLogsConfig config, GoogleStorage storage)
+ public GoogleTaskLogs(
+ GoogleTaskLogsConfig config,
+ GoogleStorage storage,
+ GoogleInputDataConfig inputDataConfig,
+ CurrentTimeMillisSupplier timeSupplier)
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
zachjsh commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394683698
##########
File path: extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentKiller.java
##########
@@ -37,11 +38,18 @@
private static final Logger LOG = new Logger(GoogleDataSegmentKiller.class);
private final GoogleStorage storage;
+ private final GoogleAccountConfig accountConfig;
+ private final GoogleInputDataConfig inputDataConfig;
@Inject
- public GoogleDataSegmentKiller(final GoogleStorage storage)
+ public GoogleDataSegmentKiller(
+ final GoogleStorage storage,
+ GoogleAccountConfig accountConfig,
+ GoogleInputDataConfig inputDataConfig)
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9519:
Ability to Delete task logs and segments from Google Storage
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9519: Ability to Delete task logs and segments from Google Storage
URL: https://github.com/apache/druid/pull/9519#discussion_r394189223
##########
File path: extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentKiller.java
##########
@@ -93,8 +101,23 @@ private void deleteIfPresent(String bucket, String path) throws IOException
}
@Override
- public void killAll()
+ public void killAll() throws IOException
{
- throw new UnsupportedOperationException("not implemented");
+ LOG.info("Deleting all segment files from gs location [bucket: '%s' prefix: '%s']",
+ accountConfig.getBucket(), accountConfig.getPrefix()
Review comment:
```suggestion
LOG.info(
"Deleting all segment files from gs location [bucket: '%s' prefix: '%s']",
accountConfig.getBucket(),
accountConfig.getPrefix()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org