You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by su...@apache.org on 2020/12/08 17:48:46 UTC
[druid] branch master updated: fix injection failure of
StorageLocationSelectorStrategy objects (#10363)
This is an automated email from the ASF dual-hosted git repository.
suneet pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new c410648 fix injection failure of StorageLocationSelectorStrategy objects (#10363)
c410648 is described below
commit c4106486306df6fadf92994f7d33a88c6d952551
Author: frank chen <fr...@outlook.com>
AuthorDate: Wed Dec 9 01:48:31 2020 +0800
fix injection failure of StorageLocationSelectorStrategy objects (#10363)
* fix to allow customer storage location selector strategy
* add test cases to check instance of selector strategy
* update doc
* code format
* resolve code review comments
* inject StorageLocation
* fix CI
* fix mismatched license item reported by CI
* change property path from druid.segmentCache.locationSelectorStrategy.type to druid.segmentCache.locationSelector.strategy
* using a helper method to bind to correct property path
---
docs/configuration/index.md | 4 +-
licenses.yaml | 11 +-
server/pom.xml | 4 +
.../org/apache/druid/guice/StorageNodeModule.java | 23 ++++
...stBytesUsedStorageLocationSelectorStrategy.java | 7 +-
...ailableSizeStorageLocationSelectorStrategy.java | 7 +-
.../RandomStorageLocationSelectorStrategy.java | 7 +-
.../RoundRobinStorageLocationSelectorStrategy.java | 11 +-
.../druid/segment/loading/SegmentLoaderConfig.java | 31 +++---
.../loading/SegmentLoaderLocalCacheManager.java | 48 ++++++---
.../loading/StorageLocationSelectorStrategy.java | 2 +-
.../SegmentLoaderLocalCacheManagerTest.java | 22 ++--
.../StorageLocationSelectorStrategyTest.java | 118 +++++++++++++++++++++
13 files changed, 226 insertions(+), 69 deletions(-)
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index 0755520..640ab0c 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -1409,7 +1409,7 @@ These Historical configurations can be defined in the `historical/runtime.proper
|Property|Description|Default|
|--------|-----------|-------|
|`druid.segmentCache.locations`|Segments assigned to a Historical process are first stored on the local file system (in a disk cache) and then served by the Historical process. These locations define where that local cache resides. This value cannot be NULL or EMPTY. Here is an example `druid.segmentCache.locations=[{"path": "/mnt/druidSegments", "maxSize": "10k", "freeSpacePercent": 1.0}]`. "freeSpacePercent" is optional, if provided then enforces that much of free disk partition space [...]
-|`druid.segmentCache.locationSelectorStrategy`|The strategy used to select a location from the configured `druid.segmentCache.locations` for segment distribution. Possible values are `leastBytesUsed`, `roundRobin`, `random`, or `mostAvailableSize`. |leastBytesUsed|
+|`druid.segmentCache.locationSelector.strategy`|The strategy used to select a location from the configured `druid.segmentCache.locations` for segment distribution. Possible values are `leastBytesUsed`, `roundRobin`, `random`, or `mostAvailableSize`. |leastBytesUsed|
|`druid.segmentCache.deleteOnRemove`|Delete segment files from cache once a process is no longer serving a segment.|true|
|`druid.segmentCache.dropSegmentDelayMillis`|How long a process delays before completely dropping segment.|30000 (30 seconds)|
|`druid.segmentCache.infoDir`|Historical processes keep track of the segments they are serving so that when the process is restarted they can reload the same segments without waiting for the Coordinator to reassign. This path defines where this metadata is kept. Directory will be created if needed.|${first_location}/info_dir|
@@ -1421,7 +1421,7 @@ These Historical configurations can be defined in the `historical/runtime.proper
In `druid.segmentCache.locations`, *freeSpacePercent* was added because *maxSize* setting is only a theoretical limit and assumes that much space will always be available for storing segments. In case of any druid bug leading to unaccounted segment files left alone on disk or some other process writing stuff to disk, This check can start failing segment loading early before filling up the disk completely and leaving the host usable otherwise.
-In `druid.segmentCache.locationSelectorStrategy`, one of leastBytesUsed, roundRobin, random, or mostAvailableSize could be specified to represent the strategy to distribute segments across multiple segment cache locations.
+In `druid.segmentCache.locationSelector.strategy`, one of `leastBytesUsed`, `roundRobin`, `random`, or `mostAvailableSize` could be specified to represent the strategy to distribute segments across multiple segment cache locations.
|Strategy|Description|
|--------|-----------|
diff --git a/licenses.yaml b/licenses.yaml
index 90a7ffd..31c3820 100644
--- a/licenses.yaml
+++ b/licenses.yaml
@@ -231,6 +231,7 @@ libraries:
- com.fasterxml.jackson.jaxrs: jackson-jaxrs-json-provider
- com.fasterxml.jackson.jaxrs: jackson-jaxrs-smile-provider
- com.fasterxml.jackson.module: jackson-module-jaxb-annotations
+ - com.fasterxml.jackson.module: jackson-module-guice
notice: |
# Jackson JSON processor
@@ -4126,16 +4127,6 @@ libraries:
---
-name: "Jackson Module: Guice"
-license_category: binary
-module: java-core
-license_name: Apache License version 2.0
-version: 2.6.7
-libraries:
- - com.fasterxml.jackson.module: jackson-module-guice
-
----
-
name: Google APIs Client Library For Java
license_category: binary
module: java-core
diff --git a/server/pom.xml b/server/pom.xml
index caddf48..01b0d03 100644
--- a/server/pom.xml
+++ b/server/pom.xml
@@ -315,6 +315,10 @@
<groupId>io.timeandspace</groupId>
<artifactId>cron-scheduler</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.module</groupId>
+ <artifactId>jackson-module-guice</artifactId>
+ </dependency>
<!-- Tests -->
<dependency>
diff --git a/server/src/main/java/org/apache/druid/guice/StorageNodeModule.java b/server/src/main/java/org/apache/druid/guice/StorageNodeModule.java
index d3750a7..a232f6d 100644
--- a/server/src/main/java/org/apache/druid/guice/StorageNodeModule.java
+++ b/server/src/main/java/org/apache/druid/guice/StorageNodeModule.java
@@ -33,11 +33,14 @@ import org.apache.druid.java.util.emitter.EmittingLogger;
import org.apache.druid.query.DruidProcessingConfig;
import org.apache.druid.segment.column.ColumnConfig;
import org.apache.druid.segment.loading.SegmentLoaderConfig;
+import org.apache.druid.segment.loading.StorageLocation;
+import org.apache.druid.segment.loading.StorageLocationSelectorStrategy;
import org.apache.druid.server.DruidNode;
import org.apache.druid.server.coordination.DruidServerMetadata;
import org.apache.druid.server.coordination.ServerType;
import javax.annotation.Nullable;
+import java.util.List;
/**
*/
@@ -52,6 +55,7 @@ public class StorageNodeModule implements Module
{
JsonConfigProvider.bind(binder, "druid.server", DruidServerConfig.class);
JsonConfigProvider.bind(binder, "druid.segmentCache", SegmentLoaderConfig.class);
+ bindLocationSelectorStrategy(binder);
binder.bind(ServerTypeConfig.class).toProvider(Providers.of(null));
binder.bind(ColumnConfig.class).to(DruidProcessingConfig.class);
@@ -117,4 +121,23 @@ public class StorageNodeModule implements Module
{
return !segmentLoaderConfig.getLocations().isEmpty();
}
+
+ /**
+ * provide a list of StorageLocation
+ * so that it can be injected into objects such as implementations of {@link StorageLocationSelectorStrategy}
+ */
+ @Provides
+ @LazySingleton
+ public List<StorageLocation> provideStorageLocation(SegmentLoaderConfig config)
+ {
+ return config.toStorageLocations();
+ }
+
+ /**
+ * a helper method for both storage module and independent unit test cases
+ */
+ public static void bindLocationSelectorStrategy(Binder binder)
+ {
+ JsonConfigProvider.bind(binder, "druid.segmentCache.locationSelector", StorageLocationSelectorStrategy.class);
+ }
}
diff --git a/server/src/main/java/org/apache/druid/segment/loading/LeastBytesUsedStorageLocationSelectorStrategy.java b/server/src/main/java/org/apache/druid/segment/loading/LeastBytesUsedStorageLocationSelectorStrategy.java
index e1e9418..bcc37e3 100644
--- a/server/src/main/java/org/apache/druid/segment/loading/LeastBytesUsedStorageLocationSelectorStrategy.java
+++ b/server/src/main/java/org/apache/druid/segment/loading/LeastBytesUsedStorageLocationSelectorStrategy.java
@@ -19,6 +19,8 @@
package org.apache.druid.segment.loading;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
import com.google.common.collect.Ordering;
import java.util.Comparator;
@@ -34,9 +36,10 @@ public class LeastBytesUsedStorageLocationSelectorStrategy implements StorageLoc
private static final Ordering<StorageLocation> ORDERING = Ordering.from(Comparator
.comparingLong(StorageLocation::currSizeBytes));
- private List<StorageLocation> storageLocations;
+ private final List<StorageLocation> storageLocations;
- public LeastBytesUsedStorageLocationSelectorStrategy(List<StorageLocation> storageLocations)
+ @JsonCreator
+ public LeastBytesUsedStorageLocationSelectorStrategy(@JacksonInject final List<StorageLocation> storageLocations)
{
this.storageLocations = storageLocations;
}
diff --git a/server/src/main/java/org/apache/druid/segment/loading/MostAvailableSizeStorageLocationSelectorStrategy.java b/server/src/main/java/org/apache/druid/segment/loading/MostAvailableSizeStorageLocationSelectorStrategy.java
index 4790f90..4e4a84c 100644
--- a/server/src/main/java/org/apache/druid/segment/loading/MostAvailableSizeStorageLocationSelectorStrategy.java
+++ b/server/src/main/java/org/apache/druid/segment/loading/MostAvailableSizeStorageLocationSelectorStrategy.java
@@ -19,6 +19,8 @@
package org.apache.druid.segment.loading;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
import com.google.common.collect.Ordering;
import java.util.Comparator;
@@ -35,9 +37,10 @@ public class MostAvailableSizeStorageLocationSelectorStrategy implements Storage
.comparingLong(StorageLocation::availableSizeBytes)
.reversed());
- private List<StorageLocation> storageLocations;
+ private final List<StorageLocation> storageLocations;
- public MostAvailableSizeStorageLocationSelectorStrategy(List<StorageLocation> storageLocations)
+ @JsonCreator
+ public MostAvailableSizeStorageLocationSelectorStrategy(@JacksonInject final List<StorageLocation> storageLocations)
{
this.storageLocations = storageLocations;
}
diff --git a/server/src/main/java/org/apache/druid/segment/loading/RandomStorageLocationSelectorStrategy.java b/server/src/main/java/org/apache/druid/segment/loading/RandomStorageLocationSelectorStrategy.java
index 85df860..ed0114a 100644
--- a/server/src/main/java/org/apache/druid/segment/loading/RandomStorageLocationSelectorStrategy.java
+++ b/server/src/main/java/org/apache/druid/segment/loading/RandomStorageLocationSelectorStrategy.java
@@ -19,6 +19,9 @@
package org.apache.druid.segment.loading;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
@@ -33,7 +36,8 @@ public class RandomStorageLocationSelectorStrategy implements StorageLocationSel
private final List<StorageLocation> storageLocations;
- public RandomStorageLocationSelectorStrategy(List<StorageLocation> storageLocations)
+ @JsonCreator
+ public RandomStorageLocationSelectorStrategy(@JacksonInject final List<StorageLocation> storageLocations)
{
this.storageLocations = storageLocations;
}
@@ -45,5 +49,4 @@ public class RandomStorageLocationSelectorStrategy implements StorageLocationSel
Collections.shuffle(copyLocation);
return copyLocation.iterator();
}
-
}
diff --git a/server/src/main/java/org/apache/druid/segment/loading/RoundRobinStorageLocationSelectorStrategy.java b/server/src/main/java/org/apache/druid/segment/loading/RoundRobinStorageLocationSelectorStrategy.java
index 5e702c9..95e57f5 100644
--- a/server/src/main/java/org/apache/druid/segment/loading/RoundRobinStorageLocationSelectorStrategy.java
+++ b/server/src/main/java/org/apache/druid/segment/loading/RoundRobinStorageLocationSelectorStrategy.java
@@ -19,6 +19,9 @@
package org.apache.druid.segment.loading;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+
import java.util.Iterator;
import java.util.List;
import java.util.NoSuchElementException;
@@ -32,11 +35,11 @@ import java.util.concurrent.atomic.AtomicInteger;
*/
public class RoundRobinStorageLocationSelectorStrategy implements StorageLocationSelectorStrategy
{
-
private final List<StorageLocation> storageLocations;
private final AtomicInteger startIndex = new AtomicInteger(0);
- public RoundRobinStorageLocationSelectorStrategy(List<StorageLocation> storageLocations)
+ @JsonCreator
+ public RoundRobinStorageLocationSelectorStrategy(@JacksonInject List<StorageLocation> storageLocations)
{
this.storageLocations = storageLocations;
}
@@ -44,7 +47,8 @@ public class RoundRobinStorageLocationSelectorStrategy implements StorageLocatio
@Override
public Iterator<StorageLocation> getLocations()
{
- return new Iterator<StorageLocation>() {
+ return new Iterator<StorageLocation>()
+ {
private final int numStorageLocations = storageLocations.size();
private int remainingIterations = numStorageLocations;
@@ -73,5 +77,4 @@ public class RoundRobinStorageLocationSelectorStrategy implements StorageLocatio
}
};
}
-
}
diff --git a/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderConfig.java b/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderConfig.java
index edd2c0d..b2cf075 100644
--- a/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderConfig.java
+++ b/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderConfig.java
@@ -20,7 +20,6 @@
package org.apache.druid.segment.loading;
import com.fasterxml.jackson.annotation.JsonProperty;
-import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Lists;
import org.apache.druid.utils.JvmUtils;
@@ -28,6 +27,7 @@ import java.io.File;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
/**
*
@@ -55,9 +55,6 @@ public class SegmentLoaderConfig
@JsonProperty("numBootstrapThreads")
private Integer numBootstrapThreads = null;
- @JsonProperty("locationSelectorStrategy")
- private StorageLocationSelectorStrategy locationSelectorStrategy;
-
@JsonProperty
private File infoDir = null;
@@ -101,15 +98,6 @@ public class SegmentLoaderConfig
return numBootstrapThreads == null ? numLoadingThreads : numBootstrapThreads;
}
- public StorageLocationSelectorStrategy getStorageLocationSelectorStrategy(List<StorageLocation> storageLocations)
- {
- if (locationSelectorStrategy == null) {
- // default strategy if no strategy is specified in the config
- locationSelectorStrategy = new LeastBytesUsedStorageLocationSelectorStrategy(storageLocations);
- }
- return locationSelectorStrategy;
- }
-
public File getInfoDir()
{
if (infoDir == null) {
@@ -140,11 +128,19 @@ public class SegmentLoaderConfig
return retVal;
}
- @VisibleForTesting
- SegmentLoaderConfig withStorageLocationSelectorStrategy(StorageLocationSelectorStrategy strategy)
+ /**
+ * Convert StorageLocationConfig objects to StorageLocation objects
+ *
+ * Note: {@link #getLocations} is called instead of variable access because some testcases overrides this method
+ */
+ public List<StorageLocation> toStorageLocations()
{
- this.locationSelectorStrategy = strategy;
- return this;
+ return this.getLocations()
+ .stream()
+ .map(locationConfig -> new StorageLocation(locationConfig.getPath(),
+ locationConfig.getMaxSize(),
+ locationConfig.getFreeSpacePercent()))
+ .collect(Collectors.toList());
}
@Override
@@ -154,7 +150,6 @@ public class SegmentLoaderConfig
"locations=" + locations +
", deleteOnRemove=" + deleteOnRemove +
", dropSegmentDelayMillis=" + dropSegmentDelayMillis +
- ", locationSelectorStrategy=" + locationSelectorStrategy +
", infoDir=" + infoDir +
'}';
}
diff --git a/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java b/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
index 80e8dc6..2aa2038 100644
--- a/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
+++ b/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
@@ -30,9 +30,9 @@ import org.apache.druid.segment.IndexIO;
import org.apache.druid.segment.Segment;
import org.apache.druid.timeline.DataSegment;
+import javax.annotation.Nonnull;
import java.io.File;
import java.io.IOException;
-import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.ConcurrentHashMap;
@@ -82,24 +82,48 @@ public class SegmentLoaderLocalCacheManager implements SegmentLoader
@Inject
public SegmentLoaderLocalCacheManager(
IndexIO indexIO,
+ List<StorageLocation> locations,
SegmentLoaderConfig config,
+ @Nonnull StorageLocationSelectorStrategy strategy,
@Json ObjectMapper mapper
)
{
this.indexIO = indexIO;
this.config = config;
this.jsonMapper = mapper;
- this.locations = new ArrayList<>();
- for (StorageLocationConfig locationConfig : config.getLocations()) {
- locations.add(
- new StorageLocation(
- locationConfig.getPath(),
- locationConfig.getMaxSize(),
- locationConfig.getFreeSpacePercent()
- )
- );
- }
- this.strategy = config.getStorageLocationSelectorStrategy(locations);
+ this.locations = locations;
+ this.strategy = strategy;
+ log.info("Using storage location strategy: [%s]", this.strategy.getClass().getSimpleName());
+ }
+
+ @VisibleForTesting
+ SegmentLoaderLocalCacheManager(
+ IndexIO indexIO,
+ SegmentLoaderConfig config,
+ @Nonnull StorageLocationSelectorStrategy strategy,
+ @Json ObjectMapper mapper
+ )
+ {
+ this(indexIO, config.toStorageLocations(), config, strategy, mapper);
+ }
+
+ /**
+ * creates instance with default storage location selector strategy
+ *
+ * This ctor is mainly for test cases, including test cases in other modules
+ */
+ public SegmentLoaderLocalCacheManager(
+ IndexIO indexIO,
+ SegmentLoaderConfig config,
+ @Json ObjectMapper mapper
+ )
+ {
+ this.indexIO = indexIO;
+ this.config = config;
+ this.jsonMapper = mapper;
+ this.locations = config.toStorageLocations();
+ this.strategy = new LeastBytesUsedStorageLocationSelectorStrategy(locations);
+ log.info("Using storage location strategy: [%s]", this.strategy.getClass().getSimpleName());
}
@Override
diff --git a/server/src/main/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategy.java b/server/src/main/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategy.java
index adc7388..57977d6 100644
--- a/server/src/main/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategy.java
+++ b/server/src/main/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategy.java
@@ -34,7 +34,7 @@ import java.util.Iterator;
* https://github.com/apache/druid/pull/8038#discussion_r325520829 of PR https://github
* .com/apache/druid/pull/8038 for more details.
*/
-@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type", defaultImpl =
+@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "strategy", defaultImpl =
LeastBytesUsedStorageLocationSelectorStrategy.class)
@JsonSubTypes(value = {
@JsonSubTypes.Type(name = "leastBytesUsed", value = LeastBytesUsedStorageLocationSelectorStrategy.class),
diff --git a/server/src/test/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManagerTest.java b/server/src/test/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManagerTest.java
index 3130696..a01aef4 100644
--- a/server/src/test/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManagerTest.java
+++ b/server/src/test/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManagerTest.java
@@ -422,9 +422,8 @@ public class SegmentLoaderLocalCacheManagerTest
manager = new SegmentLoaderLocalCacheManager(
TestHelper.getTestIndexIO(),
- new SegmentLoaderConfig().withLocations(locationConfigs).withStorageLocationSelectorStrategy(
- new RoundRobinStorageLocationSelectorStrategy(locations)
- ),
+ new SegmentLoaderConfig().withLocations(locationConfigs),
+ new RoundRobinStorageLocationSelectorStrategy(locations),
jsonMapper
);
final File segmentSrcFolder = tmpFolder.newFolder("segmentSrcFolder");
@@ -669,21 +668,12 @@ public class SegmentLoaderLocalCacheManagerTest
locationConfigs.add(locationConfig2);
locationConfigs.add(locationConfig3);
- List<StorageLocation> locations = new ArrayList<>();
- for (StorageLocationConfig locConfig : locationConfigs) {
- locations.add(
- new StorageLocation(
- locConfig.getPath(),
- locConfig.getMaxSize(),
- null
- )
- );
- }
+ SegmentLoaderConfig segmentLoaderConfig = new SegmentLoaderConfig().withLocations(locationConfigs);
+
manager = new SegmentLoaderLocalCacheManager(
TestHelper.getTestIndexIO(),
- new SegmentLoaderConfig().withLocations(locationConfigs).withStorageLocationSelectorStrategy(
- new RandomStorageLocationSelectorStrategy(locations)
- ),
+ new SegmentLoaderConfig().withLocations(locationConfigs),
+ new RandomStorageLocationSelectorStrategy(segmentLoaderConfig.toStorageLocations()),
jsonMapper
);
diff --git a/server/src/test/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategyTest.java b/server/src/test/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategyTest.java
index 82cbe70..a2bfb5b 100644
--- a/server/src/test/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategyTest.java
+++ b/server/src/test/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategyTest.java
@@ -19,16 +19,31 @@
package org.apache.druid.segment.loading;
+import com.fasterxml.jackson.module.guice.ObjectMapperModule;
+import com.google.inject.Binder;
+import com.google.inject.Guice;
+import com.google.inject.Injector;
+import com.google.inject.Module;
+import com.google.inject.Provides;
+import org.apache.druid.guice.DruidGuiceExtensions;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.JsonConfigurator;
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.guice.StorageNodeModule;
+import org.apache.druid.jackson.DefaultObjectMapper;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
+import javax.validation.Validation;
+import javax.validation.Validator;
import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
+import java.util.Properties;
public class StorageLocationSelectorStrategyTest
{
@@ -256,4 +271,107 @@ public class StorageLocationSelectorStrategyTest
Assert.assertEquals("The next element of the iterator should point to path local_storage_folder_1",
localStorageFolder1, loc3.getPath());
}
+
+ @Test
+ public void testDefaultSelectorStrategyConfig()
+ {
+ //no druid.segmentCache.locationSelector.strategy specified, the default will be used
+ final Properties props = new Properties();
+ props.setProperty("druid.segmentCache.locations", "[{\"path\": \"/tmp/druid/indexCache\"}]");
+
+ StorageLocationSelectorStrategy strategy = makeInjectorWithProperties(props).getInstance(StorageLocationSelectorStrategy.class);
+ Assert.assertEquals(LeastBytesUsedStorageLocationSelectorStrategy.class,
+ strategy.getClass());
+ Assert.assertEquals("/tmp/druid/indexCache", strategy.getLocations().next().getPath().getAbsolutePath());
+ }
+
+ @Test
+ public void testRoundRobinSelectorStrategyConfig()
+ {
+ final Properties props = new Properties();
+ props.setProperty("druid.segmentCache.locations", "[{\"path\": \"/tmp/druid/indexCache\"}]");
+ props.setProperty("druid.segmentCache.locationSelector.strategy", "roundRobin");
+
+ Injector injector = makeInjectorWithProperties(props);
+ StorageLocationSelectorStrategy strategy = injector.getInstance(StorageLocationSelectorStrategy.class);
+
+ Assert.assertEquals(RoundRobinStorageLocationSelectorStrategy.class,
+ strategy.getClass());
+ Assert.assertEquals("/tmp/druid/indexCache", strategy.getLocations().next().getPath().getAbsolutePath());
+ }
+
+ @Test
+ public void testLeastBytesUsedSelectorStrategyConfig()
+ {
+ final Properties props = new Properties();
+ props.setProperty("druid.segmentCache.locations", "[{\"path\": \"/tmp/druid/indexCache\"}]");
+ props.setProperty("druid.segmentCache.locationSelector.strategy", "leastBytesUsed");
+
+ Injector injector = makeInjectorWithProperties(props);
+ StorageLocationSelectorStrategy strategy = injector.getInstance(StorageLocationSelectorStrategy.class);
+
+ Assert.assertEquals(LeastBytesUsedStorageLocationSelectorStrategy.class,
+ strategy.getClass());
+ Assert.assertEquals("/tmp/druid/indexCache", strategy.getLocations().next().getPath().getAbsolutePath());
+ }
+
+ @Test
+ public void testRandomSelectorStrategyConfig()
+ {
+ final Properties props = new Properties();
+ props.setProperty("druid.segmentCache.locations", "[{\"path\": \"/tmp/druid/indexCache\"}]");
+ props.setProperty("druid.segmentCache.locationSelector.strategy", "random");
+
+ Injector injector = makeInjectorWithProperties(props);
+ StorageLocationSelectorStrategy strategy = injector.getInstance(StorageLocationSelectorStrategy.class);
+
+ Assert.assertEquals(RandomStorageLocationSelectorStrategy.class,
+ strategy.getClass());
+ Assert.assertEquals("/tmp/druid/indexCache", strategy.getLocations().next().getPath().getAbsolutePath());
+ }
+
+ @Test
+ public void testMostAvailableSizeSelectorStrategyConfig()
+ {
+ final Properties props = new Properties();
+ props.setProperty("druid.segmentCache.locationSelector.strategy", "mostAvailableSize");
+ props.setProperty("druid.segmentCache.locations", "[{\"path\": \"/tmp/druid/indexCache\"}]");
+
+ Injector injector = makeInjectorWithProperties(props);
+ StorageLocationSelectorStrategy strategy = injector.getInstance(StorageLocationSelectorStrategy.class);
+
+ Assert.assertEquals(MostAvailableSizeStorageLocationSelectorStrategy.class,
+ strategy.getClass());
+ Assert.assertEquals("/tmp/druid/indexCache", strategy.getLocations().next().getPath().getAbsolutePath());
+ }
+
+ private Injector makeInjectorWithProperties(final Properties props)
+ {
+ return Guice.createInjector(
+ new Module()
+ {
+ @Override
+ public void configure(Binder binder)
+ {
+ //ObjectMapperModule introduce Guice injector for jackson
+ binder.install(new ObjectMapperModule()
+ .withObjectMapper(new DefaultObjectMapper()));
+ binder.install(new DruidGuiceExtensions());
+
+ binder.bind(Validator.class).toInstance(Validation.buildDefaultValidatorFactory().getValidator());
+ binder.bind(JsonConfigurator.class).in(LazySingleton.class);
+ binder.bind(Properties.class).toInstance(props);
+
+ JsonConfigProvider.bind(binder, "druid.segmentCache", SegmentLoaderConfig.class);
+ StorageNodeModule.bindLocationSelectorStrategy(binder);
+ }
+
+ @Provides
+ public List provideStorageLocation(SegmentLoaderConfig segmentLoader)
+ {
+ return segmentLoader.toStorageLocations();
+ }
+ }
+ );
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org