You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by ca...@apache.org on 2022/09/02 13:51:33 UTC
[druid] branch master updated: Ease of hidding sensitive properties from /status/proper… (#12950)
This is an automated email from the ASF dual-hosted git repository.
capistrant pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 6805a7f9c2 Ease of hidding sensitive properties from /status/proper… (#12950)
6805a7f9c2 is described below
commit 6805a7f9c243445b95090e9aa777512f6411b506
Author: zemin <94...@users.noreply.github.com>
AuthorDate: Fri Sep 2 15:51:25 2022 +0200
Ease of hidding sensitive properties from /status/proper… (#12950)
* apache#12063 Ease of hidding sensitive properties from /status/properties endpoint
* apache#12063 Ease of hidding sensitive properties from /status/properties endpoint
* apache#12063 Ease of hidding sensitive properties from /status/properties endpoint
using one property for hiding properties, updated the index.md to document hiddenProperties
* apache#12063 Ease of hidding sensitive properties from /status/properties endpoint
Added java docs
* apache#12063 Ease of hidding sensitive properties from /status/properties endpoint
Add "password", "key", "token", "pwd" as default druid.server.hiddenProperties
fixed typo and removed redundant space
Co-authored-by: zemin <ze...@adyen.com>
---
docs/configuration/index.md | 8 ++++++
.../cluster/_common/common.runtime.properties | 2 +-
.../large/_common/common.runtime.properties | 2 +-
.../medium/_common/common.runtime.properties | 2 +-
.../_common/common.runtime.properties | 2 +-
.../_common/common.runtime.properties | 2 +-
.../small/_common/common.runtime.properties | 2 +-
.../xlarge/_common/common.runtime.properties | 2 +-
.../org/apache/druid/client/DruidServerConfig.java | 4 ++-
.../org/apache/druid/server/StatusResource.java | 31 +++++++++++++++++++---
.../apache/druid/server/StatusResourceTest.java | 31 +++++++++++++++++++---
.../status.resource.test.runtime.hpc.properties | 30 +++++++++++++++++++++
12 files changed, 104 insertions(+), 14 deletions(-)
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index 03e73db185..8d5723dae9 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -770,6 +770,14 @@ All Druid components can communicate with each other over HTTP.
|`druid.global.http.unusedConnectionTimeout`|The timeout for idle connections in connection pool. The connection in the pool will be closed after this timeout and a new one will be established. This timeout should be less than `druid.global.http.readTimeout`. Set this timeout = ~90% of `druid.global.http.readTimeout`|`PT4M`|
|`druid.global.http.numMaxThreads`|Maximum number of I/O worker threads|`max(10, ((number of cores * 17) / 16 + 2) + 30)`|
+### Common endpoints Configuration
+
+This section contains the configuration options for endpoints that are supported by all processes.
+
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
+|`druid.server.hiddenProperties`| If property names or substring of property names (case insensitive) is in this list, responses of the `/status/properties` endpoint do not show these properties | `["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"]` |
+
## Master Server
This section contains the configuration options for the processes that reside on Master servers (Coordinators and Overlords) in the suggested [three-server configuration](../design/processes.md#server-types).
diff --git a/examples/conf/druid/cluster/_common/common.runtime.properties b/examples/conf/druid/cluster/_common/common.runtime.properties
index eafa11742d..b0adb0695c 100644
--- a/examples/conf/druid/cluster/_common/common.runtime.properties
+++ b/examples/conf/druid/cluster/_common/common.runtime.properties
@@ -131,7 +131,7 @@ druid.indexing.doubleStorage=double
#
# Security
#
-druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]
+druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"]
#
diff --git a/examples/conf/druid/single-server/large/_common/common.runtime.properties b/examples/conf/druid/single-server/large/_common/common.runtime.properties
index eafa11742d..b0adb0695c 100644
--- a/examples/conf/druid/single-server/large/_common/common.runtime.properties
+++ b/examples/conf/druid/single-server/large/_common/common.runtime.properties
@@ -131,7 +131,7 @@ druid.indexing.doubleStorage=double
#
# Security
#
-druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]
+druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"]
#
diff --git a/examples/conf/druid/single-server/medium/_common/common.runtime.properties b/examples/conf/druid/single-server/medium/_common/common.runtime.properties
index eafa11742d..b0adb0695c 100644
--- a/examples/conf/druid/single-server/medium/_common/common.runtime.properties
+++ b/examples/conf/druid/single-server/medium/_common/common.runtime.properties
@@ -131,7 +131,7 @@ druid.indexing.doubleStorage=double
#
# Security
#
-druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]
+druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"]
#
diff --git a/examples/conf/druid/single-server/micro-quickstart/_common/common.runtime.properties b/examples/conf/druid/single-server/micro-quickstart/_common/common.runtime.properties
index eafa11742d..b0adb0695c 100644
--- a/examples/conf/druid/single-server/micro-quickstart/_common/common.runtime.properties
+++ b/examples/conf/druid/single-server/micro-quickstart/_common/common.runtime.properties
@@ -131,7 +131,7 @@ druid.indexing.doubleStorage=double
#
# Security
#
-druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]
+druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"]
#
diff --git a/examples/conf/druid/single-server/nano-quickstart/_common/common.runtime.properties b/examples/conf/druid/single-server/nano-quickstart/_common/common.runtime.properties
index eafa11742d..b0adb0695c 100644
--- a/examples/conf/druid/single-server/nano-quickstart/_common/common.runtime.properties
+++ b/examples/conf/druid/single-server/nano-quickstart/_common/common.runtime.properties
@@ -131,7 +131,7 @@ druid.indexing.doubleStorage=double
#
# Security
#
-druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]
+druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"]
#
diff --git a/examples/conf/druid/single-server/small/_common/common.runtime.properties b/examples/conf/druid/single-server/small/_common/common.runtime.properties
index eafa11742d..b0adb0695c 100644
--- a/examples/conf/druid/single-server/small/_common/common.runtime.properties
+++ b/examples/conf/druid/single-server/small/_common/common.runtime.properties
@@ -131,7 +131,7 @@ druid.indexing.doubleStorage=double
#
# Security
#
-druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]
+druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"]
#
diff --git a/examples/conf/druid/single-server/xlarge/_common/common.runtime.properties b/examples/conf/druid/single-server/xlarge/_common/common.runtime.properties
index eafa11742d..b0adb0695c 100644
--- a/examples/conf/druid/single-server/xlarge/_common/common.runtime.properties
+++ b/examples/conf/druid/single-server/xlarge/_common/common.runtime.properties
@@ -131,7 +131,7 @@ druid.indexing.doubleStorage=double
#
# Security
#
-druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]
+druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"]
#
diff --git a/server/src/main/java/org/apache/druid/client/DruidServerConfig.java b/server/src/main/java/org/apache/druid/client/DruidServerConfig.java
index 7c15789e5e..3ab6e87c00 100644
--- a/server/src/main/java/org/apache/druid/client/DruidServerConfig.java
+++ b/server/src/main/java/org/apache/druid/client/DruidServerConfig.java
@@ -47,7 +47,9 @@ public class DruidServerConfig
@JsonProperty
@NotNull
- private Set<String> hiddenProperties = Sets.newHashSet("druid.s3.accessKey", "druid.s3.secretKey", "druid.metadata.storage.connector.password");
+ private Set<String> hiddenProperties = Sets.newHashSet("druid.s3.accessKey", "druid.s3.secretKey",
+ "druid.metadata.storage.connector.password",
+ "password", "key", "token", "pwd");
private SegmentLoaderConfig segmentLoaderConfig;
diff --git a/server/src/main/java/org/apache/druid/server/StatusResource.java b/server/src/main/java/org/apache/druid/server/StatusResource.java
index ed4c626a37..2476cf4238 100644
--- a/server/src/main/java/org/apache/druid/server/StatusResource.java
+++ b/server/src/main/java/org/apache/druid/server/StatusResource.java
@@ -32,6 +32,7 @@ import org.apache.druid.server.http.security.StateResourceFilter;
import org.apache.druid.utils.JvmUtils;
import org.apache.druid.utils.RuntimeInfo;
+import javax.annotation.Nonnull;
import javax.inject.Inject;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.GET;
@@ -48,6 +49,7 @@ import java.util.Properties;
import java.util.Set;
/**
+ *
*/
@Path("/status")
public class StatusResource
@@ -60,7 +62,8 @@ public class StatusResource
public StatusResource(
final Properties properties,
final DruidServerConfig druidServerConfig,
- final ExtensionsLoader extnLoader)
+ final ExtensionsLoader extnLoader
+ )
{
this.properties = properties;
this.druidServerConfig = druidServerConfig;
@@ -74,8 +77,30 @@ public class StatusResource
public Map<String, String> getProperties()
{
Map<String, String> allProperties = Maps.fromProperties(properties);
- Set<String> hidderProperties = druidServerConfig.getHiddenProperties();
- return Maps.filterEntries(allProperties, (entry) -> !hidderProperties.contains(entry.getKey()));
+ Set<String> hiddenProperties = druidServerConfig.getHiddenProperties();
+ return filterHiddenProperties(hiddenProperties, allProperties);
+ }
+
+ /**
+ * filter out entries from allProperties with key containing elements in hiddenProperties (case insensitive)
+ *
+ * @return map of properties that are not filtered out.
+ */
+ @Nonnull
+ private Map<String, String> filterHiddenProperties(
+ Set<String> hiddenProperties,
+ Map<String, String> allProperties
+ )
+ {
+ return Maps.filterEntries(
+ allProperties,
+ (entry) -> hiddenProperties
+ .stream()
+ .anyMatch(
+ hiddenPropertyElement ->
+ !StringUtils.toLowerCase(entry.getKey()).contains(StringUtils.toLowerCase(hiddenPropertyElement))
+ )
+ );
}
@GET
diff --git a/server/src/test/java/org/apache/druid/server/StatusResourceTest.java b/server/src/test/java/org/apache/druid/server/StatusResourceTest.java
index 0a1de9d01f..e72c34e64b 100644
--- a/server/src/test/java/org/apache/druid/server/StatusResourceTest.java
+++ b/server/src/test/java/org/apache/druid/server/StatusResourceTest.java
@@ -26,6 +26,7 @@ import com.google.inject.Injector;
import org.apache.druid.guice.PropertiesModule;
import org.apache.druid.initialization.DruidModule;
import org.apache.druid.initialization.ServerInjectorBuilderTest;
+import org.apache.druid.java.util.common.StringUtils;
import org.junit.Assert;
import org.junit.Test;
@@ -35,6 +36,7 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.stream.Collectors;
public class StatusResourceTest
{
@@ -62,13 +64,36 @@ public class StatusResourceTest
}
@Test
- public void testPropertiesWithRestrictedConfigs()
+ public void testHiddenProperties()
+ {
+ testHiddenPropertiesWithPropertyFileName("status.resource.test.runtime.properties");
+ }
+
+ @Test
+ public void testHiddenPropertiesContain()
+ {
+ testHiddenPropertiesWithPropertyFileName("status.resource.test.runtime.hpc.properties");
+ }
+
+ private void testHiddenPropertiesWithPropertyFileName(String fileName)
{
Injector injector = Guice.createInjector(Collections.singletonList(new PropertiesModule(Collections.singletonList(
- "status.resource.test.runtime.properties"))));
+ fileName))));
Map<String, String> returnedProperties = injector.getInstance(StatusResource.class).getProperties();
+ Set<String> lowerCasePropertyNames = returnedProperties.keySet()
+ .stream()
+ .map(StringUtils::toLowerCase)
+ .collect(Collectors.toSet());
Set<String> hiddenProperties = new HashSet<>();
Splitter.on(",").split(returnedProperties.get("druid.server.hiddenProperties")).forEach(hiddenProperties::add);
- hiddenProperties.forEach((property) -> Assert.assertNull(returnedProperties.get(property)));
+ hiddenProperties.forEach(
+ (property) -> {
+ lowerCasePropertyNames.forEach(
+ lowerCasePropertyName -> Assert.assertFalse(lowerCasePropertyName.contains(StringUtils.toLowerCase(
+ property)))
+ );
+ }
+ );
}
+
}
diff --git a/server/src/test/resources/status.resource.test.runtime.hpc.properties b/server/src/test/resources/status.resource.test.runtime.hpc.properties
new file mode 100644
index 0000000000..5d278669dc
--- /dev/null
+++ b/server/src/test/resources/status.resource.test.runtime.hpc.properties
@@ -0,0 +1,30 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+druid.server.hiddenProperties=["password","key","token","pwd"]
+druid.storage.type=s3
+druid.storage.bucket=your-bucket
+druid.storage.baseKey=druid/segments
+druid.s3.accessKey=s3accesskey
+druid.s3.secretKey=s3secretkey
+
+druid.metadata.storage.type=mysql
+druid.metadata.storage.connector.connectURI=jdbc:mysql://db.example.com:3306/druid
+druid.metadata.storage.connector.user=druiduser
+druid.metadata.storage.connector.password=password123
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org