You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by ma...@apache.org on 2019/09/12 20:40:29 UTC

[tomcat] branch master updated (d7d4374 -> cae17a5)

This is an automated email from the ASF dual-hosted git repository.

markt pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/tomcat.git.


    from d7d4374  Fix https://bz.apache.org/bugzilla/show_bug.cgi?id=63710
     new 583a9f4  PropertySource: Add an environment variable based source
     new cae17a5  Additional changes required to enable EnvironmentPropertySource

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/tomcat/util/IntrospectionUtils.java | 49 ++++++++++++++++++++--
 java/org/apache/tomcat/util/digester/Digester.java | 40 ++++++++++++++----
 webapps/docs/changelog.xml                         |  6 +++
 webapps/docs/config/systemprops.xml                |  9 +++-
 4 files changed, 91 insertions(+), 13 deletions(-)


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Re: [tomcat] 02/02: Additional changes required to enable EnvironmentPropertySource

Posted by Felix Schumacher <fe...@internetallee.de>.
Am 15.09.19 um 11:03 schrieb Mark Thomas:
> On 14/09/2019 20:01, Felix Schumacher wrote:
>> Am 12.09.19 um 22:40 schrieb markt@apache.org:
>>> This is an automated email from the ASF dual-hosted git repository.
>>>
>>> markt pushed a commit to branch master
>>> in repository https://gitbox.apache.org/repos/asf/tomcat.git
>>>
>>> commit cae17a52598393680952aa21cee0e27b13a73455
>>> Author: Mark Thomas <ma...@apache.org>
>>> AuthorDate: Thu Sep 12 15:31:26 2019 +0100
>>>
>>>     Additional changes required to enable EnvironmentPropertySource
>>> ---
>>>  .../org/apache/tomcat/util/IntrospectionUtils.java | 49 ++++++++++++++++++++--
>>>  java/org/apache/tomcat/util/digester/Digester.java | 33 ++++++++++-----
>>>  webapps/docs/changelog.xml                         |  4 +-
>>>  3 files changed, 69 insertions(+), 17 deletions(-)
>>>
>>> diff --git a/java/org/apache/tomcat/util/IntrospectionUtils.java b/java/org/apache/tomcat/util/IntrospectionUtils.java
>>> index 3ffa702..f6ac737 100644
>>> --- a/java/org/apache/tomcat/util/IntrospectionUtils.java
>>> +++ b/java/org/apache/tomcat/util/IntrospectionUtils.java
>>> @@ -476,9 +499,27 @@ public final class IntrospectionUtils {
>>>      // This provides a layer of abstraction
>>>  
>>>      public static interface PropertySource {
>>> -
>>>          public String getProperty(String key);
>>> -
>>>      }
>>>  
>>> +
>>> +    public static interface PropertySourceSecure extends PropertySource {
>> I think a better name would be SecurePropertySource or
>> ClassloaderAwarePropertySource. The thing that it represents should be
>> at the end of the name IMHO.
> Fair enough. I prefer "SecurePropertySource" so I'll go with that before
> I tag.
>
>> At work I prototyped a similar approach and introduced a
>> NamespaceAwarePropertySource. It is basically an interface that has a
>> getNamespace() method that returns a prefix for the keys. I think that
>> it would be nice if these two approaches.
> Sorry, I'm not quite understanding how this works or the use case it is
> trying to address. Could you provide a simple example?

A namespaced PropertySource would look like this

interface NamespacedPropertySource extends PropertySource {
   String getNamespace(); // or getPrefix()
}

Those PropertySources would be registered by the service loader approach
into a map<String, PropertySource> with their namespace as a key.

If a property is looked up with a key, for example "env.hostname", that
key would be split into the namespace (or prefix) and the actual key for
the source. The SecureProperty from the above map (found by the
namespace) would then be asked to resolve the property.

In this setup, the multiplexer that is looking up the source could check
with the security manager whether access to the key is allowed and thus
freeing the implementer of the NamespacedProperty of doing this work.

class MultiPropertySource implements SecurePropertySource { // not a
good name

  Map<String, PropertySource> sources = findThemByServiceLoader();

  String getProperty(String key, ClassLoader classLoader) {
    String[] nameComponents = key.split(":", 2); // uses a colon for
separation as split uses a regex and a dot is a special char in that context
    String namespace = nameComponents[0];
    String realKey = nameComponents[1];
    if (!checkSecurity(namespace, realKey, classLoader)) { // this would
do the security check
      return null;
    }
   
    SecurePropertySource source = sources.get(namespace);
    return source.getProperty(realKey);
  }
}

Does this make sense to you?

Felix

>
>> My prototype didn't try to
>> call a security manager, but with this commit it would be easy to add.
>>
>> On the other hand it uses a ServiceLoader approach to automatically find
>> all NamespaceAwarePropertySources. Do you think this would be a good
>> addition for Tomcat?
> There is an entry in TOMCAT-NEXT around reducing the use of system
> properties. A ServiceLoader approach may be a good solution for some of
> those.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Re: [tomcat] 02/02: Additional changes required to enable EnvironmentPropertySource

Posted by Mark Thomas <ma...@apache.org>.
On 14/09/2019 20:01, Felix Schumacher wrote:
> 
> Am 12.09.19 um 22:40 schrieb markt@apache.org:
>> This is an automated email from the ASF dual-hosted git repository.
>>
>> markt pushed a commit to branch master
>> in repository https://gitbox.apache.org/repos/asf/tomcat.git
>>
>> commit cae17a52598393680952aa21cee0e27b13a73455
>> Author: Mark Thomas <ma...@apache.org>
>> AuthorDate: Thu Sep 12 15:31:26 2019 +0100
>>
>>     Additional changes required to enable EnvironmentPropertySource
>> ---
>>  .../org/apache/tomcat/util/IntrospectionUtils.java | 49 ++++++++++++++++++++--
>>  java/org/apache/tomcat/util/digester/Digester.java | 33 ++++++++++-----
>>  webapps/docs/changelog.xml                         |  4 +-
>>  3 files changed, 69 insertions(+), 17 deletions(-)
>>
>> diff --git a/java/org/apache/tomcat/util/IntrospectionUtils.java b/java/org/apache/tomcat/util/IntrospectionUtils.java
>> index 3ffa702..f6ac737 100644
>> --- a/java/org/apache/tomcat/util/IntrospectionUtils.java
>> +++ b/java/org/apache/tomcat/util/IntrospectionUtils.java
>> @@ -476,9 +499,27 @@ public final class IntrospectionUtils {
>>      // This provides a layer of abstraction
>>  
>>      public static interface PropertySource {
>> -
>>          public String getProperty(String key);
>> -
>>      }
>>  
>> +
>> +    public static interface PropertySourceSecure extends PropertySource {
> 
> I think a better name would be SecurePropertySource or
> ClassloaderAwarePropertySource. The thing that it represents should be
> at the end of the name IMHO.

Fair enough. I prefer "SecurePropertySource" so I'll go with that before
I tag.

> At work I prototyped a similar approach and introduced a
> NamespaceAwarePropertySource. It is basically an interface that has a
> getNamespace() method that returns a prefix for the keys. I think that
> it would be nice if these two approaches.

Sorry, I'm not quite understanding how this works or the use case it is
trying to address. Could you provide a simple example?

> My prototype didn't try to
> call a security manager, but with this commit it would be easy to add.
> 
> On the other hand it uses a ServiceLoader approach to automatically find
> all NamespaceAwarePropertySources. Do you think this would be a good
> addition for Tomcat?

There is an entry in TOMCAT-NEXT around reducing the use of system
properties. A ServiceLoader approach may be a good solution for some of
those.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Re: [tomcat] 02/02: Additional changes required to enable EnvironmentPropertySource

Posted by Felix Schumacher <fe...@internetallee.de>.
Am 12.09.19 um 22:40 schrieb markt@apache.org:
> This is an automated email from the ASF dual-hosted git repository.
>
> markt pushed a commit to branch master
> in repository https://gitbox.apache.org/repos/asf/tomcat.git
>
> commit cae17a52598393680952aa21cee0e27b13a73455
> Author: Mark Thomas <ma...@apache.org>
> AuthorDate: Thu Sep 12 15:31:26 2019 +0100
>
>     Additional changes required to enable EnvironmentPropertySource
> ---
>  .../org/apache/tomcat/util/IntrospectionUtils.java | 49 ++++++++++++++++++++--
>  java/org/apache/tomcat/util/digester/Digester.java | 33 ++++++++++-----
>  webapps/docs/changelog.xml                         |  4 +-
>  3 files changed, 69 insertions(+), 17 deletions(-)
>
> diff --git a/java/org/apache/tomcat/util/IntrospectionUtils.java b/java/org/apache/tomcat/util/IntrospectionUtils.java
> index 3ffa702..f6ac737 100644
> --- a/java/org/apache/tomcat/util/IntrospectionUtils.java
> +++ b/java/org/apache/tomcat/util/IntrospectionUtils.java
> @@ -476,9 +499,27 @@ public final class IntrospectionUtils {
>      // This provides a layer of abstraction
>  
>      public static interface PropertySource {
> -
>          public String getProperty(String key);
> -
>      }
>  
> +
> +    public static interface PropertySourceSecure extends PropertySource {

I think a better name would be SecurePropertySource or
ClassloaderAwarePropertySource. The thing that it represents should be
at the end of the name IMHO.

At work I prototyped a similar approach and introduced a
NamespaceAwarePropertySource. It is basically an interface that has a
getNamespace() method that returns a prefix for the keys. I think that
it would be nice if these two approaches. My prototype didn't try to
call a security manager, but with this commit it would be easy to add.

On the other hand it uses a ServiceLoader approach to automatically find
all NamespaceAwarePropertySources. Do you think this would be a good
addition for Tomcat?

Regards

 Felix

> +
> +        /**
> +         * Obtain a property value, checking that code associated with the
> +         * provided class loader has permission to access the property. If the
> +         * {@code classLoader} is {@code null} or if {@code classLoader} does
> +         * not implement {@link PermissionCheck} then the property value will be
> +         * looked up <b>without</b> a call to
> +         * {@link PermissionCheck#check(java.security.Permission)}
> +         *
> +         * @param key               The key of the requested property
> +         * @param classLoader   The class loader associated with the code that
> +         *                      trigger the property lookup
> +         * @return The property value or {@code null} if it could not be found
> +         *         or if {@link PermissionCheck#check(java.security.Permission)}
> +         *         fails
> +         */
> +        public String getProperty(String key, ClassLoader classLoader);
> +    }

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


[tomcat] 02/02: Additional changes required to enable EnvironmentPropertySource

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

markt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/tomcat.git

commit cae17a52598393680952aa21cee0e27b13a73455
Author: Mark Thomas <ma...@apache.org>
AuthorDate: Thu Sep 12 15:31:26 2019 +0100

    Additional changes required to enable EnvironmentPropertySource
---
 .../org/apache/tomcat/util/IntrospectionUtils.java | 49 ++++++++++++++++++++--
 java/org/apache/tomcat/util/digester/Digester.java | 33 ++++++++++-----
 webapps/docs/changelog.xml                         |  4 +-
 3 files changed, 69 insertions(+), 17 deletions(-)

diff --git a/java/org/apache/tomcat/util/IntrospectionUtils.java b/java/org/apache/tomcat/util/IntrospectionUtils.java
index 3ffa702..f6ac737 100644
--- a/java/org/apache/tomcat/util/IntrospectionUtils.java
+++ b/java/org/apache/tomcat/util/IntrospectionUtils.java
@@ -26,6 +26,7 @@ import java.util.Hashtable;
 import org.apache.juli.logging.Log;
 import org.apache.juli.logging.LogFactory;
 import org.apache.tomcat.util.res.StringManager;
+import org.apache.tomcat.util.security.PermissionCheck;
 
 /**
  * Utils for introspection and reflection
@@ -224,9 +225,27 @@ public final class IntrospectionUtils {
      * @param staticProp Replacement properties
      * @param dynamicProp Replacement properties
      * @return the replacement value
+     * @deprecated Use {@link #replaceProperties(String, Hashtable, PropertySource[], ClassLoader)}
      */
+    @Deprecated
     public static String replaceProperties(String value,
             Hashtable<Object,Object> staticProp, PropertySource dynamicProp[]) {
+        return replaceProperties(value, staticProp, dynamicProp, null);
+    }
+
+    /**
+     * Replace ${NAME} with the property value.
+     * @param value The value
+     * @param staticProp Replacement properties
+     * @param dynamicProp Replacement properties
+     * @param classLoader Class loader associated with the code requesting the
+     *                    property
+     * @return the replacement value
+     */
+    public static String replaceProperties(String value,
+            Hashtable<Object,Object> staticProp, PropertySource dynamicProp[],
+            ClassLoader classLoader) {
+
         if (value.indexOf('$') < 0) {
             return value;
         }
@@ -257,8 +276,12 @@ public final class IntrospectionUtils {
                     v = (String) staticProp.get(n);
                 }
                 if (v == null && dynamicProp != null) {
-                    for (int i = 0; i < dynamicProp.length; i++) {
-                        v = dynamicProp[i].getProperty(n);
+                    for (PropertySource propertySource : dynamicProp) {
+                        if (propertySource instanceof PropertySourceSecure) {
+                            v = ((PropertySourceSecure) propertySource).getProperty(n, classLoader);
+                        } else {
+                            v = propertySource.getProperty(n);
+                        }
                         if (v != null) {
                             break;
                         }
@@ -476,9 +499,27 @@ public final class IntrospectionUtils {
     // This provides a layer of abstraction
 
     public static interface PropertySource {
-
         public String getProperty(String key);
-
     }
 
+
+    public static interface PropertySourceSecure extends PropertySource {
+
+        /**
+         * Obtain a property value, checking that code associated with the
+         * provided class loader has permission to access the property. If the
+         * {@code classLoader} is {@code null} or if {@code classLoader} does
+         * not implement {@link PermissionCheck} then the property value will be
+         * looked up <b>without</b> a call to
+         * {@link PermissionCheck#check(java.security.Permission)}
+         *
+         * @param key               The key of the requested property
+         * @param classLoader   The class loader associated with the code that
+         *                      trigger the property lookup
+         * @return The property value or {@code null} if it could not be found
+         *         or if {@link PermissionCheck#check(java.security.Permission)}
+         *         fails
+         */
+        public String getProperty(String key, ClassLoader classLoader);
+    }
 }
diff --git a/java/org/apache/tomcat/util/digester/Digester.java b/java/org/apache/tomcat/util/digester/Digester.java
index 810274b..610e83c 100644
--- a/java/org/apache/tomcat/util/digester/Digester.java
+++ b/java/org/apache/tomcat/util/digester/Digester.java
@@ -123,13 +123,19 @@ public class Digester extends DefaultHandler2 {
     // --------------------------------------------------- Instance Variables
 
 
-    private class SystemPropertySource implements IntrospectionUtils.PropertySource {
+    private static class SystemPropertySource implements IntrospectionUtils.PropertySourceSecure {
+
         @Override
         public String getProperty(String key) {
-            ClassLoader cl = getClassLoader();
-            if (cl instanceof PermissionCheck) {
+            // For backward compatibility
+            return getProperty(key, null);
+        }
+
+        @Override
+        public String getProperty(String key, ClassLoader classLoader) {
+            if (classLoader instanceof PermissionCheck) {
                 Permission p = new PropertyPermission(key, "read");
-                if (!((PermissionCheck) cl).check(p)) {
+                if (!((PermissionCheck) classLoader).check(p)) {
                     return null;
                 }
             }
@@ -138,13 +144,18 @@ public class Digester extends DefaultHandler2 {
     }
 
 
-    public class EnvironmentPropertySource implements IntrospectionUtils.PropertySource {
+    public static class EnvironmentPropertySource implements IntrospectionUtils.PropertySourceSecure {
+
         @Override
         public String getProperty(String key) {
-            ClassLoader cl = getClassLoader();
-            if (cl instanceof PermissionCheck) {
+            return null;
+        }
+
+        @Override
+        public String getProperty(String key, ClassLoader classLoader) {
+            if (classLoader instanceof PermissionCheck) {
                 Permission p = new RuntimePermission("getenv." + key, null);
-                if (!((PermissionCheck) cl).check(p)) {
+                if (!((PermissionCheck) classLoader).check(p)) {
                     return null;
                 }
             }
@@ -351,7 +362,7 @@ public class Digester extends DefaultHandler2 {
                 String value = System.getProperty(name);
                 if (value != null) {
                     try {
-                        String newValue = IntrospectionUtils.replaceProperties(value, null, propertySources);
+                        String newValue = IntrospectionUtils.replaceProperties(value, null, propertySources, null);
                         if (!value.equals(newValue)) {
                             System.setProperty(name, newValue);
                         }
@@ -1933,7 +1944,7 @@ public class Digester extends DefaultHandler2 {
         for (int i = 0; i < nAttributes; ++i) {
             String value = newAttrs.getValue(i);
             try {
-                newAttrs.setValue(i, IntrospectionUtils.replaceProperties(value, null, source).intern());
+                newAttrs.setValue(i, IntrospectionUtils.replaceProperties(value, null, source, getClassLoader()).intern());
             } catch (Exception e) {
                 log.warn(sm.getString("digester.failedToUpdateAttributes", newAttrs.getLocalName(i), value), e);
             }
@@ -1952,7 +1963,7 @@ public class Digester extends DefaultHandler2 {
         String in = bodyText.toString();
         String out;
         try {
-            out = IntrospectionUtils.replaceProperties(in, null, source);
+            out = IntrospectionUtils.replaceProperties(in, null, source, getClassLoader());
         } catch (Exception e) {
             return bodyText; // return unchanged data
         }
diff --git a/webapps/docs/changelog.xml b/webapps/docs/changelog.xml
index e89ca90..bec625b 100644
--- a/webapps/docs/changelog.xml
+++ b/webapps/docs/changelog.xml
@@ -69,8 +69,8 @@
       <add>
         Add a new <code>PropertySource</code> implementation,
         <code>EnvironmentPropertySource</code>, that can be used to do property
-        replacement in configuration files with environment variables. Pull
-        request provided by Thomas Meyer. (markt)
+        replacement in configuration files with environment variables. Based on
+        a pull request provided by Thomas Meyer. (markt)
       </add>
     </changelog>
   </subsection>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


[tomcat] 01/02: PropertySource: Add an environment variable based source

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

markt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/tomcat.git

commit 583a9f40ed033807db11827596e999f007264adb
Author: Thomas Meyer <th...@m3y3r.de>
AuthorDate: Sat Jul 20 22:22:42 2019 +0200

    PropertySource: Add an environment variable based source
    
    When tomcat runs in an Openshift based container a Secret containing passwords
    can be map as environment variables (with an additional prefix).
    An webapp containing an embedded context.xml which defines JDBC datasources and
    placeholder variables can be used with this new PropertySource to easily inject
    configuration from a Secret or ConfigMap.
---
 java/org/apache/tomcat/util/digester/Digester.java | 15 +++++++++++++++
 webapps/docs/changelog.xml                         |  6 ++++++
 webapps/docs/config/systemprops.xml                |  9 +++++++--
 3 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/java/org/apache/tomcat/util/digester/Digester.java b/java/org/apache/tomcat/util/digester/Digester.java
index e3ed881..810274b 100644
--- a/java/org/apache/tomcat/util/digester/Digester.java
+++ b/java/org/apache/tomcat/util/digester/Digester.java
@@ -138,6 +138,21 @@ public class Digester extends DefaultHandler2 {
     }
 
 
+    public class EnvironmentPropertySource implements IntrospectionUtils.PropertySource {
+        @Override
+        public String getProperty(String key) {
+            ClassLoader cl = getClassLoader();
+            if (cl instanceof PermissionCheck) {
+                Permission p = new RuntimePermission("getenv." + key, null);
+                if (!((PermissionCheck) cl).check(p)) {
+                    return null;
+                }
+            }
+            return System.getenv(key);
+        }
+    }
+
+
     protected IntrospectionUtils.PropertySource source[] = new IntrospectionUtils.PropertySource[] {
             new SystemPropertySource() };
 
diff --git a/webapps/docs/changelog.xml b/webapps/docs/changelog.xml
index 7bdf0e7..e89ca90 100644
--- a/webapps/docs/changelog.xml
+++ b/webapps/docs/changelog.xml
@@ -66,6 +66,12 @@
         Service is embedded directly (i.e. with no Server) in an applciation
         and JNDI is enabled. Patch provided by S. Ali Tokmen. (markt)
       </fix>
+      <add>
+        Add a new <code>PropertySource</code> implementation,
+        <code>EnvironmentPropertySource</code>, that can be used to do property
+        replacement in configuration files with environment variables. Pull
+        request provided by Thomas Meyer. (markt)
+      </add>
     </changelog>
   </subsection>
   <subsection name="Coyote">
diff --git a/webapps/docs/config/systemprops.xml b/webapps/docs/config/systemprops.xml
index a17687f..d49a67d 100644
--- a/webapps/docs/config/systemprops.xml
+++ b/webapps/docs/config/systemprops.xml
@@ -43,11 +43,16 @@
       <p>Set this to a fully qualified name of a class that implements
          <code>org.apache.tomcat.util.IntrospectionUtils.PropertySource</code>.
          Required to have a public constructor with no arguments.</p>
-      <p>Use this to add a property source, that will be invoked when <code>${parameter}</code>
-         denoted parameters are found in the XML files that Tomcat parses.</p>
+      <p>Use this to add a property source, that will be invoked when
+         <code>${parameter}</code> denoted parameters are found in the XML files
+         that Tomcat parses.</p>
       <p>Property replacement from the specified property source on the JVM
          system properties can also be done using the
          <code>REPLACE_SYSTEM_PROPERTIES</code> system property.</p>
+      <p><code>org.apache.tomcat.util.digester.Digester$EnvironmentPropertySource</code>
+         can be used to replace parameters from the process' environment
+         variables, e.g. injected ConfigMaps or Secret objects in container
+         based systems like OpenShift or Kubernetes.</p>
     </property>
     <property name="org.apache.tomcat.util.digester. REPLACE_SYSTEM_PROPERTIES">
       <p>Set this boolean system property to <code>true</code> to cause


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org