You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by arunmahadevan <gi...@git.apache.org> on 2017/04/21 11:59:21 UTC
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
GitHub user arunmahadevan opened a pull request:
https://github.com/apache/storm/pull/2081
[STORM-2482] Refactor the Storm auto credential plugins to be more usable
1. Create a new storm module storm-autocreds
2. Move AutoHDFS and AutoHBase to storm-autocreds
3. Refactor code and accepts config keys for customizing the hadoop configuration for the plugins
4. Package the auto cred and dependency jars in the storm binary and deploy to lib-autocreds
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/arunmahadevan/storm STORM-2482-1.x
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2081.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2081
----
commit aa2fe29982cd688f425568a75281048aa92aca7b
Author: Arun Mahadevan <ar...@apache.org>
Date: 2017-04-18T07:59:56Z
[STORM-2482] Refactor the Storm auto credential plugins to be more usable
1. Create a new storm module storm-autocreds
2. Move AutoHDFS and AutoHBase to storm-autocreds
3. Refactor code and accepts config keys for customizing the hadoop configuration for the plugins
4. Package the auto cred and dependency jars in the storm binary and deploy to lib-autocreds
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2081#discussion_r113517229
--- Diff: docs/storm-hbase.md ---
@@ -56,22 +56,43 @@ The approach described above requires that all potential worker hosts have "stor
multiple topologies on a cluster , each with different hbase user, you will have to create multiple keytabs and distribute
it to all workers. Instead of doing that you could use the following approach:
-Your administrator can configure nimbus to automatically get delegation tokens on behalf of the topology submitter user.
-The nimbus need to start with following configurations:
+Your administrator can configure nimbus to automatically get delegation tokens on behalf of the topology submitter user. The nimbus should be started with following configurations:
+```
nimbus.autocredential.plugins.classes : ["org.apache.storm.hbase.security.AutoHBase"]
nimbus.credential.renewers.classes : ["org.apache.storm.hbase.security.AutoHBase"]
hbase.keytab.file: "/path/to/keytab/on/nimbus" (This is the keytab of hbase super user that can impersonate other users.)
hbase.kerberos.principal: "superuser@EXAMPLE.com"
-nimbus.credential.renewers.freq.secs : 518400 (6 days, hbase tokens by default expire every 7 days and can not be renewed,
-if you have custom settings for hbase.auth.token.max.lifetime in hbase-site.xml than you should ensure this value is
-atleast 1 hour less then that.)
+nimbus.credential.renewers.freq.secs : 518400 (6 days, hbase tokens by default expire every 7 days and can not be renewed, if you have custom settings for hbase.auth.token.max.lifetime in hbase-site.xml than you should ensure this value is atleast 1 hour less then that.)
+```
Your topology configuration should have:
-topology.auto-credentials :["org.apache.storm.hbase.security.AutoHBase"]
+
+```
+topology.auto-credentials :["org.apache.storm.hbase.security.AutoHBase"]
+```
If nimbus did not have the above configuration you need to add it and then restart it. Ensure the hbase configuration
-files(core-site.xml,hdfs-site.xml and hbase-site.xml) and the storm-hbase jar with all the dependencies is present in nimbus's classpath.
+files(core-site.xml, hdfs-site.xml and hbase-site.xml) and the storm-hbase jar with all the dependencies is present in nimbus's classpath.
+
+As an alternative to adding the configuration files (core-site.xml, hdfs-site.xml and hbase-site.xml) to the classpath, you could specify the configurations as a part of the topology configuration. E.g. in you custom storm.yaml (or -c option while submitting the topology),
+
+```
+hbaseCredentialsConfigKeys : ["cluster1", "cluster2"] (the hbase clusters you want to fetch the tokens from)
+cluster1: [{"config1": "value1", "config2": "value2", ... }] (A map of config key-values specific to cluster1)
+cluster2: [{"config1": "value1", "hbase.keytab.file": "/path/to/keytab/for/cluster2/on/nimubs", "hbase.kerberos.principal": "cluster2user@EXAMPLE.com"}] (here along with other configs, we have custom keytab and principal for "cluster2" which will override the keytab/principal specified at topology level)
+```
+
+Instead of specifying key values you may also directly specify the resource files for e.g.,
+
+```
+cluster1: [{"resources": ["/path/to/core-site1.xml", "/path/to/hbase-site1.xml"]}]
+cluster2: [{"resources": ["/path/to/core-site2.xml", "/path/to/hbase-site2.xml"]}]
+```
+
+Storm will download the tokens separately for each of the clusters and populate it into the subject and also renew the tokens periodically.
+This way it would be possible to run multiple bolts connecting to separate HBase cluster within the same topology.
+
--- End diff --
Just a few lines below it says "As nimbus is impersonating..... and it mentions storm.kerberos.principal. Is that right or should we change it to hbase.kerberos.principal ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2081: [STORM-2482] Refactor the Storm auto credential plugins t...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:
https://github.com/apache/storm/pull/2081
@arunmahadevan we need a PR for master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
Posted by arunmahadevan <gi...@git.apache.org>.
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/storm/pull/2081#discussion_r113686513
--- Diff: docs/SECURITY.md ---
@@ -423,16 +423,18 @@ nimbus.impersonation.acl:
### Automatic Credentials Push and Renewal
Individual topologies have the ability to push credentials (tickets and tokens) to workers so that they can access secure services. Exposing this to all of the users can be a pain for them.
-To hide this from them in the common case plugins can be used to populate the credentials, unpack them on the other side into a java Subject, and also allow Nimbus to renew the credentials if needed.
-These are controlled by the following configs. topology.auto-credentials is a list of java plugins, all of which must implement IAutoCredentials interface, that populate the credentials on gateway
-and unpack them on the worker side. On a kerberos secure cluster they should be set by default to point to org.apache.storm.security.auth.kerberos.AutoTGT.
-nimbus.credential.renewers.classes should also be set to this value so that nimbus can periodically renew the TGT on behalf of the user.
+To hide this from them in the common case plugins can be used to populate the credentials, unpack them on the other side into a java Subject, and also allow Nimbus to renew the credentials if needed. These are controlled by the following configs.
+
+`topology.auto-credentials` is a list of java plugins, all of which must implement the `IAutoCredentials` interface, that populate the credentials on gateway
--- End diff --
Gateway is the host from where the topology is submitted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2081#discussion_r113535108
--- Diff: external/storm-autocreds/src/main/java/org/apache/storm/common/AbstractAutoCreds.java ---
@@ -0,0 +1,250 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.common;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.math3.util.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.security.token.TokenIdentifier;
+import org.apache.storm.security.INimbusCredentialPlugin;
+import org.apache.storm.security.auth.IAutoCredentials;
+import org.apache.storm.security.auth.ICredentialsRenewer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import javax.xml.bind.DatatypeConverter;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * The base class that for auto credential plugins that abstracts out some of the common functionality.
+ */
+public abstract class AbstractAutoCreds implements IAutoCredentials, ICredentialsRenewer, INimbusCredentialPlugin {
+ private static final Logger LOG = LoggerFactory.getLogger(AbstractAutoCreds.class);
+ public static final String CONFIG_KEY_RESOURCES = "resources";
+
+ private List<String> configKeys = new ArrayList<>();
+ private Map<String, Map<String, Object>> configMap = new HashMap<>();
+
+ @Override
+ public void prepare(Map conf) {
+ doPrepare(conf);
+ String configKeyString = getConfigKeyString();
+ if (conf.containsKey(configKeyString)) {
+ configKeys.addAll((List<String>) conf.get(configKeyString));
+ for (String key : configKeys) {
+ if (conf.containsKey(key)) {
+ Map<String, Object> config = (Map<String, Object>) conf.get(key);
+ configMap.put(key, config);
+ LOG.info("configKey = {}, config = {}", key, config);
+ }
+ }
+ }
+ }
+
+ @Override
+ public void populateCredentials(Map<String, String> credentials, Map conf) {
+ try {
+ if (configKeys != null) {
+ Map<String, Object> updatedConf = updateConfigs(conf);
+ for (String configKey : configKeys) {
+ credentials.put(getCredentialKey(configKey),
+ DatatypeConverter.printBase64Binary(getHadoopCredentials(updatedConf, configKey)));
+ }
+ } else {
+ credentials.put(getCredentialKey(StringUtils.EMPTY),
+ DatatypeConverter.printBase64Binary(getHadoopCredentials(conf)));
+ }
+ LOG.info("Tokens added to credentials map.");
+ } catch (Exception e) {
+ LOG.error("Could not populate credentials.", e);
+ }
+ }
+
+ private Map<String, Object> updateConfigs(Map topologyConf) {
+ Map<String, Object> res = new HashMap<>(topologyConf);
+ if (configKeys != null) {
+ for (String configKey : configKeys) {
+ if (!res.containsKey(configKey) && configMap.containsKey(configKey)) {
+ res.put(configKey, configMap.get(configKey));
+ }
+ }
+ }
+ return res;
+ }
+
+ @Override
+ public void renew(Map<String, String> credentials, Map topologyConf) {
+ doRenew(credentials, updateConfigs(topologyConf));
+ }
+
+ @Override
+ public void populateCredentials(Map<String, String> credentials) {
+ credentials.put(getCredentialKey(StringUtils.EMPTY),
+ DatatypeConverter.printBase64Binary("dummy place holder".getBytes()));
+ }
+
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void populateSubject(Subject subject, Map<String, String> credentials) {
+ addCredentialToSubject(subject, credentials);
+ addTokensToUGI(subject);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void updateSubject(Subject subject, Map<String, String> credentials) {
+ addCredentialToSubject(subject, credentials);
+ addTokensToUGI(subject);
+ }
+
+ protected Set<Pair<String, Credentials>> getCredentials(Map<String, String> credentials) {
+ Set<Pair<String, Credentials>> res = new HashSet<>();
+ if (configKeys != null) {
+ for (String configKey : configKeys) {
+ Credentials cred = doGetCredentials(credentials, configKey);
+ if (cred != null) {
+ res.add(new Pair(configKey, cred));
+ }
+ }
+ } else {
+ Credentials cred = doGetCredentials(credentials, StringUtils.EMPTY);
+ if (cred != null) {
+ res.add(new Pair(StringUtils.EMPTY, cred));
+ }
+ }
+ return res;
+ }
+
+ protected void fillHadoopConfiguration(Map topoConf, String configKey, Configuration configuration) {
+ Map<String, Object> config = (Map<String, Object>) topoConf.get(configKey);
+ LOG.info("TopoConf {}, got config {}, for configKey {}", topoConf, config, configKey);
+ if (config != null) {
+ List<String> resourcesToLoad = new ArrayList<>();
+ for (Map.Entry<String, Object> entry : config.entrySet()) {
+ if (entry.getKey().equals(CONFIG_KEY_RESOURCES)) {
+ resourcesToLoad.addAll((List<String>) entry.getValue());
+ } else {
+ configuration.set(entry.getKey(), String.valueOf(entry.getValue()));
+ }
+ }
+ LOG.info("Resources to load {}", resourcesToLoad);
+ // add configs from resources like hdfs-site.xml
+ for (String pathStr : resourcesToLoad) {
+ configuration.addResource(new Path(Paths.get(pathStr).toUri()));
+ }
+ }
+ LOG.info("Initializing UGI with config {}", configuration);
+ UserGroupInformation.setConfiguration(configuration);
--- End diff --
Will this work if tow hbase or hdfs bolts connecting to different clusters are running in the same jvm? I think this was always the problem but re-factoring in this PR aims to be able to connect to two different clusters. Just checking if issue still exists.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
Posted by omkreddy <gi...@git.apache.org>.
Github user omkreddy commented on a diff in the pull request:
https://github.com/apache/storm/pull/2081#discussion_r113671063
--- Diff: external/storm-autocreds/src/main/java/org/apache/storm/common/AbstractAutoCreds.java ---
@@ -0,0 +1,250 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.common;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.math3.util.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.security.token.TokenIdentifier;
+import org.apache.storm.security.INimbusCredentialPlugin;
+import org.apache.storm.security.auth.IAutoCredentials;
+import org.apache.storm.security.auth.ICredentialsRenewer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import javax.xml.bind.DatatypeConverter;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * The base class that for auto credential plugins that abstracts out some of the common functionality.
+ */
+public abstract class AbstractAutoCreds implements IAutoCredentials, ICredentialsRenewer, INimbusCredentialPlugin {
+ private static final Logger LOG = LoggerFactory.getLogger(AbstractAutoCreds.class);
+ public static final String CONFIG_KEY_RESOURCES = "resources";
+
+ private List<String> configKeys = new ArrayList<>();
+ private Map<String, Map<String, Object>> configMap = new HashMap<>();
+
+ @Override
+ public void prepare(Map conf) {
+ doPrepare(conf);
+ String configKeyString = getConfigKeyString();
+ if (conf.containsKey(configKeyString)) {
+ configKeys.addAll((List<String>) conf.get(configKeyString));
+ for (String key : configKeys) {
+ if (conf.containsKey(key)) {
+ Map<String, Object> config = (Map<String, Object>) conf.get(key);
+ configMap.put(key, config);
+ LOG.info("configKey = {}, config = {}", key, config);
+ }
+ }
+ }
+ }
+
+ @Override
+ public void populateCredentials(Map<String, String> credentials, Map conf) {
+ try {
+ if (configKeys != null) {
+ Map<String, Object> updatedConf = updateConfigs(conf);
+ for (String configKey : configKeys) {
+ credentials.put(getCredentialKey(configKey),
+ DatatypeConverter.printBase64Binary(getHadoopCredentials(updatedConf, configKey)));
+ }
+ } else {
+ credentials.put(getCredentialKey(StringUtils.EMPTY),
+ DatatypeConverter.printBase64Binary(getHadoopCredentials(conf)));
+ }
+ LOG.info("Tokens added to credentials map.");
+ } catch (Exception e) {
+ LOG.error("Could not populate credentials.", e);
+ }
+ }
+
+ private Map<String, Object> updateConfigs(Map topologyConf) {
+ Map<String, Object> res = new HashMap<>(topologyConf);
+ if (configKeys != null) {
+ for (String configKey : configKeys) {
+ if (!res.containsKey(configKey) && configMap.containsKey(configKey)) {
+ res.put(configKey, configMap.get(configKey));
+ }
+ }
+ }
+ return res;
+ }
+
+ @Override
+ public void renew(Map<String, String> credentials, Map topologyConf) {
+ doRenew(credentials, updateConfigs(topologyConf));
+ }
+
+ @Override
+ public void populateCredentials(Map<String, String> credentials) {
+ credentials.put(getCredentialKey(StringUtils.EMPTY),
+ DatatypeConverter.printBase64Binary("dummy place holder".getBytes()));
+ }
+
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void populateSubject(Subject subject, Map<String, String> credentials) {
+ addCredentialToSubject(subject, credentials);
+ addTokensToUGI(subject);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void updateSubject(Subject subject, Map<String, String> credentials) {
+ addCredentialToSubject(subject, credentials);
+ addTokensToUGI(subject);
+ }
+
+ protected Set<Pair<String, Credentials>> getCredentials(Map<String, String> credentials) {
+ Set<Pair<String, Credentials>> res = new HashSet<>();
+ if (configKeys != null) {
--- End diff --
same as above
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2081: [STORM-2482] Refactor the Storm auto credential plugins t...
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on the issue:
https://github.com/apache/storm/pull/2081
+1(non-binding)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
Posted by omkreddy <gi...@git.apache.org>.
Github user omkreddy commented on a diff in the pull request:
https://github.com/apache/storm/pull/2081#discussion_r113670700
--- Diff: external/storm-autocreds/src/main/java/org/apache/storm/common/AbstractAutoCreds.java ---
@@ -0,0 +1,250 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.common;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.math3.util.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.security.token.TokenIdentifier;
+import org.apache.storm.security.INimbusCredentialPlugin;
+import org.apache.storm.security.auth.IAutoCredentials;
+import org.apache.storm.security.auth.ICredentialsRenewer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import javax.xml.bind.DatatypeConverter;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * The base class that for auto credential plugins that abstracts out some of the common functionality.
+ */
+public abstract class AbstractAutoCreds implements IAutoCredentials, ICredentialsRenewer, INimbusCredentialPlugin {
+ private static final Logger LOG = LoggerFactory.getLogger(AbstractAutoCreds.class);
+ public static final String CONFIG_KEY_RESOURCES = "resources";
+
+ private List<String> configKeys = new ArrayList<>();
+ private Map<String, Map<String, Object>> configMap = new HashMap<>();
+
+ @Override
+ public void prepare(Map conf) {
+ doPrepare(conf);
+ String configKeyString = getConfigKeyString();
+ if (conf.containsKey(configKeyString)) {
+ configKeys.addAll((List<String>) conf.get(configKeyString));
+ for (String key : configKeys) {
+ if (conf.containsKey(key)) {
+ Map<String, Object> config = (Map<String, Object>) conf.get(key);
+ configMap.put(key, config);
+ LOG.info("configKey = {}, config = {}", key, config);
+ }
+ }
+ }
+ }
+
+ @Override
+ public void populateCredentials(Map<String, String> credentials, Map conf) {
+ try {
+ if (configKeys != null) {
--- End diff --
configKeys is initialised to ArrayList. we need to use configKeys.isEmpty() check here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2081: [STORM-2482] Refactor the Storm auto credential plugins t...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:
https://github.com/apache/storm/pull/2081
still +1 after the above comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2081#discussion_r113515141
--- Diff: docs/SECURITY.md ---
@@ -423,16 +423,18 @@ nimbus.impersonation.acl:
### Automatic Credentials Push and Renewal
Individual topologies have the ability to push credentials (tickets and tokens) to workers so that they can access secure services. Exposing this to all of the users can be a pain for them.
-To hide this from them in the common case plugins can be used to populate the credentials, unpack them on the other side into a java Subject, and also allow Nimbus to renew the credentials if needed.
-These are controlled by the following configs. topology.auto-credentials is a list of java plugins, all of which must implement IAutoCredentials interface, that populate the credentials on gateway
-and unpack them on the worker side. On a kerberos secure cluster they should be set by default to point to org.apache.storm.security.auth.kerberos.AutoTGT.
-nimbus.credential.renewers.classes should also be set to this value so that nimbus can periodically renew the TGT on behalf of the user.
+To hide this from them in the common case plugins can be used to populate the credentials, unpack them on the other side into a java Subject, and also allow Nimbus to renew the credentials if needed. These are controlled by the following configs.
+
+`topology.auto-credentials` is a list of java plugins, all of which must implement the `IAutoCredentials` interface, that populate the credentials on gateway
--- End diff --
Is gateway here nimbus or namenode like entity? Should we rename it to nimbus?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
Posted by arunmahadevan <gi...@git.apache.org>.
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/storm/pull/2081#discussion_r113686981
--- Diff: docs/storm-hbase.md ---
@@ -56,22 +56,43 @@ The approach described above requires that all potential worker hosts have "stor
multiple topologies on a cluster , each with different hbase user, you will have to create multiple keytabs and distribute
it to all workers. Instead of doing that you could use the following approach:
-Your administrator can configure nimbus to automatically get delegation tokens on behalf of the topology submitter user.
-The nimbus need to start with following configurations:
+Your administrator can configure nimbus to automatically get delegation tokens on behalf of the topology submitter user. The nimbus should be started with following configurations:
+```
nimbus.autocredential.plugins.classes : ["org.apache.storm.hbase.security.AutoHBase"]
nimbus.credential.renewers.classes : ["org.apache.storm.hbase.security.AutoHBase"]
hbase.keytab.file: "/path/to/keytab/on/nimbus" (This is the keytab of hbase super user that can impersonate other users.)
hbase.kerberos.principal: "superuser@EXAMPLE.com"
-nimbus.credential.renewers.freq.secs : 518400 (6 days, hbase tokens by default expire every 7 days and can not be renewed,
-if you have custom settings for hbase.auth.token.max.lifetime in hbase-site.xml than you should ensure this value is
-atleast 1 hour less then that.)
+nimbus.credential.renewers.freq.secs : 518400 (6 days, hbase tokens by default expire every 7 days and can not be renewed, if you have custom settings for hbase.auth.token.max.lifetime in hbase-site.xml than you should ensure this value is atleast 1 hour less then that.)
+```
Your topology configuration should have:
-topology.auto-credentials :["org.apache.storm.hbase.security.AutoHBase"]
+
+```
+topology.auto-credentials :["org.apache.storm.hbase.security.AutoHBase"]
+```
If nimbus did not have the above configuration you need to add it and then restart it. Ensure the hbase configuration
-files(core-site.xml,hdfs-site.xml and hbase-site.xml) and the storm-hbase jar with all the dependencies is present in nimbus's classpath.
+files(core-site.xml, hdfs-site.xml and hbase-site.xml) and the storm-hbase jar with all the dependencies is present in nimbus's classpath.
+
+As an alternative to adding the configuration files (core-site.xml, hdfs-site.xml and hbase-site.xml) to the classpath, you could specify the configurations as a part of the topology configuration. E.g. in you custom storm.yaml (or -c option while submitting the topology),
+
+```
+hbaseCredentialsConfigKeys : ["cluster1", "cluster2"] (the hbase clusters you want to fetch the tokens from)
+cluster1: [{"config1": "value1", "config2": "value2", ... }] (A map of config key-values specific to cluster1)
+cluster2: [{"config1": "value1", "hbase.keytab.file": "/path/to/keytab/for/cluster2/on/nimubs", "hbase.kerberos.principal": "cluster2user@EXAMPLE.com"}] (here along with other configs, we have custom keytab and principal for "cluster2" which will override the keytab/principal specified at topology level)
+```
+
+Instead of specifying key values you may also directly specify the resource files for e.g.,
+
+```
+cluster1: [{"resources": ["/path/to/core-site1.xml", "/path/to/hbase-site1.xml"]}]
+cluster2: [{"resources": ["/path/to/core-site2.xml", "/path/to/hbase-site2.xml"]}]
+```
+
+Storm will download the tokens separately for each of the clusters and populate it into the subject and also renew the tokens periodically.
+This way it would be possible to run multiple bolts connecting to separate HBase cluster within the same topology.
+
--- End diff --
I think it should be hbase.kerberos.principal. Will update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
Posted by omkreddy <gi...@git.apache.org>.
Github user omkreddy commented on a diff in the pull request:
https://github.com/apache/storm/pull/2081#discussion_r113670746
--- Diff: external/storm-autocreds/src/main/java/org/apache/storm/common/AbstractAutoCreds.java ---
@@ -0,0 +1,250 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.common;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.math3.util.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.security.token.TokenIdentifier;
+import org.apache.storm.security.INimbusCredentialPlugin;
+import org.apache.storm.security.auth.IAutoCredentials;
+import org.apache.storm.security.auth.ICredentialsRenewer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import javax.xml.bind.DatatypeConverter;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * The base class that for auto credential plugins that abstracts out some of the common functionality.
+ */
+public abstract class AbstractAutoCreds implements IAutoCredentials, ICredentialsRenewer, INimbusCredentialPlugin {
+ private static final Logger LOG = LoggerFactory.getLogger(AbstractAutoCreds.class);
+ public static final String CONFIG_KEY_RESOURCES = "resources";
+
+ private List<String> configKeys = new ArrayList<>();
+ private Map<String, Map<String, Object>> configMap = new HashMap<>();
+
+ @Override
+ public void prepare(Map conf) {
+ doPrepare(conf);
+ String configKeyString = getConfigKeyString();
+ if (conf.containsKey(configKeyString)) {
+ configKeys.addAll((List<String>) conf.get(configKeyString));
+ for (String key : configKeys) {
+ if (conf.containsKey(key)) {
+ Map<String, Object> config = (Map<String, Object>) conf.get(key);
+ configMap.put(key, config);
+ LOG.info("configKey = {}, config = {}", key, config);
+ }
+ }
+ }
+ }
+
+ @Override
+ public void populateCredentials(Map<String, String> credentials, Map conf) {
+ try {
+ if (configKeys != null) {
+ Map<String, Object> updatedConf = updateConfigs(conf);
+ for (String configKey : configKeys) {
+ credentials.put(getCredentialKey(configKey),
+ DatatypeConverter.printBase64Binary(getHadoopCredentials(updatedConf, configKey)));
+ }
+ } else {
+ credentials.put(getCredentialKey(StringUtils.EMPTY),
+ DatatypeConverter.printBase64Binary(getHadoopCredentials(conf)));
+ }
+ LOG.info("Tokens added to credentials map.");
+ } catch (Exception e) {
+ LOG.error("Could not populate credentials.", e);
+ }
+ }
+
+ private Map<String, Object> updateConfigs(Map topologyConf) {
+ Map<String, Object> res = new HashMap<>(topologyConf);
+ if (configKeys != null) {
--- End diff --
same as above
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
Posted by priyank5485 <gi...@git.apache.org>.
Github user priyank5485 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2081#discussion_r113541511
--- Diff: external/storm-autocreds/src/main/java/org/apache/storm/common/AbstractAutoCreds.java ---
@@ -0,0 +1,250 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.common;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.math3.util.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.security.token.TokenIdentifier;
+import org.apache.storm.security.INimbusCredentialPlugin;
+import org.apache.storm.security.auth.IAutoCredentials;
+import org.apache.storm.security.auth.ICredentialsRenewer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import javax.xml.bind.DatatypeConverter;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * The base class that for auto credential plugins that abstracts out some of the common functionality.
+ */
+public abstract class AbstractAutoCreds implements IAutoCredentials, ICredentialsRenewer, INimbusCredentialPlugin {
+ private static final Logger LOG = LoggerFactory.getLogger(AbstractAutoCreds.class);
+ public static final String CONFIG_KEY_RESOURCES = "resources";
+
+ private List<String> configKeys = new ArrayList<>();
+ private Map<String, Map<String, Object>> configMap = new HashMap<>();
+
+ @Override
+ public void prepare(Map conf) {
+ doPrepare(conf);
+ String configKeyString = getConfigKeyString();
+ if (conf.containsKey(configKeyString)) {
+ configKeys.addAll((List<String>) conf.get(configKeyString));
+ for (String key : configKeys) {
+ if (conf.containsKey(key)) {
+ Map<String, Object> config = (Map<String, Object>) conf.get(key);
+ configMap.put(key, config);
+ LOG.info("configKey = {}, config = {}", key, config);
+ }
+ }
+ }
+ }
+
+ @Override
+ public void populateCredentials(Map<String, String> credentials, Map conf) {
+ try {
+ if (configKeys != null) {
+ Map<String, Object> updatedConf = updateConfigs(conf);
+ for (String configKey : configKeys) {
+ credentials.put(getCredentialKey(configKey),
+ DatatypeConverter.printBase64Binary(getHadoopCredentials(updatedConf, configKey)));
+ }
+ } else {
+ credentials.put(getCredentialKey(StringUtils.EMPTY),
+ DatatypeConverter.printBase64Binary(getHadoopCredentials(conf)));
+ }
+ LOG.info("Tokens added to credentials map.");
+ } catch (Exception e) {
+ LOG.error("Could not populate credentials.", e);
+ }
+ }
+
+ private Map<String, Object> updateConfigs(Map topologyConf) {
+ Map<String, Object> res = new HashMap<>(topologyConf);
+ if (configKeys != null) {
+ for (String configKey : configKeys) {
+ if (!res.containsKey(configKey) && configMap.containsKey(configKey)) {
+ res.put(configKey, configMap.get(configKey));
+ }
+ }
+ }
+ return res;
+ }
+
+ @Override
+ public void renew(Map<String, String> credentials, Map topologyConf) {
+ doRenew(credentials, updateConfigs(topologyConf));
+ }
+
+ @Override
+ public void populateCredentials(Map<String, String> credentials) {
+ credentials.put(getCredentialKey(StringUtils.EMPTY),
+ DatatypeConverter.printBase64Binary("dummy place holder".getBytes()));
+ }
+
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void populateSubject(Subject subject, Map<String, String> credentials) {
+ addCredentialToSubject(subject, credentials);
+ addTokensToUGI(subject);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void updateSubject(Subject subject, Map<String, String> credentials) {
+ addCredentialToSubject(subject, credentials);
+ addTokensToUGI(subject);
+ }
+
+ protected Set<Pair<String, Credentials>> getCredentials(Map<String, String> credentials) {
+ Set<Pair<String, Credentials>> res = new HashSet<>();
+ if (configKeys != null) {
+ for (String configKey : configKeys) {
+ Credentials cred = doGetCredentials(credentials, configKey);
+ if (cred != null) {
+ res.add(new Pair(configKey, cred));
+ }
+ }
+ } else {
+ Credentials cred = doGetCredentials(credentials, StringUtils.EMPTY);
+ if (cred != null) {
+ res.add(new Pair(StringUtils.EMPTY, cred));
+ }
+ }
+ return res;
+ }
+
+ protected void fillHadoopConfiguration(Map topoConf, String configKey, Configuration configuration) {
+ Map<String, Object> config = (Map<String, Object>) topoConf.get(configKey);
+ LOG.info("TopoConf {}, got config {}, for configKey {}", topoConf, config, configKey);
+ if (config != null) {
+ List<String> resourcesToLoad = new ArrayList<>();
+ for (Map.Entry<String, Object> entry : config.entrySet()) {
+ if (entry.getKey().equals(CONFIG_KEY_RESOURCES)) {
+ resourcesToLoad.addAll((List<String>) entry.getValue());
+ } else {
+ configuration.set(entry.getKey(), String.valueOf(entry.getValue()));
+ }
+ }
+ LOG.info("Resources to load {}", resourcesToLoad);
+ // add configs from resources like hdfs-site.xml
+ for (String pathStr : resourcesToLoad) {
+ configuration.addResource(new Path(Paths.get(pathStr).toUri()));
+ }
+ }
+ LOG.info("Initializing UGI with config {}", configuration);
+ UserGroupInformation.setConfiguration(configuration);
--- End diff --
Never mind. It seems that this will always be executed in sequence for hdfs and hbase. So it will be overwriting the configuration and then login and get the tokens.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2081: [STORM-2482] Refactor the Storm auto credential pl...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/2081
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---