You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2020/03/12 05:16:56 UTC
[GitHub] [bigtop] iwasakims opened a new pull request #615: BIGTOP-3300. Add
puppet manifests for hadoop-kms.
iwasakims opened a new pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] iwasakims commented on issue #615: BIGTOP-3300. Add puppet
manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on issue #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#issuecomment-598747881
TODOs in follow-up JIRAs are
* HA deployment with multiple KMS instances
* https enabled configruation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] iwasakims commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392676464
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/kms-site.xml
##########
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+
+ <!-- KMS Backend KeyProvider -->
+
+ <property>
+ <name>hadoop.kms.key.provider.uri</name>
+ <value>jceks://file@/${user.home}/kms.keystore</value>
+ <description>
+ URI of the backing KeyProvider for the KMS.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
+ <value>kms.keystore.password</value>
+ <description>
+ If using the JavaKeyStoreProvider, the file name for the keystore password.
+ </description>
+ </property>
+
+ <!-- KMS Cache -->
+
+ <property>
+ <name>hadoop.kms.cache.enable</name>
+ <value>true</value>
+ <description>
+ Whether the KMS will act as a cache for the backing KeyProvider.
+ When the cache is enabled, operations like getKeyVersion, getMetadata,
+ and getCurrentKey will sometimes return cached data without consulting
+ the backing KeyProvider. Cached values are flushed when keys are deleted
+ or modified.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.cache.timeout.ms</name>
+ <value>600000</value>
+ <description>
+ Expiry time for the KMS key version and key metadata cache, in
+ milliseconds. This affects getKeyVersion and getMetadata.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.current.key.cache.timeout.ms</name>
+ <value>30000</value>
+ <description>
+ Expiry time for the KMS current key cache, in milliseconds. This
+ affects getCurrentKey operations.
+ </description>
+ </property>
+
+ <!-- KMS Audit -->
+
+ <property>
+ <name>hadoop.kms.audit.aggregation.window.ms</name>
+ <value>10000</value>
+ <description>
+ Duplicate audit log events within the aggregation window (specified in
+ ms) are quashed to reduce log traffic. A single message for aggregated
+ events is printed at the end of the window, along with a count of the
+ number of aggregated events.
+ </description>
+ </property>
+
+ <!-- KMS Security -->
+
+ <property>
+ <name>hadoop.kms.authentication.type</name>
+ <value><%= @hadoop_security_authentication %></value>
+ </property>
+<% if @hadoop_security_authentication == "kerberos" -%>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.keytab</name>
+ <value>/etc/kms.keytab</value>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.principal</name>
+ <value>HTTP/<%= @fqdn %>@<%= @kerberos_realm %></value>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.name.rules</name>
+ <value>DEFAULT</value>
+ </property>
+<% end -%>
+
+<% if @hadoop_core_proxyusers -%>
+<% @hadoop_core_proxyusers.sort.each do |superuser, data| -%>
+ <property>
+ <name>hadoop.kms.proxyuser.<%= superuser %>.hosts</name>
+ <value><%= data['hosts'] %></value>
+ </property>
+ <property>
+ <name>hadoop.kms.proxyuser.<%= superuser %>.groups</name>
+ <value><%= data['groups'] %></value>
+ </property>
+<% end -%>
+<% end -%>
+
+ <!-- Authentication cookie signature source -->
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider</name>
+ <value>random</value>
+ <description>
+ Indicates how the secret to sign the authentication cookies will be
+ stored. Options are 'random' (default), 'string' and 'zookeeper'.
+ If using a setup with multiple KMS instances, 'zookeeper' should be used.
+ </description>
+ </property>
+
+ <!-- Configuration for 'zookeeper' authentication cookie signature source -->
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.path</name>
+ <value>/hadoop-kms/hadoop-auth-signature-secret</value>
+ <description>
+ The Zookeeper ZNode path where the KMS instances will store and retrieve
+ the secret from.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.connection.string</name>
+ <value>#HOSTNAME#:#PORT#,...</value>
+ <description>
+ The Zookeeper connection string, a list of hostnames and port comma
+ separated.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type</name>
+ <value>none</value>
+ <description>
+ The Zookeeper authentication type, 'none' (default) or 'sasl' (Kerberos).
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.keytab</name>
+ <value>/etc/hadoop/conf/kms.keytab</value>
+ <description>
+ The absolute path for the Kerberos keytab with the credentials to
+ connect to Zookeeper.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.principal</name>
+ <value>kms/#HOSTNAME#</value>
Review comment:
This part came from the original kms-site.xml bundled with Hadoop. The value is used only if the value of `hadoop.kms.authentication.signer.secret.provider` is changed to `zookeeper`. ZKSignerSecretProvider is a feature for HA setup by which multiple KMS instances share the same signer secret via ZooKeeper. Since KMS HA is not supported in this patch, it is left as is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392683393
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/kms-site.xml
##########
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+
+ <!-- KMS Backend KeyProvider -->
+
+ <property>
+ <name>hadoop.kms.key.provider.uri</name>
+ <value>jceks://file@/${user.home}/kms.keystore</value>
+ <description>
+ URI of the backing KeyProvider for the KMS.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
+ <value>kms.keystore.password</value>
+ <description>
+ If using the JavaKeyStoreProvider, the file name for the keystore password.
+ </description>
+ </property>
+
+ <!-- KMS Cache -->
+
+ <property>
+ <name>hadoop.kms.cache.enable</name>
+ <value>true</value>
+ <description>
+ Whether the KMS will act as a cache for the backing KeyProvider.
+ When the cache is enabled, operations like getKeyVersion, getMetadata,
+ and getCurrentKey will sometimes return cached data without consulting
+ the backing KeyProvider. Cached values are flushed when keys are deleted
+ or modified.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.cache.timeout.ms</name>
+ <value>600000</value>
+ <description>
+ Expiry time for the KMS key version and key metadata cache, in
+ milliseconds. This affects getKeyVersion and getMetadata.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.current.key.cache.timeout.ms</name>
+ <value>30000</value>
+ <description>
+ Expiry time for the KMS current key cache, in milliseconds. This
+ affects getCurrentKey operations.
+ </description>
+ </property>
+
+ <!-- KMS Audit -->
+
+ <property>
+ <name>hadoop.kms.audit.aggregation.window.ms</name>
+ <value>10000</value>
+ <description>
+ Duplicate audit log events within the aggregation window (specified in
+ ms) are quashed to reduce log traffic. A single message for aggregated
+ events is printed at the end of the window, along with a count of the
+ number of aggregated events.
+ </description>
+ </property>
+
+ <!-- KMS Security -->
+
+ <property>
+ <name>hadoop.kms.authentication.type</name>
+ <value><%= @hadoop_security_authentication %></value>
+ </property>
+<% if @hadoop_security_authentication == "kerberos" -%>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.keytab</name>
+ <value>/etc/kms.keytab</value>
Review comment:
Got It. Pretty comprehensive. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on issue #615: BIGTOP-3300. Add puppet
manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on issue #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#issuecomment-599078815
This is super awesome feature! I've left some comments. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] iwasakims commented on issue #615: BIGTOP-3300. Add puppet
manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on issue #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#issuecomment-598746575
also tested on kerberos enabled cluster on config.yaml::
```
docker:
memory_limit: "8g"
image: "bigtop/puppet:trunk-centos-7"
distro: centos
components: [kerberos, hdfs, yarn, kms]
enable_local_repo: true
smoke_test_components: [hdfs]
```
with the following configs in bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml::
```
# Kerberos
hadoop::hadoop_security_authentication: "kerberos"
kerberos::krb_site::domain: "bigtop.apache.org"
kerberos::krb_site::realm: "BIGTOP.APACHE.ORG"
kerberos::krb_site::kdc_server: "%{hiera('bigtop::hadoop_head_node')}"
kerberos::krb_site::kdc_port: "88"
kerberos::krb_site::admin_port: "749"
kerberos::krb_site::keytab_export_dir: "/var/lib/bigtop_keytabs"
hadoop::kerberos_realm: "%{hiera('kerberos::krb_site::realm')}"
```
test steps::
```
$ cd provisioner/docker
$ ./docker-hadoop.sh -c 3
$ ./docker-hadoop.sh --exec 3 /bin/bash
# kinit -kt /etc/hdfs.keytab hdfs/$(hostname --fqdn)
# hadoop key create key1
# hdfs dfs -mkdir /zone1
# hdfs crypto -createZone -keyName key1 -path /zone1
# hdfs dfs -put /etc/hosts /zone1/
# hdfs dfs -get /zone1/hosts /tmp/
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392664529
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/kms-site.xml
##########
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+
+ <!-- KMS Backend KeyProvider -->
+
+ <property>
+ <name>hadoop.kms.key.provider.uri</name>
+ <value>jceks://file@/${user.home}/kms.keystore</value>
+ <description>
+ URI of the backing KeyProvider for the KMS.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
+ <value>kms.keystore.password</value>
+ <description>
+ If using the JavaKeyStoreProvider, the file name for the keystore password.
+ </description>
+ </property>
+
+ <!-- KMS Cache -->
+
+ <property>
+ <name>hadoop.kms.cache.enable</name>
+ <value>true</value>
+ <description>
+ Whether the KMS will act as a cache for the backing KeyProvider.
+ When the cache is enabled, operations like getKeyVersion, getMetadata,
+ and getCurrentKey will sometimes return cached data without consulting
+ the backing KeyProvider. Cached values are flushed when keys are deleted
+ or modified.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.cache.timeout.ms</name>
+ <value>600000</value>
+ <description>
+ Expiry time for the KMS key version and key metadata cache, in
+ milliseconds. This affects getKeyVersion and getMetadata.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.current.key.cache.timeout.ms</name>
+ <value>30000</value>
+ <description>
+ Expiry time for the KMS current key cache, in milliseconds. This
+ affects getCurrentKey operations.
+ </description>
+ </property>
+
+ <!-- KMS Audit -->
+
+ <property>
+ <name>hadoop.kms.audit.aggregation.window.ms</name>
+ <value>10000</value>
+ <description>
+ Duplicate audit log events within the aggregation window (specified in
+ ms) are quashed to reduce log traffic. A single message for aggregated
+ events is printed at the end of the window, along with a count of the
+ number of aggregated events.
+ </description>
+ </property>
+
+ <!-- KMS Security -->
+
+ <property>
+ <name>hadoop.kms.authentication.type</name>
+ <value><%= @hadoop_security_authentication %></value>
+ </property>
+<% if @hadoop_security_authentication == "kerberos" -%>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.keytab</name>
+ <value>/etc/kms.keytab</value>
Review comment:
I'm not aware that there's a convention to put key tab under /etc, is it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] iwasakims commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392640690
##########
File path: bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml
##########
@@ -69,15 +69,15 @@
# Kerberos
#hadoop::hadoop_security_authentication: "kerberos"
-#kerberos::site::domain: "do.main"
-#kerberos::site::realm: "DO.MAIN"
-#kerberos::site::kdc_server: "localhost"
-#kerberos::site::kdc_port: "88"
-#kerberos::site::admin_port: "749"
-#kerberos::site::keytab_export_dir: "/var/lib/bigtop_keytabs"
+#kerberos::krb_site::domain: "bigtop.apache.org"
+#kerberos::krb_site::realm: "BIGTOP.APACHE.ORG"
+#kerberos::krb_site::kdc_server: "%{hiera('bigtop::hadoop_head_node')}"
+#kerberos::krb_site::kdc_port: "88"
+#kerberos::krb_site::admin_port: "749"
+#kerberos::krb_site::keytab_export_dir: "/var/lib/bigtop_keytabs"
Review comment:
The Hiera variables are not injected unless the name space of properties match the class name of Puppet classes. The relevant class name was changed from `site` to `krb_site` in 3386a9d2d6898be01852f1d340d8948161eacdcd. The commented out configs (as example) seem to be left unchanged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392664600
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/kms-site.xml
##########
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+
+ <!-- KMS Backend KeyProvider -->
+
+ <property>
+ <name>hadoop.kms.key.provider.uri</name>
+ <value>jceks://file@/${user.home}/kms.keystore</value>
+ <description>
+ URI of the backing KeyProvider for the KMS.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
+ <value>kms.keystore.password</value>
+ <description>
+ If using the JavaKeyStoreProvider, the file name for the keystore password.
+ </description>
+ </property>
+
+ <!-- KMS Cache -->
+
+ <property>
+ <name>hadoop.kms.cache.enable</name>
+ <value>true</value>
+ <description>
+ Whether the KMS will act as a cache for the backing KeyProvider.
+ When the cache is enabled, operations like getKeyVersion, getMetadata,
+ and getCurrentKey will sometimes return cached data without consulting
+ the backing KeyProvider. Cached values are flushed when keys are deleted
+ or modified.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.cache.timeout.ms</name>
+ <value>600000</value>
+ <description>
+ Expiry time for the KMS key version and key metadata cache, in
+ milliseconds. This affects getKeyVersion and getMetadata.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.current.key.cache.timeout.ms</name>
+ <value>30000</value>
+ <description>
+ Expiry time for the KMS current key cache, in milliseconds. This
+ affects getCurrentKey operations.
+ </description>
+ </property>
+
+ <!-- KMS Audit -->
+
+ <property>
+ <name>hadoop.kms.audit.aggregation.window.ms</name>
+ <value>10000</value>
+ <description>
+ Duplicate audit log events within the aggregation window (specified in
+ ms) are quashed to reduce log traffic. A single message for aggregated
+ events is printed at the end of the window, along with a count of the
+ number of aggregated events.
+ </description>
+ </property>
+
+ <!-- KMS Security -->
+
+ <property>
+ <name>hadoop.kms.authentication.type</name>
+ <value><%= @hadoop_security_authentication %></value>
+ </property>
+<% if @hadoop_security_authentication == "kerberos" -%>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.keytab</name>
+ <value>/etc/kms.keytab</value>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.principal</name>
+ <value>HTTP/<%= @fqdn %>@<%= @kerberos_realm %></value>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.name.rules</name>
+ <value>DEFAULT</value>
+ </property>
+<% end -%>
+
+<% if @hadoop_core_proxyusers -%>
+<% @hadoop_core_proxyusers.sort.each do |superuser, data| -%>
+ <property>
+ <name>hadoop.kms.proxyuser.<%= superuser %>.hosts</name>
+ <value><%= data['hosts'] %></value>
+ </property>
+ <property>
+ <name>hadoop.kms.proxyuser.<%= superuser %>.groups</name>
+ <value><%= data['groups'] %></value>
+ </property>
+<% end -%>
+<% end -%>
+
+ <!-- Authentication cookie signature source -->
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider</name>
+ <value>random</value>
+ <description>
+ Indicates how the secret to sign the authentication cookies will be
+ stored. Options are 'random' (default), 'string' and 'zookeeper'.
+ If using a setup with multiple KMS instances, 'zookeeper' should be used.
+ </description>
+ </property>
+
+ <!-- Configuration for 'zookeeper' authentication cookie signature source -->
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.path</name>
+ <value>/hadoop-kms/hadoop-auth-signature-secret</value>
+ <description>
+ The Zookeeper ZNode path where the KMS instances will store and retrieve
+ the secret from.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.connection.string</name>
+ <value>#HOSTNAME#:#PORT#,...</value>
+ <description>
+ The Zookeeper connection string, a list of hostnames and port comma
+ separated.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type</name>
+ <value>none</value>
+ <description>
+ The Zookeeper authentication type, 'none' (default) or 'sasl' (Kerberos).
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.keytab</name>
+ <value>/etc/hadoop/conf/kms.keytab</value>
+ <description>
+ The absolute path for the Kerberos keytab with the credentials to
+ connect to Zookeeper.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.principal</name>
+ <value>kms/#HOSTNAME#</value>
Review comment:
Is this intentional for replacement?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392596024
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml
##########
@@ -169,4 +169,11 @@
<value>com.quantcast.qfs.hadoop.QuantcastFileSystem</value>
</property>
<% end -%>
+
+<% if @kms_host %>
+ <property>
+ <name>hadoop.security.key.provider.path</name>
+ <value>kms://http@<%= @kms_host %>:<%= @kms_port %>/kms</value>
Review comment:
I'm zero knowledge about this, just wondering can this be https?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on issue #615: BIGTOP-3300. Add puppet
manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on issue #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#issuecomment-599197571
Looks nice. +1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] iwasakims commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392677712
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/kms-site.xml
##########
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+
+ <!-- KMS Backend KeyProvider -->
+
+ <property>
+ <name>hadoop.kms.key.provider.uri</name>
+ <value>jceks://file@/${user.home}/kms.keystore</value>
+ <description>
+ URI of the backing KeyProvider for the KMS.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
+ <value>kms.keystore.password</value>
+ <description>
+ If using the JavaKeyStoreProvider, the file name for the keystore password.
+ </description>
+ </property>
+
+ <!-- KMS Cache -->
+
+ <property>
+ <name>hadoop.kms.cache.enable</name>
+ <value>true</value>
+ <description>
+ Whether the KMS will act as a cache for the backing KeyProvider.
+ When the cache is enabled, operations like getKeyVersion, getMetadata,
+ and getCurrentKey will sometimes return cached data without consulting
+ the backing KeyProvider. Cached values are flushed when keys are deleted
+ or modified.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.cache.timeout.ms</name>
+ <value>600000</value>
+ <description>
+ Expiry time for the KMS key version and key metadata cache, in
+ milliseconds. This affects getKeyVersion and getMetadata.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.current.key.cache.timeout.ms</name>
+ <value>30000</value>
+ <description>
+ Expiry time for the KMS current key cache, in milliseconds. This
+ affects getCurrentKey operations.
+ </description>
+ </property>
+
+ <!-- KMS Audit -->
+
+ <property>
+ <name>hadoop.kms.audit.aggregation.window.ms</name>
+ <value>10000</value>
+ <description>
+ Duplicate audit log events within the aggregation window (specified in
+ ms) are quashed to reduce log traffic. A single message for aggregated
+ events is printed at the end of the window, along with a count of the
+ number of aggregated events.
+ </description>
+ </property>
+
+ <!-- KMS Security -->
+
+ <property>
+ <name>hadoop.kms.authentication.type</name>
+ <value><%= @hadoop_security_authentication %></value>
+ </property>
+<% if @hadoop_security_authentication == "kerberos" -%>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.keytab</name>
+ <value>/etc/kms.keytab</value>
Review comment:
I used the keytab [created by kerberos::host_keytab resource](https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/kerberos/manifests/init.pp#L189). Other modules using the host_keytab resource such as [HDFS](https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml#L166) and [YARN](https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/hadoop/templates/yarn-site.xml#L30) seem to follow this convention as I can see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] iwasakims commented on issue #615: BIGTOP-3300. Add puppet
manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on issue #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#issuecomment-599292798
Thanks, @evans-ye.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392647192
##########
File path: bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml
##########
@@ -69,15 +69,15 @@
# Kerberos
#hadoop::hadoop_security_authentication: "kerberos"
-#kerberos::site::domain: "do.main"
-#kerberos::site::realm: "DO.MAIN"
-#kerberos::site::kdc_server: "localhost"
-#kerberos::site::kdc_port: "88"
-#kerberos::site::admin_port: "749"
-#kerberos::site::keytab_export_dir: "/var/lib/bigtop_keytabs"
+#kerberos::krb_site::domain: "bigtop.apache.org"
+#kerberos::krb_site::realm: "BIGTOP.APACHE.ORG"
+#kerberos::krb_site::kdc_server: "%{hiera('bigtop::hadoop_head_node')}"
+#kerberos::krb_site::kdc_port: "88"
+#kerberos::krb_site::admin_port: "749"
+#kerberos::krb_site::keytab_export_dir: "/var/lib/bigtop_keytabs"
Review comment:
OK. That was me ;) Glad you catch this. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] iwasakims commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392641061
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml
##########
@@ -169,4 +169,11 @@
<value>com.quantcast.qfs.hadoop.QuantcastFileSystem</value>
</property>
<% end -%>
+
+<% if @kms_host %>
+ <property>
+ <name>hadoop.security.key.provider.path</name>
+ <value>kms://http@<%= @kms_host %>:<%= @kms_port %>/kms</value>
Review comment:
Yes. We need additional configurations in files such as ssl-client.xml, ssl-server.xml and server.xml (of Tomcat) for that. I would like to address that in another JIRA since all services of HDFS and YARN should be cared when we enable HTTPS on Web-UI and REST API.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392664600
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/kms-site.xml
##########
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+
+ <!-- KMS Backend KeyProvider -->
+
+ <property>
+ <name>hadoop.kms.key.provider.uri</name>
+ <value>jceks://file@/${user.home}/kms.keystore</value>
+ <description>
+ URI of the backing KeyProvider for the KMS.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
+ <value>kms.keystore.password</value>
+ <description>
+ If using the JavaKeyStoreProvider, the file name for the keystore password.
+ </description>
+ </property>
+
+ <!-- KMS Cache -->
+
+ <property>
+ <name>hadoop.kms.cache.enable</name>
+ <value>true</value>
+ <description>
+ Whether the KMS will act as a cache for the backing KeyProvider.
+ When the cache is enabled, operations like getKeyVersion, getMetadata,
+ and getCurrentKey will sometimes return cached data without consulting
+ the backing KeyProvider. Cached values are flushed when keys are deleted
+ or modified.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.cache.timeout.ms</name>
+ <value>600000</value>
+ <description>
+ Expiry time for the KMS key version and key metadata cache, in
+ milliseconds. This affects getKeyVersion and getMetadata.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.current.key.cache.timeout.ms</name>
+ <value>30000</value>
+ <description>
+ Expiry time for the KMS current key cache, in milliseconds. This
+ affects getCurrentKey operations.
+ </description>
+ </property>
+
+ <!-- KMS Audit -->
+
+ <property>
+ <name>hadoop.kms.audit.aggregation.window.ms</name>
+ <value>10000</value>
+ <description>
+ Duplicate audit log events within the aggregation window (specified in
+ ms) are quashed to reduce log traffic. A single message for aggregated
+ events is printed at the end of the window, along with a count of the
+ number of aggregated events.
+ </description>
+ </property>
+
+ <!-- KMS Security -->
+
+ <property>
+ <name>hadoop.kms.authentication.type</name>
+ <value><%= @hadoop_security_authentication %></value>
+ </property>
+<% if @hadoop_security_authentication == "kerberos" -%>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.keytab</name>
+ <value>/etc/kms.keytab</value>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.principal</name>
+ <value>HTTP/<%= @fqdn %>@<%= @kerberos_realm %></value>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.name.rules</name>
+ <value>DEFAULT</value>
+ </property>
+<% end -%>
+
+<% if @hadoop_core_proxyusers -%>
+<% @hadoop_core_proxyusers.sort.each do |superuser, data| -%>
+ <property>
+ <name>hadoop.kms.proxyuser.<%= superuser %>.hosts</name>
+ <value><%= data['hosts'] %></value>
+ </property>
+ <property>
+ <name>hadoop.kms.proxyuser.<%= superuser %>.groups</name>
+ <value><%= data['groups'] %></value>
+ </property>
+<% end -%>
+<% end -%>
+
+ <!-- Authentication cookie signature source -->
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider</name>
+ <value>random</value>
+ <description>
+ Indicates how the secret to sign the authentication cookies will be
+ stored. Options are 'random' (default), 'string' and 'zookeeper'.
+ If using a setup with multiple KMS instances, 'zookeeper' should be used.
+ </description>
+ </property>
+
+ <!-- Configuration for 'zookeeper' authentication cookie signature source -->
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.path</name>
+ <value>/hadoop-kms/hadoop-auth-signature-secret</value>
+ <description>
+ The Zookeeper ZNode path where the KMS instances will store and retrieve
+ the secret from.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.connection.string</name>
+ <value>#HOSTNAME#:#PORT#,...</value>
+ <description>
+ The Zookeeper connection string, a list of hostnames and port comma
+ separated.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type</name>
+ <value>none</value>
+ <description>
+ The Zookeeper authentication type, 'none' (default) or 'sasl' (Kerberos).
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.keytab</name>
+ <value>/etc/hadoop/conf/kms.keytab</value>
+ <description>
+ The absolute path for the Kerberos keytab with the credentials to
+ connect to Zookeeper.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.principal</name>
+ <value>kms/#HOSTNAME#</value>
Review comment:
Is this left for replacement?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392595890
##########
File path: bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml
##########
@@ -69,15 +69,15 @@
# Kerberos
#hadoop::hadoop_security_authentication: "kerberos"
-#kerberos::site::domain: "do.main"
-#kerberos::site::realm: "DO.MAIN"
-#kerberos::site::kdc_server: "localhost"
-#kerberos::site::kdc_port: "88"
-#kerberos::site::admin_port: "749"
-#kerberos::site::keytab_export_dir: "/var/lib/bigtop_keytabs"
+#kerberos::krb_site::domain: "bigtop.apache.org"
+#kerberos::krb_site::realm: "BIGTOP.APACHE.ORG"
+#kerberos::krb_site::kdc_server: "%{hiera('bigtop::hadoop_head_node')}"
+#kerberos::krb_site::kdc_port: "88"
+#kerberos::krb_site::admin_port: "749"
+#kerberos::krb_site::keytab_export_dir: "/var/lib/bigtop_keytabs"
Review comment:
May I know why the name is changed to from site to krb_site? I think this may break compatibility.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] iwasakims commented on issue #615: BIGTOP-3300. Add puppet
manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on issue #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#issuecomment-598745986
I manually tested that HDFS transparent encryption works on following config.yaml::
```
docker:
memory_limit: "8g"
image: "bigtop/puppet:trunk-centos-7"
distro: centos
components: [hdfs, yarn, kms]
enable_local_repo: true
smoke_test_components: [hdfs, yarn]
```
test steps::
```
$ cd provisioner/docker
$ ./docker-hadoop.sh -c 3
$ ./docker-hadoop.sh --exec 3 /bin/bash
# hdfs dfs -mkdir /user/root/zone1
# hadoop key create key1
key1 has been successfully created with options Options{cipher='AES/CTR/NoPadding', bitLength=128, description='null', attributes=null}.
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@1dde4cb2 has been updated.
# sudo -u hdfs hdfs crypto -createZone -keyName key1 -path /user/root/zone1
Added encryption zone /user/root/zone1
# hdfs dfs -put /etc/hosts /user/root/zone1/
# hdfs dfs -get /user/root/zone1/hosts /tmp/
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392664529
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/kms-site.xml
##########
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+
+ <!-- KMS Backend KeyProvider -->
+
+ <property>
+ <name>hadoop.kms.key.provider.uri</name>
+ <value>jceks://file@/${user.home}/kms.keystore</value>
+ <description>
+ URI of the backing KeyProvider for the KMS.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
+ <value>kms.keystore.password</value>
+ <description>
+ If using the JavaKeyStoreProvider, the file name for the keystore password.
+ </description>
+ </property>
+
+ <!-- KMS Cache -->
+
+ <property>
+ <name>hadoop.kms.cache.enable</name>
+ <value>true</value>
+ <description>
+ Whether the KMS will act as a cache for the backing KeyProvider.
+ When the cache is enabled, operations like getKeyVersion, getMetadata,
+ and getCurrentKey will sometimes return cached data without consulting
+ the backing KeyProvider. Cached values are flushed when keys are deleted
+ or modified.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.cache.timeout.ms</name>
+ <value>600000</value>
+ <description>
+ Expiry time for the KMS key version and key metadata cache, in
+ milliseconds. This affects getKeyVersion and getMetadata.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.current.key.cache.timeout.ms</name>
+ <value>30000</value>
+ <description>
+ Expiry time for the KMS current key cache, in milliseconds. This
+ affects getCurrentKey operations.
+ </description>
+ </property>
+
+ <!-- KMS Audit -->
+
+ <property>
+ <name>hadoop.kms.audit.aggregation.window.ms</name>
+ <value>10000</value>
+ <description>
+ Duplicate audit log events within the aggregation window (specified in
+ ms) are quashed to reduce log traffic. A single message for aggregated
+ events is printed at the end of the window, along with a count of the
+ number of aggregated events.
+ </description>
+ </property>
+
+ <!-- KMS Security -->
+
+ <property>
+ <name>hadoop.kms.authentication.type</name>
+ <value><%= @hadoop_security_authentication %></value>
+ </property>
+<% if @hadoop_security_authentication == "kerberos" -%>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.keytab</name>
+ <value>/etc/kms.keytab</value>
Review comment:
I'm not aware that there's a convention to put keytab under /etc, is it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] iwasakims commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392640306
##########
File path: bigtop-deploy/puppet/modules/kerberos/manifests/init.pp
##########
@@ -189,8 +193,9 @@
$keytab = "/etc/$title.keytab"
$internal_princs = $spnego ? {
- /(true|enabled)/ => [ 'HTTP' ],
- default => [ ],
+ true => [ 'HTTP' ],
+ 'enabled' => [ 'HTTP' ],
+ default => [ ],
Review comment:
Yes. The credential of HTTP/host@REALM did not written to keytab file since boolean `true` was not covered in the conditional of previous code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye merged pull request #615: BIGTOP-3300. Add puppet
manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye merged pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392647266
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml
##########
@@ -169,4 +169,11 @@
<value>com.quantcast.qfs.hadoop.QuantcastFileSystem</value>
</property>
<% end -%>
+
+<% if @kms_host %>
+ <property>
+ <name>hadoop.security.key.provider.path</name>
+ <value>kms://http@<%= @kms_host %>:<%= @kms_port %>/kms</value>
Review comment:
Sure. No problem and thanks for the explanation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392596100
##########
File path: bigtop-deploy/puppet/modules/kerberos/manifests/init.pp
##########
@@ -189,8 +193,9 @@
$keytab = "/etc/$title.keytab"
$internal_princs = $spnego ? {
- /(true|enabled)/ => [ 'HTTP' ],
- default => [ ],
+ true => [ 'HTTP' ],
+ 'enabled' => [ 'HTTP' ],
+ default => [ ],
Review comment:
The original code won't work hence you refactor here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] evans-ye commented on a change in pull request #615:
BIGTOP-3300. Add puppet manifests for hadoop-kms.
Posted by GitBox <gi...@apache.org>.
evans-ye commented on a change in pull request #615: BIGTOP-3300. Add puppet manifests for hadoop-kms.
URL: https://github.com/apache/bigtop/pull/615#discussion_r392683263
##########
File path: bigtop-deploy/puppet/modules/hadoop/templates/kms-site.xml
##########
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+
+ <!-- KMS Backend KeyProvider -->
+
+ <property>
+ <name>hadoop.kms.key.provider.uri</name>
+ <value>jceks://file@/${user.home}/kms.keystore</value>
+ <description>
+ URI of the backing KeyProvider for the KMS.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
+ <value>kms.keystore.password</value>
+ <description>
+ If using the JavaKeyStoreProvider, the file name for the keystore password.
+ </description>
+ </property>
+
+ <!-- KMS Cache -->
+
+ <property>
+ <name>hadoop.kms.cache.enable</name>
+ <value>true</value>
+ <description>
+ Whether the KMS will act as a cache for the backing KeyProvider.
+ When the cache is enabled, operations like getKeyVersion, getMetadata,
+ and getCurrentKey will sometimes return cached data without consulting
+ the backing KeyProvider. Cached values are flushed when keys are deleted
+ or modified.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.cache.timeout.ms</name>
+ <value>600000</value>
+ <description>
+ Expiry time for the KMS key version and key metadata cache, in
+ milliseconds. This affects getKeyVersion and getMetadata.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.current.key.cache.timeout.ms</name>
+ <value>30000</value>
+ <description>
+ Expiry time for the KMS current key cache, in milliseconds. This
+ affects getCurrentKey operations.
+ </description>
+ </property>
+
+ <!-- KMS Audit -->
+
+ <property>
+ <name>hadoop.kms.audit.aggregation.window.ms</name>
+ <value>10000</value>
+ <description>
+ Duplicate audit log events within the aggregation window (specified in
+ ms) are quashed to reduce log traffic. A single message for aggregated
+ events is printed at the end of the window, along with a count of the
+ number of aggregated events.
+ </description>
+ </property>
+
+ <!-- KMS Security -->
+
+ <property>
+ <name>hadoop.kms.authentication.type</name>
+ <value><%= @hadoop_security_authentication %></value>
+ </property>
+<% if @hadoop_security_authentication == "kerberos" -%>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.keytab</name>
+ <value>/etc/kms.keytab</value>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.principal</name>
+ <value>HTTP/<%= @fqdn %>@<%= @kerberos_realm %></value>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.kerberos.name.rules</name>
+ <value>DEFAULT</value>
+ </property>
+<% end -%>
+
+<% if @hadoop_core_proxyusers -%>
+<% @hadoop_core_proxyusers.sort.each do |superuser, data| -%>
+ <property>
+ <name>hadoop.kms.proxyuser.<%= superuser %>.hosts</name>
+ <value><%= data['hosts'] %></value>
+ </property>
+ <property>
+ <name>hadoop.kms.proxyuser.<%= superuser %>.groups</name>
+ <value><%= data['groups'] %></value>
+ </property>
+<% end -%>
+<% end -%>
+
+ <!-- Authentication cookie signature source -->
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider</name>
+ <value>random</value>
+ <description>
+ Indicates how the secret to sign the authentication cookies will be
+ stored. Options are 'random' (default), 'string' and 'zookeeper'.
+ If using a setup with multiple KMS instances, 'zookeeper' should be used.
+ </description>
+ </property>
+
+ <!-- Configuration for 'zookeeper' authentication cookie signature source -->
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.path</name>
+ <value>/hadoop-kms/hadoop-auth-signature-secret</value>
+ <description>
+ The Zookeeper ZNode path where the KMS instances will store and retrieve
+ the secret from.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.connection.string</name>
+ <value>#HOSTNAME#:#PORT#,...</value>
+ <description>
+ The Zookeeper connection string, a list of hostnames and port comma
+ separated.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type</name>
+ <value>none</value>
+ <description>
+ The Zookeeper authentication type, 'none' (default) or 'sasl' (Kerberos).
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.keytab</name>
+ <value>/etc/hadoop/conf/kms.keytab</value>
+ <description>
+ The absolute path for the Kerberos keytab with the credentials to
+ connect to Zookeeper.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.principal</name>
+ <value>kms/#HOSTNAME#</value>
Review comment:
Thanks for the detailed explanation!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services