You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spot.apache.org by na...@apache.org on 2018/03/19 19:28:31 UTC
[22/42] incubator-spot git commit: [SPOT-213][SPOT-77] updated
requirements and documentation to support Kerberos for ingest
[SPOT-213][SPOT-77] updated requirements and documentation to support Kerberos for ingest
Project: http://git-wip-us.apache.org/repos/asf/incubator-spot/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spot/commit/13e35fc1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spot/tree/13e35fc1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spot/diff/13e35fc1
Branch: refs/heads/SPOT-181_ODM
Commit: 13e35fc1fa0a48b6df882341ab3c8e1e98324203
Parents: 49f4934
Author: natedogs911 <na...@gmail.com>
Authored: Thu Jan 18 15:24:23 2018 -0800
Committer: natedogs911 <na...@gmail.com>
Committed: Thu Jan 18 15:24:23 2018 -0800
----------------------------------------------------------------------
spot-ingest/KERBEROS.md | 50 ++++++++++++++++++++++++++++++
spot-ingest/README.md | 6 ++++
spot-ingest/kerberos-requirements.txt | 4 +++
spot-ingest/requirements.txt | 5 ++-
4 files changed, 64 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/13e35fc1/spot-ingest/KERBEROS.md
----------------------------------------------------------------------
diff --git a/spot-ingest/KERBEROS.md b/spot-ingest/KERBEROS.md
new file mode 100644
index 0000000..2c4c034
--- /dev/null
+++ b/spot-ingest/KERBEROS.md
@@ -0,0 +1,50 @@
+## Kerberos support installation
+
+run the following in addition to the typical installation instructions
+
+### Spot-Ingest
+
+`pip install -r ./spot-ingest/kerberos-requirements.txt`
+
+### Spot-OA
+
+`pip install -r ./spot-ingest/kerberos-requirements.txt`
+
+
+## spot.conf
+
+KERBEROS = set `KERBEROS='true'` in /etc/spot.conf to enable kerberos
+KEYTAB = should be generated using `ktutil` or another approved method
+SASL_MECH = should be set to `sasl_plaintext` unless using ssl
+KAFKA_SERVICE = if not set defaults will be used
+
+SSL = enable ssl by setting to true
+SSL_VERIFY = by setting to `false` disables host checking **important** only recommended in non production environments
+CA_LOCATION = location of certificate authority file
+CERT = host certificate
+KEY = key required for host certificate
+
+sample below:
+
+```
+#kerberos config
+KERBEROS='true'
+KINIT=/usr/bin/kinit
+PRINCIPAL='spot'
+KEYTAB='/opt/security/spot.keytab'
+SASL_MECH='GSSAPI'
+SECURITY_PROTO='sasl_plaintext'
+KAFKA_SERVICE_NAME=''
+
+#ssl config
+SSL='false'
+SSL_VERIFY='true'
+CA_LOCATION=''
+CERT=''
+KEY=''
+
+```
+
+Please see [LIBRDKAFKA Configurations](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)
+for reference to additional settings that can be set by modifying `spot-ingest/common/kafka_client.py`
+
http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/13e35fc1/spot-ingest/README.md
----------------------------------------------------------------------
diff --git a/spot-ingest/README.md b/spot-ingest/README.md
index acfb382..ce4f4cc 100644
--- a/spot-ingest/README.md
+++ b/spot-ingest/README.md
@@ -20,6 +20,12 @@ Ingest data is captured or transferred into the Hadoop cluster, where they are t
### Install
1. Install Python dependencies `pip install -r requirements.txt`
+Optional:
+2. the sasl python package requires the following:
+ * Centos: `yum install cyrus-sasl-devel`
+ * Debian/Ubuntu: `apt-get install libsasl2-dev`
+3. install Python dependencies for Kerberos `pip install -r kerberos-requirements.txt`
+
### Configure Kafka
**Adding Kafka Service:**
http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/13e35fc1/spot-ingest/kerberos-requirements.txt
----------------------------------------------------------------------
diff --git a/spot-ingest/kerberos-requirements.txt b/spot-ingest/kerberos-requirements.txt
new file mode 100644
index 0000000..ae5ea26
--- /dev/null
+++ b/spot-ingest/kerberos-requirements.txt
@@ -0,0 +1,4 @@
+thrift_sasl==0.2.1
+sasl
+hdfs[kerberos]
+requests-kerberos
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/13e35fc1/spot-ingest/requirements.txt
----------------------------------------------------------------------
diff --git a/spot-ingest/requirements.txt b/spot-ingest/requirements.txt
index 7d04054..71661bc 100644
--- a/spot-ingest/requirements.txt
+++ b/spot-ingest/requirements.txt
@@ -1,2 +1,5 @@
watchdog
-kafka-python
+confluent-kafka
+impyla
+hdfs
+six >= 1.5