You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Dnyaneshwar Pawar <dn...@persistent.com> on 2018/08/30 09:53:21 UTC
Secure NiFi cluster on kubernetes.
Hi,
We have requirement of deploying NiFi on cloud platforms and we are considering kubernetes as orchestrator. I have knowledge on configuring nifi cluster, however, I am not sure on how things would go on kubernetes. Further, we are using Apache DS as LDAP server for authentication and planning to use embedded zookeeper instance to make the zookeeper cluster.
Any help or pointer to documentation would be appreciated.
Thank You.
Regards,
Dnyaneshwar Pawar
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Secure NiFi cluster on kubernetes.
Posted by Peter Wilcsinszky <pe...@gmail.com>.
Not sure I understand. State management file (conf/state-management.xml) is
pretty static and is not necessary to be on a persistent volume.
However users.xml, authorizations.xml (not authorizers.xml!) and
flow.xml.gz are mutable configurations (almost databases) instead of simple
static config files and are generated and modified during NiFi's lifecycle.
The location of these files come from different places.
Location of flow.xml.gz is defined by nifi.properties:
nifi.flow.configuration.file=*/path/to/flow.xml.gz*
Location of users.xml and authorizations.xml comes from the
authorizers.xml, e.g.:
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">*/path/to/users.xml*</property>
...
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">*/path/to/authorizations.xml*
</property>
My recommendation is to handle your initial cluster nodes differently from
the nodes you want to scale. Create and configure your initial cluster so
that all these configuration databases are on persistent storage. (The
static config files in the conf/ folder don't have to be.) Also configure
your static authorizers.xml with all the initial users/nodes, inital admin
and the policies properly on these nodes only. Don't scale your statefulset
below it.
Now for the nodes above the initial cluster size I recommend putting all
the mutable configuration files on non-persistent volume. Also important to
configure the static authorizers.xml to not contain any users, nodes
initial admins and policies. This way the scaled nodes will always inherit
these from the initial cluster nodes, which can be considered the source of
truth.
In order for this to work I recommend using two separate authorizers.xml
files configured for the two scenarios. Then in the bootstrap script it can
be decided which one to use, like this:
if [[ ${HOSTNAME##*-} -lt {{ .Values.replicaCount }} ]]; then
# use properly filled authorizers.xml, where path to users.xml and
authorizations.xml should point to a *persistent* volume
# set nifi.flow.configuration.file to point to a *persistent* volume
else
# use "empty" authorizers.xml, where path to users.xml and
authorizations.xml point to a *non-persistent* volume
# set nifi.flow.configuration.file to point to a *non-persistent* volume
as well
fi
On Tue, Sep 25, 2018 at 2:01 AM Varun Tomar <Va...@zaplabs.com> wrote:
> I figured it out finally, it was the state management file.
>
>
>
> *From: *Varun Tomar <Va...@zaplabs.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Monday, September 24, 2018 at 4:07 PM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> Same issue even with persistent disks I don’t have users.xml any other
> suggestion.
>
>
>
> apiVersion: apps/v1beta1
>
> kind: StatefulSet
>
> metadata:
>
> name: nifi
>
> spec:
>
> serviceName: nifi
>
> replicas: 3
>
> podManagementPolicy: Parallel
>
> updateStrategy:
>
> type: RollingUpdate
>
> template:
>
> metadata:
>
> labels:
>
> app: nifi
>
> imagePullSecrets:
>
> - name: us-west-2-ecr-registry
>
> containers:
>
> - name: nifi
>
> image: xxxxxxxx.amazonaws.com/devops/nifi-1.7.0:v12-stateful
>
> command:
>
> - "/bin/sh"
>
> - "-c"
>
> - >
>
> set -x;
>
> sed -i -e
> "s|^nifi.authorizer.configuration.file=.*$|nifi.authorizer.configuration.file=/mnt/conf/authorizers.xml|"
> /opt/nifi/nifi-1.7.0/conf/nifi.properties;
>
> sed -i -e
> "s|^nifi.flow.configuration.file=.*$|nifi.flow.configuration.file=/mnt/conf/flow.xml.gz|"
> /opt/nifi/nifi-1.7.0/conf/nifi.properties;
>
> sed -i -e
> "s|^nifi.state.management.configuration.file=.*$|nifi.state.management.configuration.file=/mnt/conf/state-management.xml|"
> /opt/nifi/nifi-1.7.0/conf/nifi.properties
>
> sed -i -e
> "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|"
> /opt/nifi/nifi-1.7.0/conf/nifi.properties;
>
> sed -i -e
> "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test002|"
> /opt/nifi/nifi-1.7.0/conf/nifi.properties;
>
> sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|"
> /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
>
> sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|"
> /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
>
> /opt/nifi/nifi-1.7.0/bin/nifi.sh run
>
> securityContext:
>
> privileged: true
>
> ports:
>
> - name: prometheus-jmx
>
> containerPort: 8079
>
> - name: web
>
> containerPort: 8080
>
> - name: cluster
>
> containerPort: 8082
>
> volumeMounts:
>
> - name: repositories
>
> mountPath: /mnt
>
> volumeClaimTemplates:
>
> - metadata:
>
> name: repositories
>
> annotations:
>
> volume.beta.kubernetes.io/storage-class: thick-disk
>
> spec:
>
> accessModes: [ "ReadWriteOnce" ]
>
> resources:
>
> requests:
>
> storage: 25Gi
>
>
>
>
>
> *From: *Peter Wilcsinszky <pe...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Sunday, September 23, 2018 at 11:21 AM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> I beleive the problem is that you don't use PersistentVolumeClaims with
> your statefulset. Do you have a specific reason not using persistent
> volumes for your data and/or mutable config (authorizations.xml, users.xml,
> flow.xml.gz)?
>
>
>
> On Sun, Sep 23, 2018 at 7:25 PM Peter Wilcsinszky <
> peterwilcsinszky@gmail.com> wrote:
>
> Sorry I did not understand the problem first, but now I do and will check
> this.
>
>
>
> (Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print
> $2}'|head -1` you could simply use `hostname -f`, but that is not the
> problem. )
>
>
>
>
>
>
>
> On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar <Va...@zaplabs.com>
> wrote:
>
> This is my service.yaml (2 services one headless and one regular) and
> depoloyment.yaml, let me know if this helps.:
>
>
>
> *kind: *Service
> *apiVersion: *v1
>
> *metadata: name: *nifi-sync
> *namespace: *nifi
>
> *labels: app: *nifi
>
>
> *spec: ports: *- *name: *prometheus-jmx
> *port: *8079
> - *name: *web
> *port: *8080
> - *name: *cluster
> *port: *8082
> - *name: *misc
> *port: *9001
> *type: *NodePort
>
> *selector: app: *nifi
>
>
> ---
>
> *kind: *Service
> *apiVersion: *v1
>
> *metadata: name: *nifi
>
> *spec: clusterIP: *None
>
> *selector: app: *nifi
>
> *ports: *- *protocol: *TCP
> *port: *8081
> *targetPort: *8081
>
>
>
>
>
> ################
>
>
>
> *apiVersion: *apps/v1beta1
> *kind: *StatefulSet
>
> *metadata: name: *nifi
>
> *spec: serviceName: *nifi
> *replicas: *3
> *podManagementPolicy: *Parallel
>
> *updateStrategy: type: *RollingUpdate
>
>
>
> *template: metadata: labels: app: *nifi
>
>
>
>
> *spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: *-
>
> *labelSelector: matchExpressions: *- *key: *
> *"app" **operator: *In
>
> *values: *- nifi
> *topologyKey: **"**kubernetes.io/hostname* <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_hostname&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=jlf9tOXtU4c4NhVwcht24WDM_u7SLaQ6DXhr6OlLlaY&e=>
> *" *
> *imagePullSecrets: *- *name: *us-west-2-ecr-registry
>
> *containers: *- *name: *nifi
> *image: *XXXXXXXXX.amazonaws.com/devops/nifi-1.7.0:v11-stateful <https://urldefense.proofpoint.com/v2/url?u=http-3A__XXXXXXXXX.amazonaws.com_devops_nifi-2D1.7.0-3Av11-2Dstateful&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=eB6ndvXtCzcQEPUu0OnZUPDk9IcZPTdpqO79YF5K8gc&e=>
>
> *command: *-
> *"/bin/sh" *-
> *"-c" *- >
> wget http://xxxxxxxx/build/deploy/configure_statefulset.sh <https://urldefense.proofpoint.com/v2/url?u=http-3A__xxxxxxxx_build_deploy_configure-5Fstatefulset.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=kgjjTsPFUolM5uXzxaKCFVHGtiqZA_L9wQycuBKSfq0&e=> -O /tmp/configure_statefulset.sh;
> chmod +x /tmp/configure_statefulset.sh;
> /tmp/configure_statefulset.sh;
> apt-get -y install dnsutils;
> sed -i -e "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1 min|" $path/conf/nifi.properties;
> mkdir -p $path/state/zookeeper;
> touch $path/state/zookeeper/myid;
> echo `hostname |rev | cut -d'-' -f 1 | rev` > $path/state/zookeeper/myid;
> sed -i -e "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
> sed -i -e "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
> sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
> sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
> /opt/nifi/nifi-1.7.0/bin/nifi.sh run
>
> *securityContext: privileged: *true
> *imagePullPolicy: *Always
>
> *ports: *- *containerPort: *8080
> - *containerPort: *8081
>
>
>
>
>
> *From: *Peter Wilcsinszky <pe...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Sunday, September 23, 2018 at 6:33 AM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Varun,
>
>
>
> hard to tell without seeing your statefulset config. How do you add the
> new nodes? If you add them through the statefulset (kubectl scale
> statefulset <your statefulset's name> --replicas <desired replica count>)
>
> the nodes should have the names nifi-3, nifi-4 instead of the ones on your
> screenshot. But again, this is going to be hard to debug without seeing
> your config.
>
>
>
> Peter
>
>
>
> On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar <Va...@zaplabs.com>
> wrote:
>
> Hi Peter,
>
>
>
> I tried your suggestion of using statefulset in k8s. The problem is still
> there. The new nodes join the cluster but the old nodes still remains am I
> missing something. I am guessing each nodes get an Id which is the deciding
> factor in cluster config and not the node address, that’s the reason I am
> seeing 3/5.
>
>
>
>
>
> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "message": "Connection requested from existing node. Setting
> status to connecting."
>
> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "message": "Connection requested from existing node. Setting
> status to connecting."
>
> "address": "nifi-0.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>
> "status": "DISCONNECTED",
>
> "message": "Node disconnected from cluster due to Have not
> received a heartbeat from node in 44 seconds"
>
> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>
> "status": "DISCONNECTED",
>
> "message": "Node disconnected from cluster due to Have not
> received a heartbeat from node in 44 seconds"
>
>
>
> *Error! Filename not specified.*
>
> *From: *Peter Wilcsinszky <pe...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Friday, August 31, 2018 at 10:01 AM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
>
>
> On Fri, 31 Aug 2018, 16:51 Varun Tomar, <Va...@zaplabs.com> wrote:
>
> Hi Peter,
>
>
>
> We started using nifi as statefulset last year you but moved to deployment.
>
>
>
> -CICD tool Spinnaker does not support statefulsets.
>
> - We have also customized logback.xml as it was log within log issue which
> was not getting parsed properly in ELK
>
> - For ports and cluster IP I pass them as argument so even if the pod
> reboot we don't have any issues.
>
> Why do you need to pass an IP?
>
>
>
> - we also use external zookeeper.
>
>
>
> I dint find any benefit of running statefulset .
>
>
>
> The only issue as I said is if we restart any undeying node we extra node
> and old nodes does not get deleted.
>
> With a statefulset you wouldnt have issues with that and you would have
> stable persistent volumes as well.
>
>
>
>
>
>
>
> Regards,
>
> Varun
>
>
> ------------------------------
>
> *From:* Peter Wilcsinszky <pe...@gmail.com>
> *Sent:* Friday, August 31, 2018 2:50 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Dnyaneshwar,
>
>
>
> as Andy mentioned we are working on running NiFi in Kubernetes but I'm not
> sure when it will be available publicly. Some pointers that can help by
> then:
>
> - You should use a StatefulSet to manage NiFi pods
>
> - Probably Helm charts are the most efficient way to get started
>
> - I recommend using the official NiFi image and wrapping the original
> nifi.sh script from the Kubernetes pod spec similarly how we do it in the
> Docker image [1]. Caveats: setting dynamic properties like
> nifi.web.http.host from the wrapper script is a good idea, but for more
> static properties like nifi.web.http.port you may want to use the config
> files directly as configmaps and do templating using Helm. This is
> especially true for more complex configurations like the authorizers.xml or
> the login-identity-providers.xml.
>
> - Authorizations in NiFi can be configured for the initial cluster setup,
> but needs to be done manually when you add a new Node to the cluster above
> the initial cluster size. Also these extra nodes should have a vanilla
> authorizations.xml to avoid conflicts when joining to the existing ones.
> You can use the wrapper script to decide which configmap to use when
> starting the container. Once the pod has started you still have to add the
> node and authorize it manually using the UI. There is ongoing work to make
> this more dynamic: [3]
>
> - We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in
> server mode. The NiFi pods have an init container that uses tls-toolkit in
> client mode to request and receive certificates from the CA server. The
> communication is protected using a shared secret that is generated inside
> the cluster on the fly, also you can further protect access to the CA using
> NetworkPolicies.
>
> - You should avoid using the embedded Zookeeper, but you can use an
> already existing helm chart as a dependency to install it [4] (caveat: the
> image used by that chart is not recommended for production use)
>
>
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
>
> [2]
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
>
> [3] https://issues.apache.org/jira/browse/NIFI-5542
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
>
> [4] https://github.com/helm/charts/tree/master/incubator/zookeeper
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
>
>
>
> On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>
> wrote:
>
> Hi Dnyaneshwar,
>
>
>
> We have nifi running on k8s for around 8-10 months. We create nifi cluster
> as part of CICD and then there is a stage which does the template
> deployment. Haven’t faced any major issues. Just sometime if a node reboots
> the old cluster member in nifi does not gets cleaned up.
>
>
>
> Regards,
>
> Varun
>
>
>
> *From: *Andy LoPresto <al...@apache.org>
> *Reply-To: *<us...@nifi.apache.org>
> *Date: *Thursday, August 30, 2018 at 10:23 AM
> *To: *<us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Dnyaneshwar,
>
>
>
> I know other users are working on the same thing, so yes, NiFi +
> Kubernetes will allow you to stand up secure clusters. There is ongoing
> work targeted for upcoming releases to make this easier and more performant
> (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter
> Wilcsinszky has done a lot of great work here, and he may be able to share
> some resources he used/created.
>
>
>
> [1]
> https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> *alopresto.apache@gmail.com* <al...@gmail.com>
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <
> dnyaneshwar_pawar@persistent.com> wrote:
>
>
>
> Hi,
>
>
>
> We have requirement of deploying NiFi on cloud platforms and we are
> considering kubernetes as orchestrator. I have knowledge on configuring
> nifi cluster, however, I am not sure on how things would go on kubernetes.
> Further, we are using Apache DS as LDAP server for authentication and
> planning to use embedded zookeeper instance to make the zookeeper cluster.
>
> Any help or pointer to documentation would be appreciated.
>
>
>
> Thank You.
>
>
>
> Regards,
>
> Dnyaneshwar Pawar
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
>
> ------------------------------
>
> This email may be confidential. If you are not the intended recipient,
> please notify us immediately and delete this copy from your system.
>
> *Wire Fraud is Real*. Before wiring any money, call the intended
> recipient at a number you know is valid to confirm the instructions. Additionally,
> please note that the sender does not have authority to bind a party to a
> real estate contract via written or verbal communication.
>
>
Re: Secure NiFi cluster on kubernetes.
Posted by Varun Tomar <Va...@zaplabs.com>.
I figured it out finally, it was the state management file.
From: Varun Tomar <Va...@zaplabs.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, September 24, 2018 at 4:07 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
Same issue even with persistent disks I don’t have users.xml any other suggestion.
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nifi
spec:
serviceName: nifi
replicas: 3
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: nifi
imagePullSecrets:
- name: us-west-2-ecr-registry
containers:
- name: nifi
image: xxxxxxxx.amazonaws.com/devops/nifi-1.7.0:v12-stateful
command:
- "/bin/sh"
- "-c"
- >
set -x;
sed -i -e "s|^nifi.authorizer.configuration.file=.*$|nifi.authorizer.configuration.file=/mnt/conf/authorizers.xml|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^nifi.flow.configuration.file=.*$|nifi.flow.configuration.file=/mnt/conf/flow.xml.gz|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^nifi.state.management.configuration.file=.*$|nifi.state.management.configuration.file=/mnt/conf/state-management.xml|" /opt/nifi/nifi-1.7.0/conf/nifi.properties
sed -i -e "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test002|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
/opt/nifi/nifi-1.7.0/bin/nifi.sh run
securityContext:
privileged: true
ports:
- name: prometheus-jmx
containerPort: 8079
- name: web
containerPort: 8080
- name: cluster
containerPort: 8082
volumeMounts:
- name: repositories
mountPath: /mnt
volumeClaimTemplates:
- metadata:
name: repositories
annotations:
volume.beta.kubernetes.io/storage-class: thick-disk
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 25Gi
From: Peter Wilcsinszky <pe...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Sunday, September 23, 2018 at 11:21 AM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
I beleive the problem is that you don't use PersistentVolumeClaims with your statefulset. Do you have a specific reason not using persistent volumes for your data and/or mutable config (authorizations.xml, users.xml, flow.xml.gz)?
On Sun, Sep 23, 2018 at 7:25 PM Peter Wilcsinszky <pe...@gmail.com>> wrote:
Sorry I did not understand the problem first, but now I do and will check this.
(Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1` you could simply use `hostname -f`, but that is not the problem. )
On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar <Va...@zaplabs.com>> wrote:
This is my service.yaml (2 services one headless and one regular) and depoloyment.yaml, let me know if this helps.:
kind: Service
apiVersion: v1
metadata:
name: nifi-sync
namespace: nifi
labels:
app: nifi
spec:
ports:
- name: prometheus-jmx
port: 8079
- name: web
port: 8080
- name: cluster
port: 8082
- name: misc
port: 9001
type: NodePort
selector:
app: nifi
---
kind: Service
apiVersion: v1
metadata:
name: nifi
spec:
clusterIP: None
selector:
app: nifi
ports:
- protocol: TCP
port: 8081
targetPort: 8081
################
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nifi
spec:
serviceName: nifi
replicas: 3
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: nifi
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- nifi
topologyKey: "kubernetes.io/hostname<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_hostname&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=jlf9tOXtU4c4NhVwcht24WDM_u7SLaQ6DXhr6OlLlaY&e=>"
imagePullSecrets:
- name: us-west-2-ecr-registry
containers:
- name: nifi
image: XXXXXXXXX.amazonaws.com/devops/nifi-1.7.0:v11-stateful<https://urldefense.proofpoint.com/v2/url?u=http-3A__XXXXXXXXX.amazonaws.com_devops_nifi-2D1.7.0-3Av11-2Dstateful&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=eB6ndvXtCzcQEPUu0OnZUPDk9IcZPTdpqO79YF5K8gc&e=>
command:
- "/bin/sh"
- "-c"
- >
wget http://xxxxxxxx/build/deploy/configure_statefulset.sh<https://urldefense.proofpoint.com/v2/url?u=http-3A__xxxxxxxx_build_deploy_configure-5Fstatefulset.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=kgjjTsPFUolM5uXzxaKCFVHGtiqZA_L9wQycuBKSfq0&e=> -O /tmp/configure_statefulset.sh;
chmod +x /tmp/configure_statefulset.sh;
/tmp/configure_statefulset.sh;
apt-get -y install dnsutils;
sed -i -e "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1 min|" $path/conf/nifi.properties;
mkdir -p $path/state/zookeeper;
touch $path/state/zookeeper/myid;
echo `hostname |rev | cut -d'-' -f 1 | rev` > $path/state/zookeeper/myid;
sed -i -e "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
/opt/nifi/nifi-1.7.0/bin/nifi.sh run
securityContext:
privileged: true
imagePullPolicy: Always
ports:
- containerPort: 8080
- containerPort: 8081
From: Peter Wilcsinszky <pe...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Sunday, September 23, 2018 at 6:33 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Varun,
hard to tell without seeing your statefulset config. How do you add the new nodes? If you add them through the statefulset (kubectl scale statefulset <your statefulset's name> --replicas <desired replica count>)
the nodes should have the names nifi-3, nifi-4 instead of the ones on your screenshot. But again, this is going to be hard to debug without seeing your config.
Peter
On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar <Va...@zaplabs.com>> wrote:
Hi Peter,
I tried your suggestion of using statefulset in k8s. The problem is still there. The new nodes join the cluster but the old nodes still remains am I missing something. I am guessing each nodes get an Id which is the deciding factor in cluster config and not the node address, that’s the reason I am seeing 3/5.
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status to connecting."
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status to connecting."
"address": "nifi-0.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received a heartbeat from node in 44 seconds"
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received a heartbeat from node in 44 seconds"
Error! Filename not specified.
From: Peter Wilcsinszky <pe...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Friday, August 31, 2018 at 10:01 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
On Fri, 31 Aug 2018, 16:51 Varun Tomar, <Va...@zaplabs.com>> wrote:
Hi Peter,
We started using nifi as statefulset last year you but moved to deployment.
-CICD tool Spinnaker does not support statefulsets.
- We have also customized logback.xml as it was log within log issue which was not getting parsed properly in ELK
- For ports and cluster IP I pass them as argument so even if the pod reboot we don't have any issues.
Why do you need to pass an IP?
- we also use external zookeeper.
I dint find any benefit of running statefulset .
The only issue as I said is if we restart any undeying node we extra node and old nodes does not get deleted.
With a statefulset you wouldnt have issues with that and you would have stable persistent volumes as well.
Regards,
Varun
________________________________
From: Peter Wilcsinszky <pe...@gmail.com>>
Sent: Friday, August 31, 2018 2:50 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
as Andy mentioned we are working on running NiFi in Kubernetes but I'm not sure when it will be available publicly. Some pointers that can help by then:
- You should use a StatefulSet to manage NiFi pods
- Probably Helm charts are the most efficient way to get started
- I recommend using the official NiFi image and wrapping the original nifi.sh script from the Kubernetes pod spec similarly how we do it in the Docker image [1]. Caveats: setting dynamic properties like nifi.web.http.host from the wrapper script is a good idea, but for more static properties like nifi.web.http.port you may want to use the config files directly as configmaps and do templating using Helm. This is especially true for more complex configurations like the authorizers.xml or the login-identity-providers.xml.
- Authorizations in NiFi can be configured for the initial cluster setup, but needs to be done manually when you add a new Node to the cluster above the initial cluster size. Also these extra nodes should have a vanilla authorizations.xml to avoid conflicts when joining to the existing ones. You can use the wrapper script to decide which configmap to use when starting the container. Once the pod has started you still have to add the node and authorize it manually using the UI. There is ongoing work to make this more dynamic: [3]
- We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in server mode. The NiFi pods have an init container that uses tls-toolkit in client mode to request and receive certificates from the CA server. The communication is protected using a shared secret that is generated inside the cluster on the fly, also you can further protect access to the CA using NetworkPolicies.
- You should avoid using the embedded Zookeeper, but you can use an already existing helm chart as a dependency to install it [4] (caveat: the image used by that chart is not recommended for production use)
[1] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
[2] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
[3] https://issues.apache.org/jira/browse/NIFI-5542<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
[4] https://github.com/helm/charts/tree/master/incubator/zookeeper<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>> wrote:
Hi Dnyaneshwar,
We have nifi running on k8s for around 8-10 months. We create nifi cluster as part of CICD and then there is a stage which does the template deployment. Haven’t faced any major issues. Just sometime if a node reboots the old cluster member in nifi does not gets cleaned up.
Regards,
Varun
From: Andy LoPresto <al...@apache.org>>
Reply-To: <us...@nifi.apache.org>>
Date: Thursday, August 30, 2018 at 10:23 AM
To: <us...@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
I know other users are working on the same thing, so yes, NiFi + Kubernetes will allow you to stand up secure clusters. There is ongoing work targeted for upcoming releases to make this easier and more performant (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter Wilcsinszky has done a lot of great work here, and he may be able to share some resources he used/created.
[1] https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <dn...@persistent.com>> wrote:
Hi,
We have requirement of deploying NiFi on cloud platforms and we are considering kubernetes as orchestrator. I have knowledge on configuring nifi cluster, however, I am not sure on how things would go on kubernetes. Further, we are using Apache DS as LDAP server for authentication and planning to use embedded zookeeper instance to make the zookeeper cluster.
Any help or pointer to documentation would be appreciated.
Thank You.
Regards,
Dnyaneshwar Pawar
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
________________________________
This email may be confidential. If you are not the intended recipient, please notify us immediately and delete this copy from your system.
*Wire Fraud is Real*. Before wiring any money, call the intended recipient at a number you know is valid to confirm the instructions. Additionally, please note that the sender does not have authority to bind a party to a real estate contract via written or verbal communication.
Re: Secure NiFi cluster on kubernetes.
Posted by Varun Tomar <Va...@zaplabs.com>.
Same issue even with persistent disks I don’t have users.xml any other suggestion.
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nifi
spec:
serviceName: nifi
replicas: 3
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: nifi
imagePullSecrets:
- name: us-west-2-ecr-registry
containers:
- name: nifi
image: xxxxxxxx.amazonaws.com/devops/nifi-1.7.0:v12-stateful
command:
- "/bin/sh"
- "-c"
- >
set -x;
sed -i -e "s|^nifi.authorizer.configuration.file=.*$|nifi.authorizer.configuration.file=/mnt/conf/authorizers.xml|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^nifi.flow.configuration.file=.*$|nifi.flow.configuration.file=/mnt/conf/flow.xml.gz|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^nifi.state.management.configuration.file=.*$|nifi.state.management.configuration.file=/mnt/conf/state-management.xml|" /opt/nifi/nifi-1.7.0/conf/nifi.properties
sed -i -e "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test002|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
/opt/nifi/nifi-1.7.0/bin/nifi.sh run
securityContext:
privileged: true
ports:
- name: prometheus-jmx
containerPort: 8079
- name: web
containerPort: 8080
- name: cluster
containerPort: 8082
volumeMounts:
- name: repositories
mountPath: /mnt
volumeClaimTemplates:
- metadata:
name: repositories
annotations:
volume.beta.kubernetes.io/storage-class: thick-disk
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 25Gi
From: Peter Wilcsinszky <pe...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Sunday, September 23, 2018 at 11:21 AM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
I beleive the problem is that you don't use PersistentVolumeClaims with your statefulset. Do you have a specific reason not using persistent volumes for your data and/or mutable config (authorizations.xml, users.xml, flow.xml.gz)?
On Sun, Sep 23, 2018 at 7:25 PM Peter Wilcsinszky <pe...@gmail.com>> wrote:
Sorry I did not understand the problem first, but now I do and will check this.
(Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1` you could simply use `hostname -f`, but that is not the problem. )
On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar <Va...@zaplabs.com>> wrote:
This is my service.yaml (2 services one headless and one regular) and depoloyment.yaml, let me know if this helps.:
kind: Service
apiVersion: v1
metadata:
name: nifi-sync
namespace: nifi
labels:
app: nifi
spec:
ports:
- name: prometheus-jmx
port: 8079
- name: web
port: 8080
- name: cluster
port: 8082
- name: misc
port: 9001
type: NodePort
selector:
app: nifi
---
kind: Service
apiVersion: v1
metadata:
name: nifi
spec:
clusterIP: None
selector:
app: nifi
ports:
- protocol: TCP
port: 8081
targetPort: 8081
################
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nifi
spec:
serviceName: nifi
replicas: 3
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: nifi
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- nifi
topologyKey: "kubernetes.io/hostname<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_hostname&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=jlf9tOXtU4c4NhVwcht24WDM_u7SLaQ6DXhr6OlLlaY&e=>"
imagePullSecrets:
- name: us-west-2-ecr-registry
containers:
- name: nifi
image: XXXXXXXXX.amazonaws.com/devops/nifi-1.7.0:v11-stateful<https://urldefense.proofpoint.com/v2/url?u=http-3A__XXXXXXXXX.amazonaws.com_devops_nifi-2D1.7.0-3Av11-2Dstateful&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=eB6ndvXtCzcQEPUu0OnZUPDk9IcZPTdpqO79YF5K8gc&e=>
command:
- "/bin/sh"
- "-c"
- >
wget http://xxxxxxxx/build/deploy/configure_statefulset.sh<https://urldefense.proofpoint.com/v2/url?u=http-3A__xxxxxxxx_build_deploy_configure-5Fstatefulset.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=kgjjTsPFUolM5uXzxaKCFVHGtiqZA_L9wQycuBKSfq0&e=> -O /tmp/configure_statefulset.sh;
chmod +x /tmp/configure_statefulset.sh;
/tmp/configure_statefulset.sh;
apt-get -y install dnsutils;
sed -i -e "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1 min|" $path/conf/nifi.properties;
mkdir -p $path/state/zookeeper;
touch $path/state/zookeeper/myid;
echo `hostname |rev | cut -d'-' -f 1 | rev` > $path/state/zookeeper/myid;
sed -i -e "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
/opt/nifi/nifi-1.7.0/bin/nifi.sh run
securityContext:
privileged: true
imagePullPolicy: Always
ports:
- containerPort: 8080
- containerPort: 8081
From: Peter Wilcsinszky <pe...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Sunday, September 23, 2018 at 6:33 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Varun,
hard to tell without seeing your statefulset config. How do you add the new nodes? If you add them through the statefulset (kubectl scale statefulset <your statefulset's name> --replicas <desired replica count>)
the nodes should have the names nifi-3, nifi-4 instead of the ones on your screenshot. But again, this is going to be hard to debug without seeing your config.
Peter
On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar <Va...@zaplabs.com>> wrote:
Hi Peter,
I tried your suggestion of using statefulset in k8s. The problem is still there. The new nodes join the cluster but the old nodes still remains am I missing something. I am guessing each nodes get an Id which is the deciding factor in cluster config and not the node address, that’s the reason I am seeing 3/5.
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status to connecting."
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status to connecting."
"address": "nifi-0.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received a heartbeat from node in 44 seconds"
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received a heartbeat from node in 44 seconds"
Error! Filename not specified.
From: Peter Wilcsinszky <pe...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Friday, August 31, 2018 at 10:01 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
On Fri, 31 Aug 2018, 16:51 Varun Tomar, <Va...@zaplabs.com>> wrote:
Hi Peter,
We started using nifi as statefulset last year you but moved to deployment.
-CICD tool Spinnaker does not support statefulsets.
- We have also customized logback.xml as it was log within log issue which was not getting parsed properly in ELK
- For ports and cluster IP I pass them as argument so even if the pod reboot we don't have any issues.
Why do you need to pass an IP?
- we also use external zookeeper.
I dint find any benefit of running statefulset .
The only issue as I said is if we restart any undeying node we extra node and old nodes does not get deleted.
With a statefulset you wouldnt have issues with that and you would have stable persistent volumes as well.
Regards,
Varun
________________________________
From: Peter Wilcsinszky <pe...@gmail.com>>
Sent: Friday, August 31, 2018 2:50 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
as Andy mentioned we are working on running NiFi in Kubernetes but I'm not sure when it will be available publicly. Some pointers that can help by then:
- You should use a StatefulSet to manage NiFi pods
- Probably Helm charts are the most efficient way to get started
- I recommend using the official NiFi image and wrapping the original nifi.sh script from the Kubernetes pod spec similarly how we do it in the Docker image [1]. Caveats: setting dynamic properties like nifi.web.http.host from the wrapper script is a good idea, but for more static properties like nifi.web.http.port you may want to use the config files directly as configmaps and do templating using Helm. This is especially true for more complex configurations like the authorizers.xml or the login-identity-providers.xml.
- Authorizations in NiFi can be configured for the initial cluster setup, but needs to be done manually when you add a new Node to the cluster above the initial cluster size. Also these extra nodes should have a vanilla authorizations.xml to avoid conflicts when joining to the existing ones. You can use the wrapper script to decide which configmap to use when starting the container. Once the pod has started you still have to add the node and authorize it manually using the UI. There is ongoing work to make this more dynamic: [3]
- We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in server mode. The NiFi pods have an init container that uses tls-toolkit in client mode to request and receive certificates from the CA server. The communication is protected using a shared secret that is generated inside the cluster on the fly, also you can further protect access to the CA using NetworkPolicies.
- You should avoid using the embedded Zookeeper, but you can use an already existing helm chart as a dependency to install it [4] (caveat: the image used by that chart is not recommended for production use)
[1] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
[2] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
[3] https://issues.apache.org/jira/browse/NIFI-5542<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
[4] https://github.com/helm/charts/tree/master/incubator/zookeeper<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>> wrote:
Hi Dnyaneshwar,
We have nifi running on k8s for around 8-10 months. We create nifi cluster as part of CICD and then there is a stage which does the template deployment. Haven’t faced any major issues. Just sometime if a node reboots the old cluster member in nifi does not gets cleaned up.
Regards,
Varun
From: Andy LoPresto <al...@apache.org>>
Reply-To: <us...@nifi.apache.org>>
Date: Thursday, August 30, 2018 at 10:23 AM
To: <us...@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
I know other users are working on the same thing, so yes, NiFi + Kubernetes will allow you to stand up secure clusters. There is ongoing work targeted for upcoming releases to make this easier and more performant (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter Wilcsinszky has done a lot of great work here, and he may be able to share some resources he used/created.
[1] https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <dn...@persistent.com>> wrote:
Hi,
We have requirement of deploying NiFi on cloud platforms and we are considering kubernetes as orchestrator. I have knowledge on configuring nifi cluster, however, I am not sure on how things would go on kubernetes. Further, we are using Apache DS as LDAP server for authentication and planning to use embedded zookeeper instance to make the zookeeper cluster.
Any help or pointer to documentation would be appreciated.
Thank You.
Regards,
Dnyaneshwar Pawar
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
________________________________
This email may be confidential. If you are not the intended recipient, please notify us immediately and delete this copy from your system.
*Wire Fraud is Real*. Before wiring any money, call the intended recipient at a number you know is valid to confirm the instructions. Additionally, please note that the sender does not have authority to bind a party to a real estate contract via written or verbal communication.
Re: Secure NiFi cluster on kubernetes.
Posted by Peter Wilcsinszky <pe...@gmail.com>.
I beleive the problem is that you don't use PersistentVolumeClaims with
your statefulset. Do you have a specific reason not using persistent
volumes for your data and/or mutable config (authorizations.xml, users.xml,
flow.xml.gz)?
On Sun, Sep 23, 2018 at 7:25 PM Peter Wilcsinszky <
peterwilcsinszky@gmail.com> wrote:
> Sorry I did not understand the problem first, but now I do and will check
> this.
>
> (Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print
> $2}'|head -1` you could simply use `hostname -f`, but that is not the
> problem. )
>
>
>
> On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar <Va...@zaplabs.com>
> wrote:
>
>> This is my service.yaml (2 services one headless and one regular) and
>> depoloyment.yaml, let me know if this helps.:
>>
>>
>>
>> *kind: *Service
>> *apiVersion: *v1
>>
>> *metadata: name: *nifi-sync
>> *namespace: *nifi
>>
>> *labels: app: *nifi
>>
>>
>> *spec: ports: *- *name: *prometheus-jmx
>> *port: *8079
>> - *name: *web
>> *port: *8080
>> - *name: *cluster
>> *port: *8082
>> - *name: *misc
>> *port: *9001
>> *type: *NodePort
>>
>> *selector: app: *nifi
>>
>>
>> ---
>>
>> *kind: *Service
>> *apiVersion: *v1
>>
>> *metadata: name: *nifi
>>
>> *spec: clusterIP: *None
>>
>> *selector: app: *nifi
>>
>> *ports: *- *protocol: *TCP
>> *port: *8081
>> *targetPort: *8081
>>
>>
>>
>>
>>
>> ################
>>
>>
>>
>> *apiVersion: *apps/v1beta1
>> *kind: *StatefulSet
>>
>> *metadata: name: *nifi
>>
>> *spec: serviceName: *nifi
>> *replicas: *3
>> *podManagementPolicy: *Parallel
>>
>> *updateStrategy: type: *RollingUpdate
>>
>>
>>
>> *template: metadata: labels: app: *nifi
>>
>>
>>
>>
>> *spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: *-
>>
>> *labelSelector: matchExpressions: *- *key: *
>> *"app" **operator: *In
>>
>> *values: *- nifi
>> *topologyKey: *
>> *"kubernetes.io/hostname <http://kubernetes.io/hostname>" *
>> *imagePullSecrets: *- *name: *us-west-2-ecr-registry
>>
>> *containers: *- *name: *nifi
>> *image: *XXXXXXXXX.amazonaws.com/devops/nifi-1.7.0:v11-stateful
>>
>> *command: *-
>> *"/bin/sh" *-
>> *"-c" *- >
>> wget http://xxxxxxxx/build/deploy/configure_statefulset.sh -O /tmp/configure_statefulset.sh;
>> chmod +x /tmp/configure_statefulset.sh;
>> /tmp/configure_statefulset.sh;
>> apt-get -y install dnsutils;
>> sed -i -e "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
>> sed -i -e "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
>> sed -i -e "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
>> sed -i -e "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1 min|" $path/conf/nifi.properties;
>> mkdir -p $path/state/zookeeper;
>> touch $path/state/zookeeper/myid;
>> echo `hostname |rev | cut -d'-' -f 1 | rev` > $path/state/zookeeper/myid;
>> sed -i -e "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
>> sed -i -e "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
>> sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
>> sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
>> /opt/nifi/nifi-1.7.0/bin/nifi.sh run
>>
>> *securityContext: privileged: *true
>> *imagePullPolicy: *Always
>>
>> *ports: *- *containerPort: *8080
>> - *containerPort: *8081
>>
>>
>>
>>
>>
>> *From: *Peter Wilcsinszky <pe...@gmail.com>
>> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
>> *Date: *Sunday, September 23, 2018 at 6:33 AM
>> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
>> *Subject: *Re: Secure NiFi cluster on kubernetes.
>>
>>
>>
>> Hi Varun,
>>
>>
>>
>> hard to tell without seeing your statefulset config. How do you add the
>> new nodes? If you add them through the statefulset (kubectl scale
>> statefulset <your statefulset's name> --replicas <desired replica count>)
>>
>> the nodes should have the names nifi-3, nifi-4 instead of the ones on
>> your screenshot. But again, this is going to be hard to debug without
>> seeing your config.
>>
>>
>>
>> Peter
>>
>>
>>
>> On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar <Va...@zaplabs.com>
>> wrote:
>>
>> Hi Peter,
>>
>>
>>
>> I tried your suggestion of using statefulset in k8s. The problem is still
>> there. The new nodes join the cluster but the old nodes still remains am I
>> missing something. I am guessing each nodes get an Id which is the deciding
>> factor in cluster config and not the node address, that’s the reason I am
>> seeing 3/5.
>>
>>
>>
>>
>>
>> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>>
>> "status": "CONNECTED",
>>
>> "message": "Connection requested from existing node. Setting
>> status to connecting."
>>
>> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>>
>> "status": "CONNECTED",
>>
>> "message": "Connection requested from existing node. Setting
>> status to connecting."
>>
>> "address": "nifi-0.nifi.nifi.svc.cluster.local",
>>
>> "status": "CONNECTED",
>>
>> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>>
>> "status": "DISCONNECTED",
>>
>> "message": "Node disconnected from cluster due to Have not
>> received a heartbeat from node in 44 seconds"
>>
>> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>>
>> "status": "DISCONNECTED",
>>
>> "message": "Node disconnected from cluster due to Have not
>> received a heartbeat from node in 44 seconds"
>>
>>
>>
>> [image: cid:image001.png@01D452E1.9D0F93B0]
>>
>> *From: *Peter Wilcsinszky <pe...@gmail.com>
>> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
>> *Date: *Friday, August 31, 2018 at 10:01 AM
>> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
>> *Subject: *Re: Secure NiFi cluster on kubernetes.
>>
>>
>>
>>
>>
>> On Fri, 31 Aug 2018, 16:51 Varun Tomar, <Va...@zaplabs.com> wrote:
>>
>> Hi Peter,
>>
>>
>>
>> We started using nifi as statefulset last year you but moved to
>> deployment.
>>
>>
>>
>> -CICD tool Spinnaker does not support statefulsets.
>>
>> - We have also customized logback.xml as it was log within log issue
>> which was not getting parsed properly in ELK
>>
>> - For ports and cluster IP I pass them as argument so even if the pod
>> reboot we don't have any issues.
>>
>> Why do you need to pass an IP?
>>
>>
>>
>> - we also use external zookeeper.
>>
>>
>>
>> I dint find any benefit of running statefulset .
>>
>>
>>
>> The only issue as I said is if we restart any undeying node we extra node
>> and old nodes does not get deleted.
>>
>> With a statefulset you wouldnt have issues with that and you would have
>> stable persistent volumes as well.
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>> Varun
>>
>>
>> ------------------------------
>>
>> *From:* Peter Wilcsinszky <pe...@gmail.com>
>> *Sent:* Friday, August 31, 2018 2:50 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: Secure NiFi cluster on kubernetes.
>>
>>
>>
>> Hi Dnyaneshwar,
>>
>>
>>
>> as Andy mentioned we are working on running NiFi in Kubernetes but I'm
>> not sure when it will be available publicly. Some pointers that can help by
>> then:
>>
>> - You should use a StatefulSet to manage NiFi pods
>>
>> - Probably Helm charts are the most efficient way to get started
>>
>> - I recommend using the official NiFi image and wrapping the original
>> nifi.sh script from the Kubernetes pod spec similarly how we do it in the
>> Docker image [1]. Caveats: setting dynamic properties like
>> nifi.web.http.host from the wrapper script is a good idea, but for more
>> static properties like nifi.web.http.port you may want to use the config
>> files directly as configmaps and do templating using Helm. This is
>> especially true for more complex configurations like the authorizers.xml or
>> the login-identity-providers.xml.
>>
>> - Authorizations in NiFi can be configured for the initial cluster
>> setup, but needs to be done manually when you add a new Node to the cluster
>> above the initial cluster size. Also these extra nodes should have a
>> vanilla authorizations.xml to avoid conflicts when joining to the existing
>> ones. You can use the wrapper script to decide which configmap to use when
>> starting the container. Once the pod has started you still have to add the
>> node and authorize it manually using the UI. There is ongoing work to make
>> this more dynamic: [3]
>>
>> - We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in
>> server mode. The NiFi pods have an init container that uses tls-toolkit in
>> client mode to request and receive certificates from the CA server. The
>> communication is protected using a shared secret that is generated inside
>> the cluster on the fly, also you can further protect access to the CA using
>> NetworkPolicies.
>>
>> - You should avoid using the embedded Zookeeper, but you can use an
>> already existing helm chart as a dependency to install it [4] (caveat: the
>> image used by that chart is not recommended for production use)
>>
>>
>>
>> [1]
>> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
>>
>> [2]
>> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
>>
>> [3] https://issues.apache.org/jira/browse/NIFI-5542
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
>>
>> [4] https://github.com/helm/charts/tree/master/incubator/zookeeper
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
>>
>>
>>
>> On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>
>> wrote:
>>
>> Hi Dnyaneshwar,
>>
>>
>>
>> We have nifi running on k8s for around 8-10 months. We create nifi
>> cluster as part of CICD and then there is a stage which does the template
>> deployment. Haven’t faced any major issues. Just sometime if a node reboots
>> the old cluster member in nifi does not gets cleaned up.
>>
>>
>>
>> Regards,
>>
>> Varun
>>
>>
>>
>> *From: *Andy LoPresto <al...@apache.org>
>> *Reply-To: *<us...@nifi.apache.org>
>> *Date: *Thursday, August 30, 2018 at 10:23 AM
>> *To: *<us...@nifi.apache.org>
>> *Subject: *Re: Secure NiFi cluster on kubernetes.
>>
>>
>>
>> Hi Dnyaneshwar,
>>
>>
>>
>> I know other users are working on the same thing, so yes, NiFi +
>> Kubernetes will allow you to stand up secure clusters. There is ongoing
>> work targeted for upcoming releases to make this easier and more performant
>> (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter
>> Wilcsinszky has done a lot of great work here, and he may be able to share
>> some resources he used/created.
>>
>>
>>
>> [1]
>> https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
>>
>>
>>
>> Andy LoPresto
>>
>> alopresto@apache.org
>>
>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>>
>>
>> On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <
>> dnyaneshwar_pawar@persistent.com> wrote:
>>
>>
>>
>> Hi,
>>
>>
>>
>> We have requirement of deploying NiFi on cloud platforms and we are
>> considering kubernetes as orchestrator. I have knowledge on configuring
>> nifi cluster, however, I am not sure on how things would go on kubernetes.
>> Further, we are using Apache DS as LDAP server for authentication and
>> planning to use embedded zookeeper instance to make the zookeeper cluster.
>>
>> Any help or pointer to documentation would be appreciated.
>>
>>
>>
>> Thank You.
>>
>>
>>
>> Regards,
>>
>> Dnyaneshwar Pawar
>>
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the property of Persistent Systems Ltd. It is intended only for the use of
>> the individual or entity to which it is addressed. If you are not the
>> intended recipient, you are not authorized to read, retain, copy, print,
>> distribute or use this message. If you have received this communication in
>> error, please notify the sender and delete all copies of this message.
>> Persistent Systems Ltd. does not accept any liability for virus infected
>> mails.
>>
>>
>> ------------------------------
>>
>> This email may be confidential. If you are not the intended recipient,
>> please notify us immediately and delete this copy from your system.
>>
>> *Wire Fraud is Real*. Before wiring any money, call the intended
>> recipient at a number you know is valid to confirm the instructions. Additionally,
>> please note that the sender does not have authority to bind a party to a
>> real estate contract via written or verbal communication.
>>
>>
Re: Secure NiFi cluster on kubernetes.
Posted by Peter Wilcsinszky <pe...@gmail.com>.
Sorry I did not understand the problem first, but now I do and will check
this.
(Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print
$2}'|head -1` you could simply use `hostname -f`, but that is not the
problem. )
On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar <Va...@zaplabs.com> wrote:
> This is my service.yaml (2 services one headless and one regular) and
> depoloyment.yaml, let me know if this helps.:
>
>
>
> *kind: *Service
> *apiVersion: *v1
>
> *metadata: name: *nifi-sync
> *namespace: *nifi
>
> *labels: app: *nifi
>
>
> *spec: ports: *- *name: *prometheus-jmx
> *port: *8079
> - *name: *web
> *port: *8080
> - *name: *cluster
> *port: *8082
> - *name: *misc
> *port: *9001
> *type: *NodePort
>
> *selector: app: *nifi
>
>
> ---
>
> *kind: *Service
> *apiVersion: *v1
>
> *metadata: name: *nifi
>
> *spec: clusterIP: *None
>
> *selector: app: *nifi
>
> *ports: *- *protocol: *TCP
> *port: *8081
> *targetPort: *8081
>
>
>
>
>
> ################
>
>
>
> *apiVersion: *apps/v1beta1
> *kind: *StatefulSet
>
> *metadata: name: *nifi
>
> *spec: serviceName: *nifi
> *replicas: *3
> *podManagementPolicy: *Parallel
>
> *updateStrategy: type: *RollingUpdate
>
>
>
> *template: metadata: labels: app: *nifi
>
>
>
>
> *spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: *-
>
> *labelSelector: matchExpressions: *- *key: *
> *"app" **operator: *In
>
> *values: *- nifi
> *topologyKey: *
> *"kubernetes.io/hostname <http://kubernetes.io/hostname>" *
> *imagePullSecrets: *- *name: *us-west-2-ecr-registry
>
> *containers: *- *name: *nifi
> *image: *XXXXXXXXX.amazonaws.com/devops/nifi-1.7.0:v11-stateful
>
> *command: *-
> *"/bin/sh" *-
> *"-c" *- >
> wget http://xxxxxxxx/build/deploy/configure_statefulset.sh -O /tmp/configure_statefulset.sh;
> chmod +x /tmp/configure_statefulset.sh;
> /tmp/configure_statefulset.sh;
> apt-get -y install dnsutils;
> sed -i -e "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1 min|" $path/conf/nifi.properties;
> mkdir -p $path/state/zookeeper;
> touch $path/state/zookeeper/myid;
> echo `hostname |rev | cut -d'-' -f 1 | rev` > $path/state/zookeeper/myid;
> sed -i -e "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
> sed -i -e "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
> sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
> sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
> /opt/nifi/nifi-1.7.0/bin/nifi.sh run
>
> *securityContext: privileged: *true
> *imagePullPolicy: *Always
>
> *ports: *- *containerPort: *8080
> - *containerPort: *8081
>
>
>
>
>
> *From: *Peter Wilcsinszky <pe...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Sunday, September 23, 2018 at 6:33 AM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Varun,
>
>
>
> hard to tell without seeing your statefulset config. How do you add the
> new nodes? If you add them through the statefulset (kubectl scale
> statefulset <your statefulset's name> --replicas <desired replica count>)
>
> the nodes should have the names nifi-3, nifi-4 instead of the ones on your
> screenshot. But again, this is going to be hard to debug without seeing
> your config.
>
>
>
> Peter
>
>
>
> On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar <Va...@zaplabs.com>
> wrote:
>
> Hi Peter,
>
>
>
> I tried your suggestion of using statefulset in k8s. The problem is still
> there. The new nodes join the cluster but the old nodes still remains am I
> missing something. I am guessing each nodes get an Id which is the deciding
> factor in cluster config and not the node address, that’s the reason I am
> seeing 3/5.
>
>
>
>
>
> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "message": "Connection requested from existing node. Setting
> status to connecting."
>
> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "message": "Connection requested from existing node. Setting
> status to connecting."
>
> "address": "nifi-0.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>
> "status": "DISCONNECTED",
>
> "message": "Node disconnected from cluster due to Have not
> received a heartbeat from node in 44 seconds"
>
> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>
> "status": "DISCONNECTED",
>
> "message": "Node disconnected from cluster due to Have not
> received a heartbeat from node in 44 seconds"
>
>
>
> [image: cid:image001.png@01D452E1.9D0F93B0]
>
> *From: *Peter Wilcsinszky <pe...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Friday, August 31, 2018 at 10:01 AM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
>
>
> On Fri, 31 Aug 2018, 16:51 Varun Tomar, <Va...@zaplabs.com> wrote:
>
> Hi Peter,
>
>
>
> We started using nifi as statefulset last year you but moved to deployment.
>
>
>
> -CICD tool Spinnaker does not support statefulsets.
>
> - We have also customized logback.xml as it was log within log issue which
> was not getting parsed properly in ELK
>
> - For ports and cluster IP I pass them as argument so even if the pod
> reboot we don't have any issues.
>
> Why do you need to pass an IP?
>
>
>
> - we also use external zookeeper.
>
>
>
> I dint find any benefit of running statefulset .
>
>
>
> The only issue as I said is if we restart any undeying node we extra node
> and old nodes does not get deleted.
>
> With a statefulset you wouldnt have issues with that and you would have
> stable persistent volumes as well.
>
>
>
>
>
>
>
> Regards,
>
> Varun
>
>
> ------------------------------
>
> *From:* Peter Wilcsinszky <pe...@gmail.com>
> *Sent:* Friday, August 31, 2018 2:50 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Dnyaneshwar,
>
>
>
> as Andy mentioned we are working on running NiFi in Kubernetes but I'm not
> sure when it will be available publicly. Some pointers that can help by
> then:
>
> - You should use a StatefulSet to manage NiFi pods
>
> - Probably Helm charts are the most efficient way to get started
>
> - I recommend using the official NiFi image and wrapping the original
> nifi.sh script from the Kubernetes pod spec similarly how we do it in the
> Docker image [1]. Caveats: setting dynamic properties like
> nifi.web.http.host from the wrapper script is a good idea, but for more
> static properties like nifi.web.http.port you may want to use the config
> files directly as configmaps and do templating using Helm. This is
> especially true for more complex configurations like the authorizers.xml or
> the login-identity-providers.xml.
>
> - Authorizations in NiFi can be configured for the initial cluster setup,
> but needs to be done manually when you add a new Node to the cluster above
> the initial cluster size. Also these extra nodes should have a vanilla
> authorizations.xml to avoid conflicts when joining to the existing ones.
> You can use the wrapper script to decide which configmap to use when
> starting the container. Once the pod has started you still have to add the
> node and authorize it manually using the UI. There is ongoing work to make
> this more dynamic: [3]
>
> - We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in
> server mode. The NiFi pods have an init container that uses tls-toolkit in
> client mode to request and receive certificates from the CA server. The
> communication is protected using a shared secret that is generated inside
> the cluster on the fly, also you can further protect access to the CA using
> NetworkPolicies.
>
> - You should avoid using the embedded Zookeeper, but you can use an
> already existing helm chart as a dependency to install it [4] (caveat: the
> image used by that chart is not recommended for production use)
>
>
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
>
> [2]
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
>
> [3] https://issues.apache.org/jira/browse/NIFI-5542
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
>
> [4] https://github.com/helm/charts/tree/master/incubator/zookeeper
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
>
>
>
> On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>
> wrote:
>
> Hi Dnyaneshwar,
>
>
>
> We have nifi running on k8s for around 8-10 months. We create nifi cluster
> as part of CICD and then there is a stage which does the template
> deployment. Haven’t faced any major issues. Just sometime if a node reboots
> the old cluster member in nifi does not gets cleaned up.
>
>
>
> Regards,
>
> Varun
>
>
>
> *From: *Andy LoPresto <al...@apache.org>
> *Reply-To: *<us...@nifi.apache.org>
> *Date: *Thursday, August 30, 2018 at 10:23 AM
> *To: *<us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Dnyaneshwar,
>
>
>
> I know other users are working on the same thing, so yes, NiFi +
> Kubernetes will allow you to stand up secure clusters. There is ongoing
> work targeted for upcoming releases to make this easier and more performant
> (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter
> Wilcsinszky has done a lot of great work here, and he may be able to share
> some resources he used/created.
>
>
>
> [1]
> https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> *alopresto.apache@gmail.com <al...@gmail.com>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <
> dnyaneshwar_pawar@persistent.com> wrote:
>
>
>
> Hi,
>
>
>
> We have requirement of deploying NiFi on cloud platforms and we are
> considering kubernetes as orchestrator. I have knowledge on configuring
> nifi cluster, however, I am not sure on how things would go on kubernetes.
> Further, we are using Apache DS as LDAP server for authentication and
> planning to use embedded zookeeper instance to make the zookeeper cluster.
>
> Any help or pointer to documentation would be appreciated.
>
>
>
> Thank You.
>
>
>
> Regards,
>
> Dnyaneshwar Pawar
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
>
> ------------------------------
>
> This email may be confidential. If you are not the intended recipient,
> please notify us immediately and delete this copy from your system.
>
> *Wire Fraud is Real*. Before wiring any money, call the intended
> recipient at a number you know is valid to confirm the instructions. Additionally,
> please note that the sender does not have authority to bind a party to a
> real estate contract via written or verbal communication.
>
>
Re: Secure NiFi cluster on kubernetes.
Posted by Varun Tomar <Va...@zaplabs.com>.
This is my service.yaml (2 services one headless and one regular) and depoloyment.yaml, let me know if this helps.:
kind: Service
apiVersion: v1
metadata:
name: nifi-sync
namespace: nifi
labels:
app: nifi
spec:
ports:
- name: prometheus-jmx
port: 8079
- name: web
port: 8080
- name: cluster
port: 8082
- name: misc
port: 9001
type: NodePort
selector:
app: nifi
---
kind: Service
apiVersion: v1
metadata:
name: nifi
spec:
clusterIP: None
selector:
app: nifi
ports:
- protocol: TCP
port: 8081
targetPort: 8081
################
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nifi
spec:
serviceName: nifi
replicas: 3
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: nifi
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- nifi
topologyKey: "kubernetes.io/hostname"
imagePullSecrets:
- name: us-west-2-ecr-registry
containers:
- name: nifi
image: XXXXXXXXX.amazonaws.com/devops/nifi-1.7.0:v11-stateful
command:
- "/bin/sh"
- "-c"
- >
wget http://xxxxxxxx/build/deploy/configure_statefulset.sh -O /tmp/configure_statefulset.sh;
chmod +x /tmp/configure_statefulset.sh;
/tmp/configure_statefulset.sh;
apt-get -y install dnsutils;
sed -i -e "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1 min|" $path/conf/nifi.properties;
mkdir -p $path/state/zookeeper;
touch $path/state/zookeeper/myid;
echo `hostname |rev | cut -d'-' -f 1 | rev` > $path/state/zookeeper/myid;
sed -i -e "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|" /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
/opt/nifi/nifi-1.7.0/bin/nifi.sh run
securityContext:
privileged: true
imagePullPolicy: Always
ports:
- containerPort: 8080
- containerPort: 8081
From: Peter Wilcsinszky <pe...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Sunday, September 23, 2018 at 6:33 AM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Varun,
hard to tell without seeing your statefulset config. How do you add the new nodes? If you add them through the statefulset (kubectl scale statefulset <your statefulset's name> --replicas <desired replica count>)
the nodes should have the names nifi-3, nifi-4 instead of the ones on your screenshot. But again, this is going to be hard to debug without seeing your config.
Peter
On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar <Va...@zaplabs.com>> wrote:
Hi Peter,
I tried your suggestion of using statefulset in k8s. The problem is still there. The new nodes join the cluster but the old nodes still remains am I missing something. I am guessing each nodes get an Id which is the deciding factor in cluster config and not the node address, that’s the reason I am seeing 3/5.
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status to connecting."
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status to connecting."
"address": "nifi-0.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received a heartbeat from node in 44 seconds"
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received a heartbeat from node in 44 seconds"
[cid:image001.png@01D452E1.9D0F93B0]
From: Peter Wilcsinszky <pe...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Friday, August 31, 2018 at 10:01 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
On Fri, 31 Aug 2018, 16:51 Varun Tomar, <Va...@zaplabs.com>> wrote:
Hi Peter,
We started using nifi as statefulset last year you but moved to deployment.
-CICD tool Spinnaker does not support statefulsets.
- We have also customized logback.xml as it was log within log issue which was not getting parsed properly in ELK
- For ports and cluster IP I pass them as argument so even if the pod reboot we don't have any issues.
Why do you need to pass an IP?
- we also use external zookeeper.
I dint find any benefit of running statefulset .
The only issue as I said is if we restart any undeying node we extra node and old nodes does not get deleted.
With a statefulset you wouldnt have issues with that and you would have stable persistent volumes as well.
Regards,
Varun
________________________________
From: Peter Wilcsinszky <pe...@gmail.com>>
Sent: Friday, August 31, 2018 2:50 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
as Andy mentioned we are working on running NiFi in Kubernetes but I'm not sure when it will be available publicly. Some pointers that can help by then:
- You should use a StatefulSet to manage NiFi pods
- Probably Helm charts are the most efficient way to get started
- I recommend using the official NiFi image and wrapping the original nifi.sh script from the Kubernetes pod spec similarly how we do it in the Docker image [1]. Caveats: setting dynamic properties like nifi.web.http.host from the wrapper script is a good idea, but for more static properties like nifi.web.http.port you may want to use the config files directly as configmaps and do templating using Helm. This is especially true for more complex configurations like the authorizers.xml or the login-identity-providers.xml.
- Authorizations in NiFi can be configured for the initial cluster setup, but needs to be done manually when you add a new Node to the cluster above the initial cluster size. Also these extra nodes should have a vanilla authorizations.xml to avoid conflicts when joining to the existing ones. You can use the wrapper script to decide which configmap to use when starting the container. Once the pod has started you still have to add the node and authorize it manually using the UI. There is ongoing work to make this more dynamic: [3]
- We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in server mode. The NiFi pods have an init container that uses tls-toolkit in client mode to request and receive certificates from the CA server. The communication is protected using a shared secret that is generated inside the cluster on the fly, also you can further protect access to the CA using NetworkPolicies.
- You should avoid using the embedded Zookeeper, but you can use an already existing helm chart as a dependency to install it [4] (caveat: the image used by that chart is not recommended for production use)
[1] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
[2] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
[3] https://issues.apache.org/jira/browse/NIFI-5542<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
[4] https://github.com/helm/charts/tree/master/incubator/zookeeper<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>> wrote:
Hi Dnyaneshwar,
We have nifi running on k8s for around 8-10 months. We create nifi cluster as part of CICD and then there is a stage which does the template deployment. Haven’t faced any major issues. Just sometime if a node reboots the old cluster member in nifi does not gets cleaned up.
Regards,
Varun
From: Andy LoPresto <al...@apache.org>>
Reply-To: <us...@nifi.apache.org>>
Date: Thursday, August 30, 2018 at 10:23 AM
To: <us...@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
I know other users are working on the same thing, so yes, NiFi + Kubernetes will allow you to stand up secure clusters. There is ongoing work targeted for upcoming releases to make this easier and more performant (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter Wilcsinszky has done a lot of great work here, and he may be able to share some resources he used/created.
[1] https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <dn...@persistent.com>> wrote:
Hi,
We have requirement of deploying NiFi on cloud platforms and we are considering kubernetes as orchestrator. I have knowledge on configuring nifi cluster, however, I am not sure on how things would go on kubernetes. Further, we are using Apache DS as LDAP server for authentication and planning to use embedded zookeeper instance to make the zookeeper cluster.
Any help or pointer to documentation would be appreciated.
Thank You.
Regards,
Dnyaneshwar Pawar
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
________________________________
This email may be confidential. If you are not the intended recipient, please notify us immediately and delete this copy from your system.
*Wire Fraud is Real*. Before wiring any money, call the intended recipient at a number you know is valid to confirm the instructions. Additionally, please note that the sender does not have authority to bind a party to a real estate contract via written or verbal communication.
Re: Secure NiFi cluster on kubernetes.
Posted by Peter Wilcsinszky <pe...@gmail.com>.
Hi Varun,
hard to tell without seeing your statefulset config. How do you add the new
nodes? If you add them through the statefulset (kubectl scale statefulset
<your statefulset's name> --replicas <desired replica count>)
the nodes should have the names nifi-3, nifi-4 instead of the ones on your
screenshot. But again, this is going to be hard to debug without seeing
your config.
Peter
On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar <Va...@zaplabs.com>
wrote:
> Hi Peter,
>
>
>
> I tried your suggestion of using statefulset in k8s. The problem is still
> there. The new nodes join the cluster but the old nodes still remains am I
> missing something. I am guessing each nodes get an Id which is the deciding
> factor in cluster config and not the node address, that’s the reason I am
> seeing 3/5.
>
>
>
>
>
> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "message": "Connection requested from existing node. Setting
> status to connecting."
>
> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "message": "Connection requested from existing node. Setting
> status to connecting."
>
> "address": "nifi-0.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>
> "status": "DISCONNECTED",
>
> "message": "Node disconnected from cluster due to Have not
> received a heartbeat from node in 44 seconds"
>
> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>
> "status": "DISCONNECTED",
>
> "message": "Node disconnected from cluster due to Have not
> received a heartbeat from node in 44 seconds"
>
>
>
> *From: *Peter Wilcsinszky <pe...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Friday, August 31, 2018 at 10:01 AM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
>
>
> On Fri, 31 Aug 2018, 16:51 Varun Tomar, <Va...@zaplabs.com> wrote:
>
> Hi Peter,
>
>
>
> We started using nifi as statefulset last year you but moved to deployment.
>
>
>
> -CICD tool Spinnaker does not support statefulsets.
>
> - We have also customized logback.xml as it was log within log issue which
> was not getting parsed properly in ELK
>
> - For ports and cluster IP I pass them as argument so even if the pod
> reboot we don't have any issues.
>
> Why do you need to pass an IP?
>
>
>
> - we also use external zookeeper.
>
>
>
> I dint find any benefit of running statefulset .
>
>
>
> The only issue as I said is if we restart any undeying node we extra node
> and old nodes does not get deleted.
>
> With a statefulset you wouldnt have issues with that and you would have
> stable persistent volumes as well.
>
>
>
>
>
>
>
> Regards,
>
> Varun
>
>
> ------------------------------
>
> *From:* Peter Wilcsinszky <pe...@gmail.com>
> *Sent:* Friday, August 31, 2018 2:50 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Dnyaneshwar,
>
>
>
> as Andy mentioned we are working on running NiFi in Kubernetes but I'm not
> sure when it will be available publicly. Some pointers that can help by
> then:
>
> - You should use a StatefulSet to manage NiFi pods
>
> - Probably Helm charts are the most efficient way to get started
>
> - I recommend using the official NiFi image and wrapping the original
> nifi.sh script from the Kubernetes pod spec similarly how we do it in the
> Docker image [1]. Caveats: setting dynamic properties like
> nifi.web.http.host from the wrapper script is a good idea, but for more
> static properties like nifi.web.http.port you may want to use the config
> files directly as configmaps and do templating using Helm. This is
> especially true for more complex configurations like the authorizers.xml or
> the login-identity-providers.xml.
>
> - Authorizations in NiFi can be configured for the initial cluster setup,
> but needs to be done manually when you add a new Node to the cluster above
> the initial cluster size. Also these extra nodes should have a vanilla
> authorizations.xml to avoid conflicts when joining to the existing ones.
> You can use the wrapper script to decide which configmap to use when
> starting the container. Once the pod has started you still have to add the
> node and authorize it manually using the UI. There is ongoing work to make
> this more dynamic: [3]
>
> - We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in
> server mode. The NiFi pods have an init container that uses tls-toolkit in
> client mode to request and receive certificates from the CA server. The
> communication is protected using a shared secret that is generated inside
> the cluster on the fly, also you can further protect access to the CA using
> NetworkPolicies.
>
> - You should avoid using the embedded Zookeeper, but you can use an
> already existing helm chart as a dependency to install it [4] (caveat: the
> image used by that chart is not recommended for production use)
>
>
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
>
> [2]
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
>
> [3] https://issues.apache.org/jira/browse/NIFI-5542
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
>
> [4] https://github.com/helm/charts/tree/master/incubator/zookeeper
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
>
>
>
> On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>
> wrote:
>
> Hi Dnyaneshwar,
>
>
>
> We have nifi running on k8s for around 8-10 months. We create nifi cluster
> as part of CICD and then there is a stage which does the template
> deployment. Haven’t faced any major issues. Just sometime if a node reboots
> the old cluster member in nifi does not gets cleaned up.
>
>
>
> Regards,
>
> Varun
>
>
>
> *From: *Andy LoPresto <al...@apache.org>
> *Reply-To: *<us...@nifi.apache.org>
> *Date: *Thursday, August 30, 2018 at 10:23 AM
> *To: *<us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Dnyaneshwar,
>
>
>
> I know other users are working on the same thing, so yes, NiFi +
> Kubernetes will allow you to stand up secure clusters. There is ongoing
> work targeted for upcoming releases to make this easier and more performant
> (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter
> Wilcsinszky has done a lot of great work here, and he may be able to share
> some resources he used/created.
>
>
>
> [1]
> https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> *alopresto.apache@gmail.com <al...@gmail.com>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <
> dnyaneshwar_pawar@persistent.com> wrote:
>
>
>
> Hi,
>
>
>
> We have requirement of deploying NiFi on cloud platforms and we are
> considering kubernetes as orchestrator. I have knowledge on configuring
> nifi cluster, however, I am not sure on how things would go on kubernetes.
> Further, we are using Apache DS as LDAP server for authentication and
> planning to use embedded zookeeper instance to make the zookeeper cluster.
>
> Any help or pointer to documentation would be appreciated.
>
>
>
> Thank You.
>
>
>
> Regards,
>
> Dnyaneshwar Pawar
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
>
> ------------------------------
>
> This email may be confidential. If you are not the intended recipient,
> please notify us immediately and delete this copy from your system.
>
> *Wire Fraud is Real*. Before wiring any money, call the intended
> recipient at a number you know is valid to confirm the instructions. Additionally,
> please note that the sender does not have authority to bind a party to a
> real estate contract via written or verbal communication.
>
>
Re: Secure NiFi cluster on kubernetes.
Posted by Varun Tomar <Va...@zaplabs.com>.
Hi Peter,
I tried your suggestion of using statefulset in k8s. The problem is still there. The new nodes join the cluster but the old nodes still remains am I missing something. I am guessing each nodes get an Id which is the deciding factor in cluster config and not the node address, that’s the reason I am seeing 3/5.
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status to connecting."
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status to connecting."
"address": "nifi-0.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received a heartbeat from node in 44 seconds"
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received a heartbeat from node in 44 seconds"
[cid:image001.png@01D452E1.9D0F93B0]
From: Peter Wilcsinszky <pe...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Friday, August 31, 2018 at 10:01 AM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
On Fri, 31 Aug 2018, 16:51 Varun Tomar, <Va...@zaplabs.com>> wrote:
Hi Peter,
We started using nifi as statefulset last year you but moved to deployment.
-CICD tool Spinnaker does not support statefulsets.
- We have also customized logback.xml as it was log within log issue which was not getting parsed properly in ELK
- For ports and cluster IP I pass them as argument so even if the pod reboot we don't have any issues.
Why do you need to pass an IP?
- we also use external zookeeper.
I dint find any benefit of running statefulset .
The only issue as I said is if we restart any undeying node we extra node and old nodes does not get deleted.
With a statefulset you wouldnt have issues with that and you would have stable persistent volumes as well.
Regards,
Varun
________________________________
From: Peter Wilcsinszky <pe...@gmail.com>>
Sent: Friday, August 31, 2018 2:50 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
as Andy mentioned we are working on running NiFi in Kubernetes but I'm not sure when it will be available publicly. Some pointers that can help by then:
- You should use a StatefulSet to manage NiFi pods
- Probably Helm charts are the most efficient way to get started
- I recommend using the official NiFi image and wrapping the original nifi.sh script from the Kubernetes pod spec similarly how we do it in the Docker image [1]. Caveats: setting dynamic properties like nifi.web.http.host from the wrapper script is a good idea, but for more static properties like nifi.web.http.port you may want to use the config files directly as configmaps and do templating using Helm. This is especially true for more complex configurations like the authorizers.xml or the login-identity-providers.xml.
- Authorizations in NiFi can be configured for the initial cluster setup, but needs to be done manually when you add a new Node to the cluster above the initial cluster size. Also these extra nodes should have a vanilla authorizations.xml to avoid conflicts when joining to the existing ones. You can use the wrapper script to decide which configmap to use when starting the container. Once the pod has started you still have to add the node and authorize it manually using the UI. There is ongoing work to make this more dynamic: [3]
- We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in server mode. The NiFi pods have an init container that uses tls-toolkit in client mode to request and receive certificates from the CA server. The communication is protected using a shared secret that is generated inside the cluster on the fly, also you can further protect access to the CA using NetworkPolicies.
- You should avoid using the embedded Zookeeper, but you can use an already existing helm chart as a dependency to install it [4] (caveat: the image used by that chart is not recommended for production use)
[1] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
[2] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
[3] https://issues.apache.org/jira/browse/NIFI-5542<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
[4] https://github.com/helm/charts/tree/master/incubator/zookeeper<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>> wrote:
Hi Dnyaneshwar,
We have nifi running on k8s for around 8-10 months. We create nifi cluster as part of CICD and then there is a stage which does the template deployment. Haven’t faced any major issues. Just sometime if a node reboots the old cluster member in nifi does not gets cleaned up.
Regards,
Varun
From: Andy LoPresto <al...@apache.org>>
Reply-To: <us...@nifi.apache.org>>
Date: Thursday, August 30, 2018 at 10:23 AM
To: <us...@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
I know other users are working on the same thing, so yes, NiFi + Kubernetes will allow you to stand up secure clusters. There is ongoing work targeted for upcoming releases to make this easier and more performant (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter Wilcsinszky has done a lot of great work here, and he may be able to share some resources he used/created.
[1] https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <dn...@persistent.com>> wrote:
Hi,
We have requirement of deploying NiFi on cloud platforms and we are considering kubernetes as orchestrator. I have knowledge on configuring nifi cluster, however, I am not sure on how things would go on kubernetes. Further, we are using Apache DS as LDAP server for authentication and planning to use embedded zookeeper instance to make the zookeeper cluster.
Any help or pointer to documentation would be appreciated.
Thank You.
Regards,
Dnyaneshwar Pawar
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
________________________________
This email may be confidential. If you are not the intended recipient, please notify us immediately and delete this copy from your system.
*Wire Fraud is Real*. Before wiring any money, call the intended recipient at a number you know is valid to confirm the instructions. Additionally, please note that the sender does not have authority to bind a party to a real estate contract via written or verbal communication.
Re: Secure NiFi cluster on kubernetes.
Posted by Peter Wilcsinszky <pe...@gmail.com>.
On Fri, 31 Aug 2018, 16:51 Varun Tomar, <Va...@zaplabs.com> wrote:
> Hi Peter,
>
> We started using nifi as statefulset last year you but moved to deployment.
>
> -CICD tool Spinnaker does not support statefulsets.
> - We have also customized logback.xml as it was log within log issue which
> was not getting parsed properly in ELK
> - For ports and cluster IP I pass them as argument so even if the pod
> reboot we don't have any issues.
>
Why do you need to pass an IP?
- we also use external zookeeper.
>
> I dint find any benefit of running statefulset .
>
> The only issue as I said is if we restart any undeying node we extra node
> and old nodes does not get deleted.
>
With a statefulset you wouldnt have issues with that and you would have
stable persistent volumes as well.
>
> Regards,
> Varun
>
> ------------------------------
> *From:* Peter Wilcsinszky <pe...@gmail.com>
> *Sent:* Friday, August 31, 2018 2:50 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Secure NiFi cluster on kubernetes.
>
> Hi Dnyaneshwar,
>
> as Andy mentioned we are working on running NiFi in Kubernetes but I'm not
> sure when it will be available publicly. Some pointers that can help by
> then:
> - You should use a StatefulSet to manage NiFi pods
> - Probably Helm charts are the most efficient way to get started
> - I recommend using the official NiFi image and wrapping the original
> nifi.sh script from the Kubernetes pod spec similarly how we do it in the
> Docker image [1]. Caveats: setting dynamic properties like
> nifi.web.http.host from the wrapper script is a good idea, but for more
> static properties like nifi.web.http.port you may want to use the config
> files directly as configmaps and do templating using Helm. This is
> especially true for more complex configurations like the authorizers.xml or
> the login-identity-providers.xml.
> - Authorizations in NiFi can be configured for the initial cluster setup,
> but needs to be done manually when you add a new Node to the cluster above
> the initial cluster size. Also these extra nodes should have a vanilla
> authorizations.xml to avoid conflicts when joining to the existing ones.
> You can use the wrapper script to decide which configmap to use when
> starting the container. Once the pod has started you still have to add the
> node and authorize it manually using the UI. There is ongoing work to make
> this more dynamic: [3]
> - We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in
> server mode. The NiFi pods have an init container that uses tls-toolkit in
> client mode to request and receive certificates from the CA server. The
> communication is protected using a shared secret that is generated inside
> the cluster on the fly, also you can further protect access to the CA using
> NetworkPolicies.
> - You should avoid using the embedded Zookeeper, but you can use an
> already existing helm chart as a dependency to install it [4] (caveat: the
> image used by that chart is not recommended for production use)
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
> [2]
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
> [3] https://issues.apache.org/jira/browse/NIFI-5542
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
> [4] https://github.com/helm/charts/tree/master/incubator/zookeeper
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
>
> On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>
> wrote:
>
>> Hi Dnyaneshwar,
>>
>>
>>
>> We have nifi running on k8s for around 8-10 months. We create nifi
>> cluster as part of CICD and then there is a stage which does the template
>> deployment. Haven’t faced any major issues. Just sometime if a node reboots
>> the old cluster member in nifi does not gets cleaned up.
>>
>>
>>
>> Regards,
>>
>> Varun
>>
>>
>>
>> *From: *Andy LoPresto <al...@apache.org>
>> *Reply-To: *<us...@nifi.apache.org>
>> *Date: *Thursday, August 30, 2018 at 10:23 AM
>> *To: *<us...@nifi.apache.org>
>> *Subject: *Re: Secure NiFi cluster on kubernetes.
>>
>>
>>
>> Hi Dnyaneshwar,
>>
>>
>>
>> I know other users are working on the same thing, so yes, NiFi +
>> Kubernetes will allow you to stand up secure clusters. There is ongoing
>> work targeted for upcoming releases to make this easier and more performant
>> (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter
>> Wilcsinszky has done a lot of great work here, and he may be able to share
>> some resources he used/created.
>>
>>
>>
>> [1]
>> https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
>>
>>
>>
>> Andy LoPresto
>>
>> alopresto@apache.org
>>
>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>>
>>
>> On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <
>> dnyaneshwar_pawar@persistent.com> wrote:
>>
>>
>>
>> Hi,
>>
>>
>>
>> We have requirement of deploying NiFi on cloud platforms and we are
>> considering kubernetes as orchestrator. I have knowledge on configuring
>> nifi cluster, however, I am not sure on how things would go on kubernetes.
>> Further, we are using Apache DS as LDAP server for authentication and
>> planning to use embedded zookeeper instance to make the zookeeper cluster.
>>
>> Any help or pointer to documentation would be appreciated.
>>
>>
>>
>> Thank You.
>>
>>
>>
>> Regards,
>>
>> Dnyaneshwar Pawar
>>
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the property of Persistent Systems Ltd. It is intended only for the use of
>> the individual or entity to which it is addressed. If you are not the
>> intended recipient, you are not authorized to read, retain, copy, print,
>> distribute or use this message. If you have received this communication in
>> error, please notify the sender and delete all copies of this message.
>> Persistent Systems Ltd. does not accept any liability for virus infected
>> mails.
>>
>>
>>
>> ------------------------------
>> This email may be confidential. If you are not the intended recipient,
>> please notify us immediately and delete this copy from your system.
>>
>> *Wire Fraud is Real*. Before wiring any money, call the intended
> recipient at a number you know is valid to confirm the instructions. Additionally,
> please note that the sender does not have authority to bind a party to a
> real estate contract via written or verbal communication.
>
Re: Secure NiFi cluster on kubernetes.
Posted by Varun Tomar <Va...@zaplabs.com>.
Hi Peter,
We started using nifi as statefulset last year you but moved to deployment.
-CICD tool Spinnaker does not support statefulsets.
- We have also customized logback.xml as it was log within log issue which was not getting parsed properly in ELK
- For ports and cluster IP I pass them as argument so even if the pod reboot we don't have any issues.
- we also use external zookeeper.
I dint find any benefit of running statefulset .
The only issue as I said is if we restart any undeying node we extra node and old nodes does not get deleted.
Regards,
Varun
________________________________
From: Peter Wilcsinszky <pe...@gmail.com>
Sent: Friday, August 31, 2018 2:50 AM
To: users@nifi.apache.org
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
as Andy mentioned we are working on running NiFi in Kubernetes but I'm not sure when it will be available publicly. Some pointers that can help by then:
- You should use a StatefulSet to manage NiFi pods
- Probably Helm charts are the most efficient way to get started
- I recommend using the official NiFi image and wrapping the original nifi.sh script from the Kubernetes pod spec similarly how we do it in the Docker image [1]. Caveats: setting dynamic properties like nifi.web.http.host from the wrapper script is a good idea, but for more static properties like nifi.web.http.port you may want to use the config files directly as configmaps and do templating using Helm. This is especially true for more complex configurations like the authorizers.xml or the login-identity-providers.xml.
- Authorizations in NiFi can be configured for the initial cluster setup, but needs to be done manually when you add a new Node to the cluster above the initial cluster size. Also these extra nodes should have a vanilla authorizations.xml to avoid conflicts when joining to the existing ones. You can use the wrapper script to decide which configmap to use when starting the container. Once the pod has started you still have to add the node and authorize it manually using the UI. There is ongoing work to make this more dynamic: [3]
- We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in server mode. The NiFi pods have an init container that uses tls-toolkit in client mode to request and receive certificates from the CA server. The communication is protected using a shared secret that is generated inside the cluster on the fly, also you can further protect access to the CA using NetworkPolicies.
- You should avoid using the embedded Zookeeper, but you can use an already existing helm chart as a dependency to install it [4] (caveat: the image used by that chart is not recommended for production use)
[1] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
[2] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
[3] https://issues.apache.org/jira/browse/NIFI-5542<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
[4] https://github.com/helm/charts/tree/master/incubator/zookeeper<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>> wrote:
Hi Dnyaneshwar,
We have nifi running on k8s for around 8-10 months. We create nifi cluster as part of CICD and then there is a stage which does the template deployment. Haven’t faced any major issues. Just sometime if a node reboots the old cluster member in nifi does not gets cleaned up.
Regards,
Varun
From: Andy LoPresto <al...@apache.org>>
Reply-To: <us...@nifi.apache.org>>
Date: Thursday, August 30, 2018 at 10:23 AM
To: <us...@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
I know other users are working on the same thing, so yes, NiFi + Kubernetes will allow you to stand up secure clusters. There is ongoing work targeted for upcoming releases to make this easier and more performant (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter Wilcsinszky has done a lot of great work here, and he may be able to share some resources he used/created.
[1] https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <dn...@persistent.com>> wrote:
Hi,
We have requirement of deploying NiFi on cloud platforms and we are considering kubernetes as orchestrator. I have knowledge on configuring nifi cluster, however, I am not sure on how things would go on kubernetes. Further, we are using Apache DS as LDAP server for authentication and planning to use embedded zookeeper instance to make the zookeeper cluster.
Any help or pointer to documentation would be appreciated.
Thank You.
Regards,
Dnyaneshwar Pawar
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
________________________________
This email may be confidential. If you are not the intended recipient, please notify us immediately and delete this copy from your system.
*Wire Fraud is Real*. Before wiring any money, call the intended recipient at a number you know is valid to confirm the instructions. Additionally, please note that the sender does not have authority to bind a party to a real estate contract via written or verbal communication.
Re: Secure NiFi cluster on kubernetes.
Posted by Peter Wilcsinszky <pe...@gmail.com>.
Hi Dnyaneshwar,
as Andy mentioned we are working on running NiFi in Kubernetes but I'm not
sure when it will be available publicly. Some pointers that can help by
then:
- You should use a StatefulSet to manage NiFi pods
- Probably Helm charts are the most efficient way to get started
- I recommend using the official NiFi image and wrapping the original
nifi.sh script from the Kubernetes pod spec similarly how we do it in the
Docker image [1]. Caveats: setting dynamic properties like
nifi.web.http.host from the wrapper script is a good idea, but for more
static properties like nifi.web.http.port you may want to use the config
files directly as configmaps and do templating using Helm. This is
especially true for more complex configurations like the authorizers.xml or
the login-identity-providers.xml.
- Authorizations in NiFi can be configured for the initial cluster setup,
but needs to be done manually when you add a new Node to the cluster above
the initial cluster size. Also these extra nodes should have a vanilla
authorizations.xml to avoid conflicts when joining to the existing ones.
You can use the wrapper script to decide which configmap to use when
starting the container. Once the pod has started you still have to add the
node and authorize it manually using the UI. There is ongoing work to make
this more dynamic: [3]
- We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in
server mode. The NiFi pods have an init container that uses tls-toolkit in
client mode to request and receive certificates from the CA server. The
communication is protected using a shared secret that is generated inside
the cluster on the fly, also you can further protect access to the CA using
NetworkPolicies.
- You should avoid using the embedded Zookeeper, but you can use an
already existing helm chart as a dependency to install it [4] (caveat: the
image used by that chart is not recommended for production use)
[1]
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh
[2]
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23
[3] https://issues.apache.org/jira/browse/NIFI-5542
[4] https://github.com/helm/charts/tree/master/incubator/zookeeper
On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <Va...@zaplabs.com>
wrote:
> Hi Dnyaneshwar,
>
>
>
> We have nifi running on k8s for around 8-10 months. We create nifi cluster
> as part of CICD and then there is a stage which does the template
> deployment. Haven’t faced any major issues. Just sometime if a node reboots
> the old cluster member in nifi does not gets cleaned up.
>
>
>
> Regards,
>
> Varun
>
>
>
> *From: *Andy LoPresto <al...@apache.org>
> *Reply-To: *<us...@nifi.apache.org>
> *Date: *Thursday, August 30, 2018 at 10:23 AM
> *To: *<us...@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Dnyaneshwar,
>
>
>
> I know other users are working on the same thing, so yes, NiFi +
> Kubernetes will allow you to stand up secure clusters. There is ongoing
> work targeted for upcoming releases to make this easier and more performant
> (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter
> Wilcsinszky has done a lot of great work here, and he may be able to share
> some resources he used/created.
>
>
>
> [1]
> https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC
> <https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20=%20%22Apache%20NiFi%22%20and%20resolution%20%20=%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20=%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC>
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> *alopresto.apache@gmail.com <al...@gmail.com>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <
> dnyaneshwar_pawar@persistent.com> wrote:
>
>
>
> Hi,
>
>
>
> We have requirement of deploying NiFi on cloud platforms and we are
> considering kubernetes as orchestrator. I have knowledge on configuring
> nifi cluster, however, I am not sure on how things would go on kubernetes.
> Further, we are using Apache DS as LDAP server for authentication and
> planning to use embedded zookeeper instance to make the zookeeper cluster.
>
> Any help or pointer to documentation would be appreciated.
>
>
>
> Thank You.
>
>
>
> Regards,
>
> Dnyaneshwar Pawar
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
>
>
> ------------------------------
> This email may be confidential. If you are not the intended recipient,
> please notify us immediately and delete this copy from your system.
>
>
Re: Secure NiFi cluster on kubernetes.
Posted by Varun Tomar <Va...@zaplabs.com>.
Hi Dnyaneshwar,
We have nifi running on k8s for around 8-10 months. We create nifi cluster as part of CICD and then there is a stage which does the template deployment. Haven’t faced any major issues. Just sometime if a node reboots the old cluster member in nifi does not gets cleaned up.
Regards,
Varun
From: Andy LoPresto <al...@apache.org>
Reply-To: <us...@nifi.apache.org>
Date: Thursday, August 30, 2018 at 10:23 AM
To: <us...@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
I know other users are working on the same thing, so yes, NiFi + Kubernetes will allow you to stand up secure clusters. There is ongoing work targeted for upcoming releases to make this easier and more performant (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter Wilcsinszky has done a lot of great work here, and he may be able to share some resources he used/created.
[1] https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC<https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20=%20%22Apache%20NiFi%22%20and%20resolution%20%20=%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20=%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC>
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <dn...@persistent.com>> wrote:
Hi,
We have requirement of deploying NiFi on cloud platforms and we are considering kubernetes as orchestrator. I have knowledge on configuring nifi cluster, however, I am not sure on how things would go on kubernetes. Further, we are using Apache DS as LDAP server for authentication and planning to use embedded zookeeper instance to make the zookeeper cluster.
Any help or pointer to documentation would be appreciated.
Thank You.
Regards,
Dnyaneshwar Pawar
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
________________________________
This email may be confidential. If you are not the intended recipient, please notify us immediately and delete this copy from your system.
Re: Secure NiFi cluster on kubernetes.
Posted by Andy LoPresto <al...@apache.org>.
Hi Dnyaneshwar,
I know other users are working on the same thing, so yes, NiFi + Kubernetes will allow you to stand up secure clusters. There is ongoing work targeted for upcoming releases to make this easier and more performant (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter Wilcsinszky has done a lot of great work here, and he may be able to share some resources he used/created.
[1] https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC <https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20=%20%22Apache%20NiFi%22%20and%20resolution%20%20=%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20=%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <dn...@persistent.com> wrote:
>
> Hi,
>
> We have requirement of deploying NiFi on cloud platforms and we are considering kubernetes as orchestrator. I have knowledge on configuring nifi cluster, however, I am not sure on how things would go on kubernetes. Further, we are using Apache DS as LDAP server for authentication and planning to use embedded zookeeper instance to make the zookeeper cluster.
> Any help or pointer to documentation would be appreciated.
>
> Thank You.
>
> Regards,
> Dnyaneshwar Pawar
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.